μP (Maximal Update Parametrization)

A framework enabling optimal hyperparameter search on small models and direct transfer to large ones by keeping learning rates c