How to set AdamW’s weight decay as you scale model and dataset size
Published in arXiv, 2024
Recommended citation: Wang X and Aitchison L. (2024). "How to set AdamW's weight decay as you scale model and dataset size." arXiv:2405.13698.
Download Paper