Published in ICLR Workshop on Scaling Self-Improving Foundation Models without Human Supervision (Oral), 2025
This paper presents a coding agent that can improve itself without human supervision.
Recommended citation: Robeyns M, Szummer M, and Aitchison L. (2025). "A self-improving coding agent." ICLR Workshop on Scaling Self-Improving Foundation Models without Human Supervision (Oral).
Published in arXiv, 2024
This paper examines questionable research practices in the field of machine learning.
Recommended citation: Leech G, Vazquez JJ, Kupper N, Yagudin M, and Aitchison L. (2024). "Questionable practices in machine learning." arXiv:2407.12220.
Download Paper
Published in arXiv, 2024
This paper provides guidelines for setting AdamW weight decay parameters when scaling model and dataset size.
Recommended citation: Wang X and Aitchison L. (2024). "How to set AdamW's weight decay as you scale model and dataset size." arXiv:2405.13698.
Download Paper
Published in arXiv, 2024
This position paper argues against using the Central Limit Theorem in LLM evaluations with small datasets.
Recommended citation: Bowyer S, Aitchison L, and Ivanova DR. (2024). "Position: Don't use the CLT in LLM evals with fewer than a few hundred datapoints." arXiv:2503.01747.
Download Paper
Published in arXiv, 2024
This paper introduces Jacobian sparse autoencoders that sparsify computations rather than just activations.
Recommended citation: Farnik L, Lawson T, Houghton C, and Aitchison L. (2024). "Jacobian sparse autoencoders: Sparsify computations, not just activations." arXiv:2502.18147.
Download Paper
Published in arXiv, 2024
This paper introduces a variant of Adam that is invariant to batch size.
Recommended citation: Wang X and Aitchison L. (2024). "Batch size invariant Adam." arXiv:2402.18824.
Download Paper
Published in arXiv, 2024
This paper explores learning rates in function space.
Recommended citation: Milsom E, Anson B, and Aitchison L. (2024). "Function-space learning rates." arXiv:2502.17405.
Download Paper
Published in arXiv, 2024
This paper demonstrates how sparse autoencoders can be used to interpret randomly initialized transformers.
Recommended citation: Heap T, Lawson T, Farnik L, and Aitchison L. (2024). "Sparse autoencoders can interpret randomly initialized transformers." arXiv:2501.17727.
Download Paper
Published in Workshop on Secure and Trustworthy Large Language Models, 2024
This paper explores Bayesian reward models for aligning large language models.
Recommended citation: Yang AX, Robeyns M, Coste T, Wang J, Bou-Ammar H, and Aitchison L. (2024). "Bayesian reward models for LLM alignment." Workshop on Secure and Trustworthy Large Language Models.
Published in NeurIPS Workshop: Efficient Natural Language and Speech Processing, 2023
This paper investigates the use of LoRA ensembles for fine-tuning large language models.
Recommended citation: Wang X, Aitchison L, and Rudolph M. (2023). "LoRA ensembles for large language model fine-tuning." NeurIPS Workshop: Efficient Natural Language and Speech Processing.