AI models
Mixture-of-Depths
Training compute
10²⁰ FLOP
Parameters
3B
Published
Apr 2, 2024
Mixture-of-Depths is an AI model developed by Google DeepMind, McGill University and Mila - Quebec AI (originally Montreal Institute for Learning Algorithms) (United States and Canada), first published in April 2024. It works in the language domain, on tasks such as language modeling/generation.
Training it took an estimated 10²⁰ FLOP of compute (estimation method: reported). The model has 3,000,000,000 parameters.
Access: Unreleased. Its weights are not openly released. The reference paper has 187 citations. Epoch AI rates the confidence of this record as confident.
Full record
- Organization
- Google DeepMind, McGill University, Mila - Quebec AI (originally Montreal Institute for Learning Algorithms)
- Country of organization
- United States, Canada
- Domain
- Language
- Task
- Language modeling/generation
- Training compute
- 10²⁰ FLOP
- Compute estimation method
- Reported
- Parameters
- 3,000,000,000
- Model accessibility
- Unreleased
- Open weights
- No
- Citations
- 187
- Epoch confidence
- Confident
SourceEpoch AI, 'AI Models'. Published online at epoch.ai. Retrieved 2026-07-29 from https://epoch.ai/data/ai-models. Licensed under CC BY 4.0.
← All ai models