Llama 3.1-405B
Llama 3.1-405B is an AI model developed by Meta AI (United States), first published in July 2024. It works in the language domain, on tasks such as language modeling/generation, question answering, code generation and mathematical reasoning. It counts among the frontier models: the systems trained with the most compute of their moment.
Training it took an estimated 3.8×10²⁵ FLOP of compute (estimation method: reported,operation counting). The model has 405,000,000,000 parameters. It was trained on roughly 15.6T datapoints. Training ran on 16,384 NVIDIA H100 SXM5 80GB for about 2.1K hours. The compute alone is estimated at $53M in 2023 dollars.
Access: Open weights (restricted use). Its weights are openly available. Epoch AI rates the confidence of this record as confident.
- Organization
- Meta AI
- Country of organization
- United States
- Domain
- Language
- Task
- Language modeling/generation, Question answering, Code generation, Mathematical reasoning
- Training compute
- 3.8×10²⁵ FLOP
- Compute estimation method
- Reported, Operation counting
- Parameters
- 405,000,000,000
- Dataset size
- 15.6T
- Training hardware
- NVIDIA H100 SXM5 80GB
- Chips used
- 16,384
- Training time
- 2,142 h
- Training power draw
- 22.6 MW
- Training cost (2023 USD)
- $53M
- Numerical format
- BF16
- Model accessibility
- Open weights (restricted use)
- Open weights
- Yes
- Epoch confidence
- Confident
| 01 | Epoch Capabilities Index | ECI Score | 129 |
| 02 | ARC AI2 | Challenge score | 0.95 |
| 03 | WinoGrande | Accuracy | 0.89 |
| 04 | HellaSwag | Overall accuracy | 0.89 |
| 05 | PIQA | Score | 0.86 |
| 06 | MMLU | EM | 0.84 |
| 07 | BIG-Bench Hard | Average | 0.83 |
| 08 | TriviaQA | EM | 0.83 |