Falcon-180B
Falcon-180B is an AI model developed by Technology Innovation Institute (United Arab Emirates), first published in September 2023. It works in the language domain, on tasks such as language modeling, language modeling/generation and question answering. It counts among the frontier models: the systems trained with the most compute of their moment.
Training it took an estimated 3.8×10²⁴ FLOP of compute (estimation method: reported,operation counting). The model has 180,000,000,000 parameters. It was trained on roughly 3.5T datapoints. Training ran on 4,096 NVIDIA A100 SXM4 40 GB for about 4.3K hours. The compute alone is estimated at $11M in 2023 dollars.
Access: Open weights (restricted use). Its weights are openly available. The reference paper has 691 citations. Epoch AI rates the confidence of this record as confident.
- Organization
- Technology Innovation Institute
- Country of organization
- United Arab Emirates
- Domain
- Language
- Task
- Language modeling, Language modeling/generation, Question answering
- Training compute
- 3.8×10²⁴ FLOP
- Compute estimation method
- Reported, Operation counting
- Parameters
- 180,000,000,000
- Dataset size
- 3.5T
- Training hardware
- NVIDIA A100 SXM4 40 GB
- Chips used
- 4,096
- Training time
- 4,320 h
- Chip-hours
- 17.7M
- Training power draw
- 3.3 MW
- Training cost (2023 USD)
- $11M
- Numerical format
- BF16
- Model accessibility
- Open weights (restricted use)
- Open weights
- Yes
- Citations
- 691
- Epoch confidence
- Confident
| 01 | Epoch Capabilities Index | ECI Score | 111.9 |
| 02 | HellaSwag | Overall accuracy | 0.89 |
| 03 | BoolQ | Score | 0.89 |
| 04 | WinoGrande | Accuracy | 0.87 |
| 05 | PIQA | Score | 0.85 |
| 06 | LAMBADA | Score | 0.8 |
| 07 | MMLU | EM | 0.71 |
| 08 | ARC AI2 | Challenge score | 0.68 |
| 09 | OpenBookQA | Accuracy | 0.64 |
| 10 | GSM8K | EM | 0.54 |