Pythia-12b
Pythia-12b is an AI model developed by EleutherAI, Booz Allen Hamilton, McLean, University of Cambridge, Indraprastha Institute of Information Technology Delhi, Stability AI and 2 more (United States, United Kingdom, India and Netherlands), first published in April 2023. It works in the language domain, on tasks such as language modeling/generation and question answering.
Training it took an estimated 2.2×10²² FLOP of compute (estimation method: operation counting). The model has 12,000,000,000 parameters. It was trained on roughly 299.9B datapoints. Training ran on 256 NVIDIA A100 SXM4 40 GB.
Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 1,862 citations. Epoch AI rates the confidence of this record as confident.
- Organization
- EleutherAI, Booz Allen Hamilton, McLean, University of Cambridge, Indraprastha Institute of Information Technology Delhi, Stability AI, datasaur.ai, University of Amsterdam
- Country of organization
- United States, United Kingdom, India, Netherlands
- Domain
- Language
- Task
- Language modeling/generation, Question answering
- Training compute
- 2.2×10²² FLOP
- Compute estimation method
- Operation counting
- Parameters
- 12,000,000,000
- Dataset size
- 299.9B
- Training hardware
- NVIDIA A100 SXM4 40 GB
- Chips used
- 256
- Chip-hours
- 72.3K
- Training power draw
- 204.1 kW
- Model accessibility
- Open weights (unrestricted)
- Open weights
- Yes
- Citations
- 1,862
- Epoch confidence
- Confident