RoBERTa Large
RoBERTa Large is an AI model developed by Facebook and University of Washington (United States), first published in July 2019. It works in the language domain, on tasks such as question answering and language modeling/generation. It counts among the frontier models: the systems trained with the most compute of their moment.
Training it took an estimated 8.5×10²¹ FLOP of compute (estimation method: hardware,operation counting,third-party estimation). The model has 355,000,000 parameters. It was trained on roughly 42.7B datapoints. Training ran on 1,024 NVIDIA Tesla V100 DGXS 32 GB for about 120 hours. The compute alone is estimated at $85K in 2023 dollars.
Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 29,641 citations. Epoch AI rates the confidence of this record as confident.
- Organization
- Facebook, University of Washington
- Country of organization
- United States
- Domain
- Language
- Task
- Question answering, Language modeling/generation
- Training compute
- 8.5×10²¹ FLOP
- Compute estimation method
- Hardware, Operation counting, Third-party estimation
- Parameters
- 355,000,000
- Dataset size
- 42.7B
- Training hardware
- NVIDIA Tesla V100 DGXS 32 GB
- Chips used
- 1,024
- Training time
- 120 h
- Chip-hours
- 122.9K
- Training power draw
- 526.2 kW
- Training cost (2023 USD)
- $85K
- Numerical format
- FP16
- Model accessibility
- Open weights (unrestricted)
- Open weights
- Yes
- Citations
- 29,641
- Epoch confidence
- Confident