XLM-RoBERTa
XLM-RoBERTa is an AI model developed by Facebook AI (United States), first published in November 2019. It works in the language domain, on tasks such as named entity recognition (ner), question answering and text classification. It counts among the frontier models: the systems trained with the most compute of their moment.
Training it took an estimated 2.1×10²² FLOP of compute (estimation method: operation counting). The model has 550,000,000 parameters. It was trained on roughly 167B datapoints. Training ran on 500 NVIDIA Tesla V100 DGXS 32 GB. The compute alone is estimated at $78K in 2023 dollars.
Access: Open weights (non-commercial). Its weights are openly available. The reference paper has 8,373 citations. Epoch AI rates the confidence of this record as confident.
- Organization
- Facebook AI
- Country of organization
- United States
- Domain
- Language
- Task
- Named entity recognition (NER), Question answering, Text classification
- Training compute
- 2.1×10²² FLOP
- Compute estimation method
- Operation counting
- Parameters
- 550,000,000
- Dataset size
- 167B
- Training hardware
- NVIDIA Tesla V100 DGXS 32 GB
- Chips used
- 500
- Training power draw
- 256.2 kW
- Training cost (2023 USD)
- $78K
- Model accessibility
- Open weights (non-commercial)
- Open weights
- Yes
- Citations
- 8,373
- Epoch confidence
- Confident