LUKE
LUKE is an AI model developed by University of Washington and National Institute of Informatics (United States and Japan), first published in October 2020. It works in the language domain, on tasks such as question answering, relation extraction and named entity recognition (ner).
Training it took an estimated 1.8×10²² FLOP of compute (estimation method: hardware). The model has 483,000,000 parameters. It was trained on roughly 4.7B datapoints. Training ran on 16 NVIDIA V100 for about 720 hours. The compute alone is estimated at $4K in 2023 dollars.
Access: Open weights (unrestricted). Its weights are openly available. It is built on top of RoBERTa Large. The reference paper has 766 citations. Epoch AI rates the confidence of this record as likely.
- Organization
- University of Washington, National Institute of Informatics
- Country of organization
- United States, Japan
- Domain
- Language
- Task
- Question answering, Relation extraction, Named entity recognition (NER)
- Training compute
- 1.8×10²² FLOP
- Compute estimation method
- Hardware
- Parameters
- 483,000,000
- Dataset size
- 4.7B
- Training hardware
- NVIDIA V100
- Chips used
- 16
- Training time
- 720 h
- Chip-hours
- 11.5K
- Training power draw
- 9.8 kW
- Training cost (2023 USD)
- $4K
- Numerical format
- FP16
- Model accessibility
- Open weights (unrestricted)
- Open weights
- Yes
- Base model
- RoBERTa Large
- Citations
- 766
- Epoch confidence
- Likely