AI models
TinyBert
Training compute
4×10¹⁸ FLOP
Parameters
67M
Published
Oct 16, 2020
TinyBert is an AI model developed by Huazhong University of Science and Technology, Huawei Noah's Ark Lab and Huawei (China), first published in October 2020. It works in the language domain, on tasks such as language modeling/generation and question answering.
Training it took an estimated 4×10¹⁸ FLOP of compute (estimation method: operation counting). The model has 67,000,000 parameters. It was trained on roughly 3.3B datapoints.
Access: Open weights (unrestricted). Its weights are openly available. Epoch AI rates the confidence of this record as confident.
Full record
- Organization
- Huazhong University of Science and Technology, Huawei Noah's Ark Lab, Huawei
- Country of organization
- China
- Domain
- Language
- Task
- Language modeling/generation, Question answering
- Training compute
- 4×10¹⁸ FLOP
- Compute estimation method
- Operation counting
- Parameters
- 67,000,000
- Dataset size
- 3.3B
- Model accessibility
- Open weights (unrestricted)
- Open weights
- Yes
- Epoch confidence
- Confident
SourceEpoch AI, 'AI Models'. Published online at epoch.ai. Retrieved 2026-07-29 from https://epoch.ai/data/ai-models. Licensed under CC BY 4.0.
← All ai models