DistilBERT

Training compute

1.2×10¹⁹ FLOP

Parameters

66M

Published

Oct 2, 2019

DistilBERT is an AI model developed by Hugging Face (United States), first published in October 2019. It works in the language domain, on tasks such as text autocompletion.

Training it took an estimated 1.2×10¹⁹ FLOP of compute (estimation method: hardware). The model has 66,000,000 parameters. It was trained on roughly 495M datapoints. Training ran on NVIDIA Tesla V100 DGXS 16 GB.

Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 9,714 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Hugging Face
Country of organization: United States
Domain: Language
Task: Text autocompletion
Training compute: 1.2×10¹⁹ FLOP
Compute estimation method: Hardware
Parameters: 66,000,000
Dataset size: 495M
Training hardware: NVIDIA Tesla V100 DGXS 16 GB
Model accessibility: Open weights (unrestricted)
Open weights: Yes
Citations: 9,714
Epoch confidence: Confident