GPT-2 (1.5B)

Training compute

1.9×10²¹ FLOP

Parameters

1.5B

Published

Feb 14, 2019

GPT-2 (1.5B) is an AI model developed by OpenAI (United States), first published in February 2019. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 1.9×10²¹ FLOP of compute (estimation method: operation counting). The model has 1,500,000,000 parameters. It was trained on roughly 10.7B datapoints. Training ran on Google TPU v3. The compute alone is estimated at $4K in 2023 dollars.

Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 26,463 citations. Epoch AI rates the confidence of this record as speculative.

Full record

Organization: OpenAI
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 1.9×10²¹ FLOP
Compute estimation method: Operation counting
Parameters: 1,500,000,000
Dataset size: 10.7B
Training hardware: Google TPU v3
Training cost (2023 USD): $4K
Model accessibility: Open weights (unrestricted)
Open weights: Yes
Citations: 26,463
Epoch confidence: Speculative