GPT-2 (1.5B, Curriculum Learning 45K)

Training compute

2.4×10²¹ FLOP

Parameters

1.5B

Published

Aug 13, 2021

GPT-2 (1.5B, Curriculum Learning 45K) is an AI model developed by Microsoft (United States), first published in August 2021. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 2.4×10²¹ FLOP of compute (estimation method: comparison with other models,hardware,operation counting). The model has 1,500,000,000 parameters. It was trained on roughly 157B datapoints. Training ran on 128 NVIDIA V100 for about 155 hours.

Access: Unreleased. Its weights are not openly released. The reference paper has 55 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Microsoft
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 2.4×10²¹ FLOP
Compute estimation method: Comparison with other models, Hardware, Operation counting
Parameters: 1,500,000,000
Dataset size: 157B
Training hardware: NVIDIA V100
Chips used: 128
Training time: 155 h
Training power draw: 77.6 kW
Model accessibility: Unreleased
Open weights: No
Citations: 55
Epoch confidence: Likely