GPT-2 (117M, SLW 110K)

Training compute

1.1×10²⁰ FLOP

Parameters

117M

Published

Aug 13, 2021

GPT-2 (117M, SLW 110K) is an AI model developed by Microsoft (United States), first published in August 2021. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 1.1×10²⁰ FLOP of compute (estimation method: operation counting). The model has 117,000,000 parameters. Training ran on 128 NVIDIA V100.

Access: Unreleased. Its weights are not openly released. The reference paper has 55 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Microsoft
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 1.1×10²⁰ FLOP
Compute estimation method: Operation counting
Parameters: 117,000,000
Training hardware: NVIDIA V100
Chips used: 128
Training power draw: 77.6 kW
Model accessibility: Unreleased
Open weights: No
Citations: 55
Epoch confidence: Confident