Transformer

Training compute

7.4×10¹⁸ FLOP

Parameters

213M

Published

Jun 12, 2017

Transformer is an AI model developed by Google Research and Google Brain (United States), first published in June 2017. It works in the language domain, on tasks such as translation.

Training it took an estimated 7.4×10¹⁸ FLOP of compute (estimation method: hardware,third-party estimation). The model has 213,000,000 parameters. It was trained on roughly 832M datapoints. Training ran on 8 NVIDIA P100 for about 84 hours. The compute alone is estimated at $438 in 2023 dollars.

Access: Unreleased. Its weights are not openly released. The reference paper has 159,251 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Google Research, Google Brain
Country of organization: United States
Domain: Language
Task: Translation
Training compute: 7.4×10¹⁸ FLOP
Compute estimation method: Hardware, Third-party estimation
Parameters: 213,000,000
Dataset size: 832M
Training hardware: NVIDIA P100
Chips used: 8
Training time: 84 h
Chip-hours: 672
Training power draw: 4.2 kW
Training cost (2023 USD): $438
Numerical format: FP32
Model accessibility: Unreleased
Open weights: No
Citations: 159,251
Epoch confidence: Confident