Mesh-TensorFlow Transformer 4.9B (language)

Training compute

1.6×10²⁰ FLOP

Parameters

4.9B

Published

Nov 5, 2018

Mesh-TensorFlow Transformer 4.9B (language) is an AI model developed by Google Brain (United States), first published in November 2018. It works in the language domain, on tasks such as language modeling/generation and translation.

Training it took an estimated 1.6×10²⁰ FLOP of compute (estimation method: hardware). The model has 4,900,000,000 parameters. It was trained on roughly 5B datapoints. Training ran on 256 Google TPU v2 for about 13 hours. The compute alone is estimated at $935 in 2023 dollars.

Access: Unreleased. Its weights are not openly released. The reference paper has 437 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Google Brain
Country of organization: United States
Domain: Language
Task: Language modeling/generation, Translation
Training compute: 1.6×10²⁰ FLOP
Compute estimation method: Hardware
Parameters: 4,900,000,000
Dataset size: 5B
Training hardware: Google TPU v2
Chips used: 256
Training time: 13 h
Chip-hours: 3.3K
Training power draw: 148.1 kW
Training cost (2023 USD): $935
Model accessibility: Unreleased
Open weights: No
Citations: 437
Epoch confidence: Confident