Mesh-TensorFlow Transformer 2.9B (translation)

Training compute

6.8×10¹⁹ FLOP

Parameters

2.9B

Published

Nov 5, 2018

Mesh-TensorFlow Transformer 2.9B (translation) is an AI model developed by Google Brain (United States), first published in November 2018. It works in the language domain, on tasks such as language modeling/generation and translation.

Training it took an estimated 6.8×10¹⁹ FLOP of compute (estimation method: hardware). The model has 2,900,000,000 parameters. It was trained on roughly 1.6B datapoints. Training ran on 64 Google TPU v2 for about 22 hours. The compute alone is estimated at $396 in 2023 dollars.

Access: Unreleased. Its weights are not openly released. The reference paper has 437 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Google Brain
Country of organization: United States
Domain: Language
Task: Language modeling/generation, Translation
Training compute: 6.8×10¹⁹ FLOP
Compute estimation method: Hardware
Parameters: 2,900,000,000
Dataset size: 1.6B
Training hardware: Google TPU v2
Chips used: 64
Training time: 22 h
Chip-hours: 1.4K
Training power draw: 37.0 kW
Training cost (2023 USD): $396
Model accessibility: Unreleased
Open weights: No
Citations: 437
Epoch confidence: Likely