Long-range sequence Compressive Transformers

Training compute

10²⁰ FLOP

Published

Nov 13, 2019

Long-range sequence Compressive Transformers is an AI model developed by DeepMind (United Kingdom), first published in November 2019. It works in the language domain, on tasks such as language modeling.

Training it took an estimated 10²⁰ FLOP of compute (estimation method: hardware). It was trained on roughly 103M datapoints. Training ran on 64 Google TPU v3 for about 12 hours.

Access: Unreleased. Its weights are not openly released. The reference paper has 845 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: DeepMind
Country of organization: United Kingdom
Domain: Language
Task: Language modeling
Training compute: 10²⁰ FLOP
Compute estimation method: Hardware
Dataset size: 103M
Training hardware: Google TPU v3
Chips used: 64
Training time: 12 h
Training power draw: 59.0 kW
Model accessibility: Unreleased
Open weights: No
Citations: 845
Epoch confidence: Confident