RETRO-7B

Training compute

1.7×10²² FLOP

Parameters

7.5B

Published

Feb 7, 2022

RETRO-7B is an AI model developed by DeepMind (United Kingdom), first published in February 2022. It works in the language domain, on tasks such as language modeling/generation and language modeling.

Training it took an estimated 1.7×10²² FLOP of compute (estimation method: operation counting). The model has 7,500,000,000 parameters. It was trained on roughly 419.4B datapoints.

Access: Unreleased. Its weights are not openly released. The reference paper has 1,665 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: DeepMind
Country of organization: United Kingdom
Domain: Language
Task: Language modeling/generation, Language modeling
Training compute: 1.7×10²² FLOP
Compute estimation method: Operation counting
Parameters: 7,500,000,000
Dataset size: 419.4B
Model accessibility: Unreleased
Open weights: No
Citations: 1,665
Epoch confidence: Confident