Flan-PaLM 540B

Training compute

2.5×10²⁴ FLOP

Parameters

540B

Published

Oct 20, 2022

Flan-PaLM 540B is an AI model developed by Google (United States), first published in October 2022. It works in the language domain, on tasks such as language modeling/generation, question answering and mathematical reasoning. It counts among the frontier models: the systems trained with the most compute of their moment.

Training it took an estimated 2.5×10²⁴ FLOP of compute (estimation method: reported,hardware). The model has 540,000,000,000 parameters. It was trained on roughly 1.4B datapoints. Training ran on 512 Google TPU v4 for about 37 hours.

Access: Unreleased. Its weights are not openly released. It is built on top of PaLM (540B). The reference paper has 4,129 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Google
Country of organization: United States
Domain: Language
Task: Language modeling/generation, Question answering, Mathematical reasoning
Training compute: 2.5×10²⁴ FLOP
Compute estimation method: Reported, Hardware
Parameters: 540,000,000,000
Dataset size: 1.4B
Training hardware: Google TPU v4
Chips used: 512
Training time: 37 h
Chip-hours: 18.9K
Training power draw: 348.3 kW
Model accessibility: Unreleased
Open weights: No
Base model: PaLM (540B)
Citations: 4,129
Epoch confidence: Confident