PaLM (540B)

Training compute

2.5×10²⁴ FLOP

Parameters

540.4B

Published

Apr 4, 2022

PaLM (540B) is an AI model developed by Google Research (United States), first published in April 2022. It works in the language domain, on tasks such as language modeling, code generation and translation. It counts among the frontier models: the systems trained with the most compute of their moment.

Training it took an estimated 2.5×10²⁴ FLOP of compute (estimation method: hardware). The model has 540,350,000,000 parameters. It was trained on roughly 780B datapoints. Training ran on 6,144 Google TPU v4 for about 1.5K hours. The compute alone is estimated at $3M in 2023 dollars.

Access: Unreleased. Its weights are not openly released. The reference paper has 7,999 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Google Research
Country of organization: United States
Domain: Language
Task: Language modeling, Code generation, Translation
Training compute: 2.5×10²⁴ FLOP
Compute estimation method: Hardware
Parameters: 540,350,000,000
Dataset size: 780B
Training hardware: Google TPU v4
Chips used: 6,144
Training time: 1,536 h
Chip-hours: 8.4M
Training power draw: 4.2 MW
Training cost (2023 USD): $3M
Model accessibility: Unreleased
Open weights: No
Citations: 7,999
Epoch confidence: Confident

Benchmark results

01	BoolQ	Score	0.89
02	ARC AI2	Challenge score	0.85
03	WinoGrande	Accuracy	0.85
04	HellaSwag	Overall accuracy	0.84
05	PIQA	Score	0.82
06	TriviaQA	EM	0.81
07	LAMBADA	Score	0.78
08	MMLU	EM	0.69
09	OpenBookQA	Accuracy	0.68
10	GSM8K	EM	0.56