PaLM 2

Training compute

7.3×10²⁴ FLOP

Parameters

340B

Published

May 10, 2023

PaLM 2 is an AI model developed by Google (United States), first published in May 2023. It works in the language domain, on tasks such as language modeling and language modeling/generation. It counts among the frontier models: the systems trained with the most compute of their moment.

Training it took an estimated 7.3×10²⁴ FLOP of compute (estimation method: operation counting). The model has 340,000,000,000 parameters. It was trained on roughly 3.6T datapoints. Training ran on Google TPU v4. The compute alone is estimated at $5M in 2023 dollars.

Access: API access. Its weights are not openly released. The reference paper has 1,734 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Google
Country of organization: United States
Domain: Language
Task: Language modeling, Language modeling/generation
Training compute: 7.3×10²⁴ FLOP
Compute estimation method: Operation counting
Parameters: 340,000,000,000
Dataset size: 3.6T
Training hardware: Google TPU v4
Training cost (2023 USD): $5M
Model accessibility: API access
Open weights: No
Citations: 1,734
Epoch confidence: Likely