BLOOM-176B

Training compute

3.7×10²³ FLOP

Parameters

176.2B

Published

Jul 11, 2022

BLOOM-176B is an AI model developed by Hugging Face and BigScience (United States and France), first published in July 2022. It works in the language domain, on tasks such as language modeling, translation and code generation.

Training it took an estimated 3.7×10²³ FLOP of compute (estimation method: hardware). The model has 176,247,271,424 parameters. It was trained on roughly 379B datapoints. Training ran on 384 NVIDIA A100 SXM4 80 GB for about 2.8K hours. The compute alone is estimated at $996K in 2023 dollars.

Access: Open weights (restricted use). Its weights are openly available. The reference paper has 2,917 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Hugging Face, BigScience
Country of organization: United States, France
Domain: Language
Task: Language modeling, Translation, Code generation
Training compute: 3.7×10²³ FLOP
Compute estimation method: Hardware
Parameters: 176,247,271,424
Dataset size: 379B
Training hardware: NVIDIA A100 SXM4 80 GB
Chips used: 384
Training time: 2,808 h
Chip-hours: 1.1M
Training power draw: 308.0 kW
Training cost (2023 USD): $996K
Numerical format: BF16
Model accessibility: Open weights (restricted use)
Open weights: Yes
Citations: 2,917
Epoch confidence: Confident