MegaScale (530B)

Training compute

9.7×10²³ FLOP

Parameters

530B

Published

Feb 23, 2024

MegaScale (530B) is an AI model developed by ByteDance and Peking University (China), first published in February 2024. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 9.7×10²³ FLOP of compute (estimation method: comparison with other models). The model has 530,000,000,000 parameters. It was trained on roughly 300B datapoints. Training ran on 11,200 NVIDIA A100 for about 118 hours.

Access: Unreleased. Its weights are not openly released. The reference paper has 302 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: ByteDance, Peking University
Country of organization: China
Domain: Language
Task: Language modeling/generation
Training compute: 9.7×10²³ FLOP
Compute estimation method: Comparison with other models
Parameters: 530,000,000,000
Dataset size: 300B
Training hardware: NVIDIA A100
Chips used: 11,200
Training time: 118 h
Training power draw: 8.9 MW
Model accessibility: Unreleased
Open weights: No
Citations: 302
Epoch confidence: Confident