Megatron-LM (355M)

Training compute

3.4×10²⁰ FLOP

Parameters

355M

Published

Sep 17, 2019

Megatron-LM (355M) is an AI model developed by NVIDIA (United States), first published in September 2019. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 3.4×10²⁰ FLOP of compute (estimation method: operation counting). The model has 355,000,000 parameters. It was trained on roughly 46.4B datapoints.

Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 2,766 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: NVIDIA
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 3.4×10²⁰ FLOP
Compute estimation method: Operation counting
Parameters: 355,000,000
Dataset size: 46.4B
Model accessibility: Open weights (unrestricted)
Open weights: Yes
Citations: 2,766
Epoch confidence: Confident