Megatron-LM (2.5B)

Training compute

2.4×10²¹ FLOP

Parameters

2.5B

Published

Sep 17, 2019

Megatron-LM (2.5B) is an AI model developed by NVIDIA (United States), first published in September 2019. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 2.4×10²¹ FLOP of compute (estimation method: operation counting). The model has 2,500,000,000 parameters. Training ran on 128 NVIDIA V100.

Access: Unreleased. Its weights are not openly released. The reference paper has 2,766 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: NVIDIA
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 2.4×10²¹ FLOP
Compute estimation method: Operation counting
Parameters: 2,500,000,000
Training hardware: NVIDIA V100
Chips used: 128
Training power draw: 78.8 kW
Model accessibility: Unreleased
Open weights: No
Citations: 2,766
Epoch confidence: Likely