Megatron-LM (1.2B)

Training compute

1.1×10²¹ FLOP

Parameters

1.2B

Published

Sep 17, 2019

Megatron-LM (1.2B) is an AI model developed by NVIDIA (United States), first published in September 2019. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 1.1×10²¹ FLOP of compute (estimation method: operation counting). The model has 1,200,000,000 parameters. It was trained on roughly 157B datapoints. Training ran on 1 NVIDIA Tesla V100 SXM3 32 GB.

Access: Unreleased. Its weights are not openly released. The reference paper has 2,766 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: NVIDIA
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 1.1×10²¹ FLOP
Compute estimation method: Operation counting
Parameters: 1,200,000,000
Dataset size: 157B
Training hardware: NVIDIA Tesla V100 SXM3 32 GB
Chips used: 1
Training power draw: 338 W
Model accessibility: Unreleased
Open weights: No
Citations: 2,766
Epoch confidence: Confident