Nemotron-4 15B

Training compute

7.5×10²³ FLOP

Parameters

15B

Published

Feb 27, 2024

Nemotron-4 15B is an AI model developed by NVIDIA (United States), first published in February 2024. It works in the language domain, on tasks such as language modeling/generation, code generation, question answering and 2 more.

Training it took an estimated 7.5×10²³ FLOP of compute (estimation method: operation counting,hardware). The model has 15,000,000,000 parameters. Training ran on 3,072 NVIDIA H100 SXM5 80GB for about 312 hours.

Access: Unreleased. Its weights are not openly released. Epoch AI rates the confidence of this record as confident.

Full record

Organization: NVIDIA
Country of organization: United States
Domain: Language
Task: Language modeling/generation, Code generation, Question answering, Translation, Quantitative reasoning
Training compute: 7.5×10²³ FLOP
Compute estimation method: Operation counting, Hardware
Parameters: 15,000,000,000
Training hardware: NVIDIA H100 SXM5 80GB
Chips used: 3,072
Training time: 312 h
Training power draw: 4.3 MW
Numerical format: BF16
Model accessibility: Unreleased
Open weights: No
Epoch confidence: Confident

Benchmark results

01	Epoch Capabilities Index	ECI Score	107.4
02	HellaSwag	Overall accuracy	0.82
03	PIQA	Score	0.82
04	WinoGrande	Accuracy	0.78
05	MMLU	EM	0.59
06	BIG-Bench Hard	Average	0.59
07	ARC AI2	Challenge score	0.56
08	GSM8K	EM	0.46