DeepSeek-V3

Training compute

3.3×10²⁴ FLOP

Parameters

671B

Published

Dec 24, 2024

DeepSeek-V3 is an AI model developed by DeepSeek (China), first published in December 2024. It works in the language domain, on tasks such as language modeling/generation, code generation, quantitative reasoning and question answering.

Training it took an estimated 3.3×10²⁴ FLOP of compute (estimation method: operation counting,hardware). The model has 671,000,000,000 parameters. It was trained on roughly 14.8T datapoints. Training ran on 2,048 NVIDIA H800 SXM5. The compute alone is estimated at $5M in 2023 dollars.

Access: Open weights (restricted use). Its weights are openly available. Epoch AI rates the confidence of this record as confident.

Full record

Organization: DeepSeek
Country of organization: China
Domain: Language
Task: Language modeling/generation, Code generation, Quantitative reasoning, Question answering
Training compute: 3.3×10²⁴ FLOP
Compute estimation method: Operation counting, Hardware
Parameters: 671,000,000,000
Dataset size: 14.8T
Training hardware: NVIDIA H800 SXM5
Chips used: 2,048
Chip-hours: 2.8M
Training power draw: 2.8 MW
Training cost (2023 USD): $5M
Numerical format: FP8
Model accessibility: Open weights (restricted use)
Open weights: Yes
Epoch confidence: Confident

Benchmark results

01	Epoch Capabilities Index	ECI Score	133
02	LiveBench	Global average	60.5
03	ForecastBench	Overall score	59.1
04	Aider Polyglot	Percent correct	48.4
05	ARC AI2	Challenge score	0.95
06	HellaSwag	Overall accuracy	0.89
07	BIG-Bench Hard	Average	0.88
08	MMLU	EM	0.87
09	WinoGrande	Accuracy	0.85
10	TriviaQA	EM	0.83
11	MATH Level 5	mean_score	0.65
12	GPQA Diamond	mean_score	0.57
13	METR Time Horizons	average_score	0.47
14	SimpleBench	Score (AVG@5)	0.19
15	OTIS Mock AIME 2024-2025	mean_score	0.16
16	FrontierMath	mean_score	0.02
17	Critpt	Accuracy	0