Cerebras-GPT-13B

Training compute

2.3×10²² FLOP

Parameters

13B

Published

Apr 6, 2023

Cerebras-GPT-13B is an AI model developed by Cerebras Systems (United States), first published in April 2023. It works in the language domain, on tasks such as language modeling.

Training it took an estimated 2.3×10²² FLOP of compute (estimation method: reported). The model has 13,000,000,000 parameters. It was trained on roughly 257.1B datapoints. Training ran on Cerebras CS-2.

Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 136 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Cerebras Systems
Country of organization: United States
Domain: Language
Task: Language modeling
Training compute: 2.3×10²² FLOP
Compute estimation method: Reported
Parameters: 13,000,000,000
Dataset size: 257.1B
Training hardware: Cerebras CS-2
Model accessibility: Open weights (unrestricted)
Open weights: Yes
Citations: 136
Epoch confidence: Confident

Benchmark results

01	Epoch Capabilities Index	ECI Score	82.8
02	PIQA	Score	0.73
03	BoolQ	Score	0.61
04	WinoGrande	Accuracy	0.61
05	HellaSwag	Overall accuracy	0.59
06	OpenBookQA	Accuracy	0.36
07	ARC AI2	Challenge score	0.32
08	MMLU	EM	0.26