Sparse Wide GPT-3 Small

Training compute

1.9×10¹⁸ FLOP

Parameters

1.3B

Published

Mar 21, 2023

Sparse Wide GPT-3 Small is an AI model developed by Cerebras Systems (United States), first published in March 2023. It works in the language domain, on tasks such as language modeling/generation and question answering.

Training it took an estimated 1.9×10¹⁸ FLOP of compute (estimation method: operation counting). The model has 1,300,000,000 parameters. It was trained on roughly 2.5B datapoints. Training ran on Cerebras CS-2.

Access: Unreleased. Its weights are not openly released. The reference paper has 8 citations. Epoch AI rates the confidence of this record as speculative.

Full record

Organization: Cerebras Systems
Country of organization: United States
Domain: Language
Task: Language modeling/generation, Question answering
Training compute: 1.9×10¹⁸ FLOP
Compute estimation method: Operation counting
Parameters: 1,300,000,000
Dataset size: 2.5B
Training hardware: Cerebras CS-2
Model accessibility: Unreleased
Open weights: No
Citations: 8
Epoch confidence: Speculative