GPT3-6.7B + muP

Training compute

1.3×10²² FLOP

Parameters

6.7B

Published

Mar 7, 2022

GPT3-6.7B + muP is an AI model developed by Microsoft and OpenAI (United States), first published in March 2022. It works in the language domain, on tasks such as language modeling/generation.

Training it took an estimated 1.3×10²² FLOP of compute (estimation method: comparison with other models,operation counting). The model has 6,700,000,000 parameters. It was trained on roughly 300B datapoints.

Access: Unreleased. Its weights are not openly released. The reference paper has 262 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Microsoft, OpenAI
Country of organization: United States
Domain: Language
Task: Language modeling/generation
Training compute: 1.3×10²² FLOP
Compute estimation method: Comparison with other models, Operation counting
Parameters: 6,700,000,000
Dataset size: 300B
Model accessibility: Unreleased
Open weights: No
Citations: 262
Epoch confidence: Confident