Multi-Token Prediction 13B

Training compute

1.5×10²³ FLOP

Parameters

13B

Published

Apr 30, 2024

Multi-Token Prediction 13B is an AI model developed by Facebook AI Research (United States and France), first published in April 2024. It works in the language domain, on tasks such as code generation.

Training it took an estimated 1.5×10²³ FLOP of compute (estimation method: hardware). The model has 13,000,000,000 parameters. Training ran on NVIDIA A100 SXM4 80 GB,NVIDIA H100 SXM5 80GB.

Access: Unreleased. Its weights are not openly released. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Facebook AI Research
Country of organization: United States, France
Domain: Language
Task: Code generation
Training compute: 1.5×10²³ FLOP
Compute estimation method: Hardware
Parameters: 13,000,000,000
Training hardware: NVIDIA A100 SXM4 80 GB, NVIDIA H100 SXM5 80GB
Model accessibility: Unreleased
Open weights: No
Epoch confidence: Likely