WizardLM-7B

Training compute

4×10²² FLOP

Parameters

6.7B

Published

Apr 24, 2023

WizardLM-7B is an AI model developed by Microsoft and Peking University (United States and China), first published in April 2023. It works in the language domain, on tasks such as language modeling.

Training it took an estimated 4×10²² FLOP of compute (estimation method: hardware). The model has 6,700,000,000 parameters. Training ran on 8 NVIDIA V100 for about 70 hours. The compute alone is estimated at $47K in 2023 dollars.

Access: Open weights (non-commercial). Its weights are openly available. It is built on top of LLaMA-7B. The reference paper has 1,211 citations. Epoch AI rates the confidence of this record as confident.

Full record

Organization: Microsoft, Peking University
Country of organization: United States, China
Domain: Language
Task: Language modeling
Training compute: 4×10²² FLOP
Compute estimation method: Hardware
Parameters: 6,700,000,000
Training hardware: NVIDIA V100
Chips used: 8
Training time: 70 h
Training power draw: 4.8 kW
Training cost (2023 USD): $47K
Model accessibility: Open weights (non-commercial)
Open weights: Yes
Base model: LLaMA-7B
Citations: 1,211
Epoch confidence: Confident