CodeT5-large

Training compute

2.7×10²¹ FLOP

Parameters

770M

Published

Jul 5, 2022

CodeT5-large is an AI model developed by Salesforce (United States), first published in July 2022. It works in the language domain, on tasks such as code generation.

Training it took an estimated 2.7×10²¹ FLOP of compute (estimation method: hardware). The model has 770,000,000 parameters. It was trained on roughly 10.5B datapoints. Training ran on NVIDIA A100 for about 504 hours. The compute alone is estimated at $4K in 2023 dollars.

Access: Open weights (unrestricted). Its weights are openly available. The reference paper has 465 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Salesforce
Country of organization: United States
Domain: Language
Task: Code generation
Training compute: 2.7×10²¹ FLOP
Compute estimation method: Hardware
Parameters: 770,000,000
Dataset size: 10.5B
Training hardware: NVIDIA A100
Training time: 504 h
Training cost (2023 USD): $4K
Numerical format: FP16
Model accessibility: Open weights (unrestricted)
Open weights: Yes
Citations: 465
Epoch confidence: Likely