StarCoder
StarCoder is an AI model developed by Hugging Face, ServiceNow, Northeastern University, Mila - Quebec AI (originally Montreal Institute for Learning Algorithms), Carnegie Mellon University (CMU), Johns Hopkins University and 31 more (United States, Canada, Germany, United Kingdom and 7 more), first published in May 2023. It works in the language domain, on tasks such as code generation.
Training it took an estimated 8.5×10²² FLOP of compute (estimation method: reported,hardware). The model has 15,500,000,000 parameters. It was trained on roughly 203.8B datapoints. Training ran on 512 NVIDIA A100 SXM4 80 GB for about 626 hours. The compute alone is estimated at $212K in 2023 dollars.
Access: Open weights (restricted use). Its weights are openly available. The reference paper has 1,181 citations. Epoch AI rates the confidence of this record as confident.
- Organization
- Hugging Face, ServiceNow, Northeastern University, Mila - Quebec AI (originally Montreal Institute for Learning Algorithms), Carnegie Mellon University (CMU), Johns Hopkins University, Leipzig University, ScaDS.AI, Queen Mary University of London, Roblox, Sea AI Lab, Technion - Israel Institute of Technology, Monash University, CSIRO, Data61, McGill University, Saama, University of British Columbia (UBC), Massachusetts Institute of Technology (MIT), Technical University of Munich, IBM, University of Vermont, UnfoldML, SAP, University of Notre Dame, Columbia University, New York University (NYU), University of Allahabad, Discover Dollar, Toloka, Telefonica, Stanford University, Weizmann Institute of Science, Alan Turing Institute, Wellesley College, EleutherAI, Forschungszentrum Julich
- Country of organization
- United States, Canada, Germany, United Kingdom, Singapore, Israel, Australia, Sweden, India, Netherlands, Spain
- Domain
- Language
- Task
- Code generation
- Training compute
- 8.5×10²² FLOP
- Compute estimation method
- Reported, Hardware
- Parameters
- 15,500,000,000
- Dataset size
- 203.8B
- Training hardware
- NVIDIA A100 SXM4 80 GB
- Chips used
- 512
- Training time
- 626 h
- Chip-hours
- 320.3K
- Training power draw
- 408.0 kW
- Training cost (2023 USD)
- $212K
- Numerical format
- BF16
- Model accessibility
- Open weights (restricted use)
- Open weights
- Yes
- Citations
- 1,181
- Epoch confidence
- Confident