AI models
QMoE: compressed SwitchTransformer
Parameters
1.6T
Published
Oct 25, 2023
QMoE: compressed SwitchTransformer is an AI model developed by Institute of Science and Technology Austria (ISTA) and Neural Magic (Austria and United States), first published in October 2023. It works in the language domain, on tasks such as language modeling.
Epoch AI has no training-compute estimate for this model. The model has 1,571,000,000,000 parameters. It was trained on roughly 576B datapoints. Training ran on NVIDIA RTX A6000.
Access: Open weights (unrestricted). Its weights are openly available. It is built on top of Switch. The reference paper has 47 citations. Epoch AI rates the confidence of this record as confident.
Full record
- Organization
- Institute of Science and Technology Austria (ISTA), Neural Magic
- Country of organization
- Austria, United States
- Domain
- Language
- Task
- Language modeling
- Parameters
- 1,571,000,000,000
- Dataset size
- 576B
- Training hardware
- NVIDIA RTX A6000
- Model accessibility
- Open weights (unrestricted)
- Open weights
- Yes
- Base model
- Switch
- Citations
- 47
- Epoch confidence
- Confident
More from Institute of Science and Technology Austria (ISTA),Neural Magic
SourceEpoch AI, 'AI Models'. Published online at epoch.ai. Retrieved 2026-07-29 from https://epoch.ai/data/ai-models. Licensed under CC BY 4.0.
← All ai models