PermuteFormer

Training compute

2.8×10¹⁸ FLOP

Parameters

149.7M

Published

Sep 6, 2021

PermuteFormer is an AI model developed by Peking University (China), first published in September 2021. It works in the language domain, on tasks such as language modeling.

Training it took an estimated 2.8×10¹⁸ FLOP of compute (estimation method: operation counting). The model has 149,697,024 parameters. It was trained on roughly 103M datapoints. Training ran on 8 NVIDIA V100.

Access: Unreleased. Its weights are not openly released. The reference paper has 23 citations. Epoch AI rates the confidence of this record as speculative.

Full record

Organization: Peking University
Country of organization: China
Domain: Language
Task: Language modeling
Training compute: 2.8×10¹⁸ FLOP
Compute estimation method: Operation counting
Parameters: 149,697,024
Dataset size: 103M
Training hardware: NVIDIA V100
Chips used: 8
Training power draw: 4.8 kW
Model accessibility: Unreleased
Open weights: No
Citations: 23
Epoch confidence: Speculative