Fusion in Encoder

Training compute

1.3×10²⁰ FLOP

Parameters

330M

Published

Nov 18, 2022

Fusion in Encoder is an AI model developed by Samsung (South Korea), first published in November 2022. It works in the language domain, on tasks such as question answering and language modeling/generation.

Training it took an estimated 1.3×10²⁰ FLOP of compute (estimation method: hardware). The model has 330,000,000 parameters. It was trained on roughly 960K datapoints. Training ran on NVIDIA A100 SXM4 80 GB for about 48 hours. The compute alone is estimated at $233 in 2023 dollars.

Access: Unreleased. Its weights are not openly released. The reference paper has 12 citations. Epoch AI rates the confidence of this record as likely.

Full record

Organization: Samsung
Country of organization: South Korea
Domain: Language
Task: Question answering, Language modeling/generation
Training compute: 1.3×10²⁰ FLOP
Compute estimation method: Hardware
Parameters: 330,000,000
Dataset size: 960K
Training hardware: NVIDIA A100 SXM4 80 GB
Training time: 48 h
Training cost (2023 USD): $233
Model accessibility: Unreleased
Open weights: No
Citations: 12
Epoch confidence: Likely