main
official NVIDIA Nemotron 3 Super architecture docs + 50 BF16 shards
NVIDIA / HYBRID MAMBA MOE / 2026

Nemotron 3 Super 120B-A12B

TOTAL123.6Bparameters
ACTIVE12.7Bper token
DEPTH88 + 2repeated blocks
HIDDEN4,096main trunk
CONTEXT1Mnative tokens
CHECKPOINT247.25GB50 shards · NVIDIA Open Model
INTERACTIVE SYSTEM BLUEPRINT

MODEL MRI

Nemotron 3 Super 120B-A12B · Architecture Explorer

100K INPUT + 1K OUTPUT · 3.50P arithmetic FLOPs · click any module for evidence

VERIFIED ADAPTER
INPUT100.0KTEXT
TRANSFORMER STACK88 HYBRID LAYERSMamba state layers interleaved with global attention
Pre-Norm Residual
++
OUTPUT1.0KTOKENS
wraps every repeated block
MAMBA ⇄ GQA · LATENTMOE每个 expert 在低维 latent 空间工作，以更多激活专家换取组合容量。
Nemotron MoE · 123.6B CAPACITY12.7B active · total capacity ≠ active compute
DEPTH88 + 2main + MTP
HIDDEN4,096trunk
EXPERTS512top-22
LATENT1,024expert dim
CONTEXT1Mtokens
ACTIVE12.7Bper token