main
official Meta Llama 3.1 model card and 405B config
META / DENSE BASELINE / 2024

Llama 3.1 405B

TOTAL405Bparameters
ACTIVE405Bper token
DEPTH126repeated blocks
HIDDEN16,384main trunk
CONTEXT128Knative tokens
CHECKPOINT≈810GB191 shards · Llama 3.1 Community
INTERACTIVE SYSTEM BLUEPRINT

MODEL MRI

Llama 3.1 405B · Architecture Explorer

100K INPUT + 1K OUTPUT · 123.93P arithmetic FLOPs · click any module for evidence

VERIFIED ADAPTER
INPUT100.0KTEXT
TRANSFORMER STACK126 DENSE BLOCKSGQA + SwiGLU · every parameter active
RMSNorm Residual
++
OUTPUT1.0KTOKENS
wraps every repeated block
GROUPED-QUERY ATTENTION · DENSE SCALE REFERENCE它是比较 MoE active compute 时最重要的纯 Dense 对照组。
SwiGLU FFN没有 router、专家稀疏或跳层；权重带宽压力直接随模型规模增长。
DEPTH126decoder
HIDDEN16,384main trunk
HEADS128 / 8Q / KV
INTERMEDIATE53,248SwiGLU
CONTEXT128Ktokens
ACTIVE405Bevery token