MOONSHOT AI / LONG CONTEXT / 2025

Kimi-Linear 48B-A3B

TOTAL48Bparameters

ACTIVE3Bper token

DEPTH27repeated blocks

HIDDEN2,304main trunk

CONTEXT1Mnative tokens

CHECKPOINT98.25GB20 shards · MIT

FAMILY EVOLUTION

Kimi LinearKDAKimi K21T MoEK2.5native visionK2.7code agent

INTERACTIVE SYSTEM BLUEPRINT

MODEL MRI

100K INPUT + 1K OUTPUT · 1.34P arithmetic FLOPs · click any module for evidence

VERIFIED ADAPTER

INPUT100.0KTEXT

TRANSFORMER STACK27 HYBRID LAYERS20 KDA · 7 global MLA · full at L4/8/12/16/20/24/27

pre-norm additive residual · recurrent state stays outside KV growth

OUTPUT1.0KTOKENS

wraps every repeated block

KDA ⇄ MLA · KIMI DELTA ATTENTIONKDA 不为每个历史 token 保存完整 KV，而是持续更新固定大小状态；长上下文内存主要来自少数全局 MLA 层。

Kimi MoE · 48B CAPACITY3B active path · total capacity ≠ active compute

DEPTH2720 KDA + 7 MLA

HIDDEN2,30432 heads

KDA32 × 128finite state

EXPERTS256top-8 + 1

CONTEXT1Mtokens

TRAIN5.7Ttokens