Architectural Deep Dive into Kimi K3 and Kimi Delta Attention
SemiAnalysis published a technical deep dive into the architectural design of the Kimi K3 open frontier model. To optimize memory and computation, Kimi K3 introduces a hybrid attention mechanism featuring Kimi Delta Attention (KDA). By eliminating the softmax operation, KDA converts quadratic computational complexity into linear complexity. This enables past key and value vectors to be compressed into a hidden state matrix functioning as associative memory. Additional mechanisms in the model include compressed memory, attention across depth, and latent expert routing. (source: https://newsletter.semianalysis.com/p/kimi-k3-the-manos-the-mythos-the)