blog

systems, hardware, and ML at scale

Browse by series, tag, or year

Series

Tags

amd 1 architecture 1 attention 3 compilers 1 distributed-systems 1 gpu 1 gpu-kernels 1 llm 4 llm-agents 1 ml-infra 1 ml-systems 5 model-architecture 3 moe 3 reinforcement-learning 1

Archive

  • ML Systems

    Muse Spark Open and Muse Glimmer, Layer by Layer: One Skeleton, a 30B Dense Model and an 816B Latent MoE

    October 27, 2026 · 105 min read

    A source-backed tour of Meta's two open-weight Muse models: Muse Glimmer 30B (dense, 52 layers) and Muse Spark Open (816B total, 42B active, 62 layers). Both use gated grouped-query attention with a scaleless QK-norm, three sliding-window RoPE layers per global NoPE layer, sandwich norms and a soft-capped head; Spark adds a 4,096-wide latent MoE with 256 experts, norm-preserving residual gates, an output norm in attention and a convolutional audio encoder. With five interactive 3D models and seven animated figures.

    llm model-architecture ml-systems moe attention

  • ML Systems

    CAKE from Scratch: Agents Write GPU Kernels, and the Compiler Learns from Their Mistakes

    October 10, 2026 · 65 min read

    A textbook-style tour of NVIDIA and CMU's CAKE (arXiv 2608.12629) for readers who have never written a GPU kernel: what kernels and expert schedules are, how kernel agents work today, CAKE IR and its pre-compile checks, a compiler that evolves from agent failures, and the evidence (1.144× vs 0.928× in a controlled Flash-KMeans study, a 2.05× KDA prefill, 10 of 11 expert kernels matched or beaten), with nine animated figures.

    ml-systems gpu-kernels compilers llm-agents

  • ML Systems

    DeepSeek-V4 and V4-Flash, Layer by Layer: Compressing Attention to Reach a Million Tokens

    October 5, 2026 · 90 min read

    A source-backed tour of DeepSeek-V4-Pro (1.6T, 49B active, 61 layers) and V4-Flash (284B, 13B active, 43 layers): a learned KV compressor, Compressed Sparse Attention and Heavily Compressed Attention over a 128-token window, a 512-wide shared-KV MQA core with a grouped output projection, four mHC residual streams, 384/256-expert MoE with hash routing and FP4 experts, and what changed in the 0731/0813 releases and V4.1-Flash — with eight interactive 3D models and six animated figures.

    llm model-architecture ml-systems moe attention

  • ML Systems

    GLM-5.3 and GLM-5.3-Flash, Layer by Layer: Same Skeleton, New Skills — and a Brand-New Hybrid

    October 4, 2026 · 80 min read

    A source-backed tour of Z.ai's GLM-5.3 (744B, 78 layers of MLA + DeepSeek Sparse Attention with IndexShare) and GLM-5.3-Flash (320B, a 45-layer KDA + DSA hybrid with NoPE attention, a pooled lightning indexer, and mHC residual streams), with seven interactive 3D models and six animated figures.

    llm model-architecture ml-systems moe attention

  • ML Systems

    Miles v0.1 Deep Dive: From Train–Rollout Mismatch to Trillion-Parameter Agentic RL

    September 29, 2026 · 48 min read

    A source-backed deep dive into Miles v0.1: mismatch control with TITO, R3, and TIS; asynchronous staleness; P2P and disk-delta transport; and on-policy distillation.

    ml-systems reinforcement-learning llm distributed-systems

  • GPU Architecture

    AMD MI350 GPU Architecture Deep Dive

    April 1, 2026 · 7 min read

    A deep dive into AMD MI350 (CDNA 4) — architecture, programming model, hardware features, and what sets it apart from NVIDIA.

    gpu amd architecture ml-infra