Google TPU v6e Trillium: Matrix Accelerators and Optical Circuit Switches

The Sixth Generation of Custom Silicon at Google

While the commercial AI industry remains heavily reliant on general-purpose GPUs, Google has pursued an independent silicon trajectory for over a decade. Beginning with the original TPU v1 deployment in 2015 for neural network inference, Google's internal accelerator roadmap has scaled through five iterative generations.

In May 2024 at Google I/O, Google unveiled TPU v6e, codenamed Trillium (*Google Cloud Next & I/O 2024 Technical Disclosures*). Designed explicitly to train and serve frontier multimodal models such as Gemini 1.5, Imagen 3, and AlphaFold 3, Trillium represents Google's most aggressive leap in compute density and energy efficiency to date.

Unlike general-purpose GPUs burdened by legacy graphics rasterizers, ray-tracing units, and display controllers, Trillium is a dedicated domain-specific Application-Specific Integrated Circuit (ASIC). Every square millimeter of silicon die area is dedicated to tensor contraction, vector mathematics, on-chip scratchpad SRAM, and high-speed inter-chip networking.

┌────────────────────────────────────────────────────────────────────────┐ │ Google TPU v6e Trillium Single-Chip Block Architecture │ ├────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Dual Matrix Multiply Units (MXUs) │ │ │ │ - 926 TFLOPS Peak BF16 / FP8 Compute Density │ │ │ │ - Microscaling Format (MXFP4) Native Tensor Hardware │ │ │ └──────────────────────────────┬───────────────────────────────┘ │ │ │ High-Bandwidth Local Bus │ │ ┌──────────────────────────────┼───────────────────────────────┐ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌──────────────────┐ ┌───────────────────┐ ┌─────────────────────┐ │ │ │ Vector Unit (VPU)│ │ 32 MB On-Chip │ │ High-Bandwidth │ │ │ │ - LayerNorm/SiLU │ │ Scratchpad SRAM │ │ Memory (32 GB HBM) │ │ │ │ - Softmax/RoPE │ │ - Low-latency DMA │ │ - 1,638 GB/s Band- │ │ │ └──────────────────┘ └───────────────────┘ │ width (2x v5e) │ │ │ └─────────────────────┘ │ │ ▲ │ │ │ Inter-Chip Interconnect (ICI) │ │ ┌──────────────────────────────┴───────────────────────────────┐ │ │ │ Palomar Optical Circuit Switch (OCS) Transceivers │ │ │ │ - Reconfigurable 3D Torus Dynamic Topology Steering │ │ │ └──────────────────────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────────────────────────┘

---

Microarchitecture: 4.7x Compute Density Uplift

Compared to its predecessor, TPU v5e (which was optimized for cost-effective inference and mid-scale training), TPU v6e Trillium delivers dramatic architectural improvements across every compute metric: - 4.7x Peak Compute Density: Trillium delivers up to 926 TFLOPS of peak 16-bit matrix multiplication performance per chip, representing a 4.7-fold increase in raw floating-point throughput per socket compared to TPU v5e. - Native Microscaling Support (MXFP4): In addition to standard IEEE FP16, Brain Floating Point (BF16), and FP8 (E4M3/E5M2), Trillium incorporates hardware acceleration for the OCP Microscaling Formats (MXFP4). By sharing an 8-bit scale factor across a block of 32 4-bit floating-point numbers, Trillium halves memory bandwidth requirements during weight streaming while preserving numerical training stability. - Doubled High-Bandwidth Memory (HBM): Memory capacity is increased to 32 GB of HBM, coupled with a 1,638 GB/s memory bandwidth—precisely double the bandwidth of TPU v5e ($819\text{ GB/s}$). This eliminates memory-bus stalls during large-batch inference and attention key-value cache lookups.

---

The Interconnect Fabric: Palomar Optical Circuit Switches (OCS)

The defining architectural differentiator of Google's TPU clusters is not merely the ASIC silicon, but the interconnect fabric. While conventional GPU clusters rely on multi-tier spine-and-leaf Ethernet or InfiniBand electrical packet switches (such as NVIDIA Quantum-2 switches), Google connects its TPUs using Optical Circuit Switches (OCS), codenamed Palomar.

An OCS switch does not convert photons into electrons to inspect packet headers. Instead, it routes light directly across freespace using a two-dimensional grid of micro-electro-mechanical system (MEMS) mirrors:

┌────────────────────────────────────────────────────────┐ │ Google Palomar Optical Circuit Switch (OCS) Operation │ ├────────────────────────────────────────────────────────┤ │ Optical Fiber Inputs (Wavelengths λ1..λN) │ │ │ │ │ ▼ │ │ [ 2D Piezoelectric MEMS Mirror Array (Input Port) ] │ │ │ (Tilt mirror angle in millisecond timeframes) │ │ ▼ │ │ [ Dynamic Free-Space Optical Reflection Path ] │ │ │ │ │ ▼ │ │ [ 2D Piezoelectric MEMS Mirror Array (Output Port) ] │ │ │ │ │ ▼ │ │ Optical Fiber Outputs (Zero Packet Buffering / 0 Drops)│ └────────────────────────────────────────────────────────┘

This optical architecture provides three decisive operational benefits: 1. Zero Packet Buffering and Drop Immunity: Photons pass through the mirrors without queuing, buffering, or serialization delays. Packet loss is physically impossible within the optical switch matrix. 2. Dynamic Topology Reconfiguration: The cluster's physical 3D torus topology can be reconfigured in software within milliseconds. If an engineering team switches from pipeline-parallel LLM pretraining to tensor-parallel mixture-of-experts (MoE) routing, the OCS tilts its mirrors to re-route bandwidth toward the active communication graph. 3. Automated Defect Bypass: If a chip or optical transceiver fails mid-training, the OCS reroutes optical paths around the defective slice in less than one second, eliminating the need to take down the entire pod or restart from a global checkpoint.

---

Pod-Level Scaling: From 256 Chips to Tens of Thousands

Trillium scales natively into pods of 256 chips interconnected in a high-bandwidth 3D torus topology via Google's proprietary Inter-Chip Interconnect (ICI).

Beyond a single pod, Google couples thousands of Trillium chips across datacenter campuses using Jupiter Datacenter Fabric, creating superclusters scaling to tens of thousands of interconnected ASICs: - Energy Efficiency: Trillium achieves over 67% higher energy efficiency per watt compared to TPU v5e, heavily reducing the datacenter thermal overhead discussed in our guide to Thermal Physics of AI Datacenters. - Direct-to-Chip Liquid Cooling: Each Trillium pod utilizes closed-loop liquid cold plates to maintain silicon junction temperatures below $65^\circ\text{C}$ during sustained matrix operations.

---

Software Integration: XLA, JAX, and the Pathways Runtime

Silicon is useless without a compiler capable of fusing arithmetic kernels. Google's custom silicon moat is powered by three deeply integrated software systems: - XLA (Accelerated Linear Algebra): An optimizing compiler that ingests high-level computation graphs from JAX and PyTorch, fusing linear transformations, activations, and normalization layers into monolithic machine-code kernels that execute entirely within Trillium's 32 MB on-chip scratchpad SRAM. - JAX: The preferred functional programming framework for Google DeepMind research, offering native automatic differentiation, vectorization (vmap), and parallel shard orchestration (pmap / shard_map). - Pathways: A distributed orchestrator that multiplexes single computational jobs across heterogeneous clusters of tens of thousands of TPU pods, dynamically balancing memory and compute boundaries.

Through TPU v6e Trillium, Google retains total vertical control over its compute stack—decoupling the training of frontier systems like Google Gemini from third-party merchant silicon margins.