Dwarkesh Podcast

Reiner Pope – Chip design from the bottom up

Dwarkesh Podcast·May 22, 2026

OVERVIEW

This episode provides a deep dive into the design of AI chips, starting from basic logic gates and explaining how they perform core operations like matrix multiplication through multiply-accumulate units. It contrasts traditional CPU/GPU architectures with modern AI accelerators like TPUs, focusing on the efficiency gains achieved by integrating specific operations into hardware, particularly using systolic arrays. The discussion also covers the underlying principles of clock cycles, pipelining, and the strategic design choices made to optimize for compute versus communication costs.

KEY TOPICS

  • Fundamental units of chip design: Logic gates (AND, OR, NOT).
  • Core operation of AI chips: Matrix multiplication and multiply-accumulate.
  • Binary arithmetic: Long multiplication, full adders (3-to-2 compressors).
  • Wallace tree multiplier architecture for summation.
  • Data movement costs in CPU/GPU (register files, ALUs, multiplexers).
  • Architectural shift to systolic arrays/Tensor Cores in AI chips.
  • Compute-to-communication tradeoff.
  • Floating-point precision and bit-width choices (e.g., FP4 vs. FP8).
  • Clock cycles, pipelining, and synchronization in chip design.
  • Pipelining registers (flip-flops) to increase clock speed at cost of area.
  • FPGA vs. ASIC design choices and their business implications.
  • Look-up tables (LUTs) as programmable gates in FPGAs.
  • Non-deterministic latency in CPUs (cache systems).
  • Deterministic latency in TPUs (scratchpads).
  • Differences in high-level organization between GPUs and TPUs.

MAIN TAKEAWAYS

  • The fundamental primitive for AI chips is the multiply-accumulate unit, crucial for efficient matrix multiplication, which is central to neural network operations.
  • Designing chips for lower precision (e.g., FP4) offers significant area and power efficiency due to quadratic scaling of components with bit-width, making it highly advantageous for AI workloads.
  • The cost of moving data between memory (register files) and processing units often far outweighs the cost of the computation itself, driving the design towards architectures that reduce data movement, such as systolic arrays.
  • Clock cycles synchronize parallel operations on a chip, and optimizing clock speed involves strategically inserting pipeline registers to break down complex logic paths.
  • FPGAs offer flexibility and lower initial costs compared to ASICs, making them suitable for rapidly evolving workloads or lower volume production, while ASICs provide superior performance and efficiency at high volumes due to fixed, optimized hardware.
  • Non-deterministic latency in CPUs primarily stems from the cache hierarchy, where variable access times to memory introduce unpredictable delays, whereas TPUs use scratchpads and software-managed memory to achieve deterministic latency.
  • The high-level organization of GPUs emphasizes a large number of smaller, highly parallel streaming multiprocessors (SMs), while TPUs opt for fewer, larger, and more specialized matrix units, balancing fine-grained parallelism with reduced inter-unit communication overhead for specific AI tasks.

NOTABLE QUOTES

"The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply-accumulate."
"The cost of moving data from the register file to the ALU and back is many, many times more expensive than the logic unit itself."
"The single reason why low precision arithmetic has worked so well for neural nets" is the quadratic scaling with bit-width.
"You can actually design a CPU that has deterministic latency as well... The challenge is getting deterministic latency and high speed at the same time."
"The purpose of the branch predictor is like genuinely to predict, based on, like, before you even get to this instruction, to be like, five cycles earlier to predict there was going to be a branch that's going to happen."

Summarized with DriftNote — AI-powered podcast summaries

Try it free