What is HBM? Is it made of gold? Why is it so expensive?

The board that Jensen Huang is holding up on the GTC stage is the heart of today’s computing power. But very few people know that the most expensive components on this board are not the GPU itself, but those few inconspicuous little black bricks right next to the GPU.

Figure 1: GTC现场，GB300主板。The protagonist of the story is on this board.

It’s called HBM—High Bandwidth Memory, high-bandwidth video memory.

This article tells where it is, how it connects, why it’s indispensable, and how it’s made.

I. First find it: where did the video memory go

First, look at an “anatomy photo” of a typical graphics card. RTX 5090, the GPU (GB202) is in the center, surrounded by a whole ring of black small chips—those are 16 pieces of GDDR7 video memory. They “live outside” the GPU, with only a few centimeters of PCB routing in between, communicating with the GPU through traces.

Figure 2: RTX 5090 PCB real shot. The blue box is the GPU, and the green box contains the 16 GDDR7 chips—traditional solution, where the video memory is outside the package.

Now look at the AI chip Blackwell. Something strange happens: there isn’t a single stick of video memory “outside.” Because the video memory has moved into the package itself—lift the metal lid, and snug against the outer sides of the two GPU dies are 8 stacks of HBM; the distance shrinks from centimeters down to millimeters.

Figure 3: Inside the Blackwell package. The copper frame is 8 stacks of HBM3E, the blue frame is the two GPU dies. The video memory goes from “outside neighbor” to “roommate.”

This isn’t just a simple relocation. Inside the GPU, they specially built 8 HBM controllers to talk to it; 8 stacks of HBM3E total 288GB capacity and 8TB/s bandwidth—these two numbers are the lifeline of AI chips.

Figure 4: Blackwell Ultra official architecture diagram. The green box is 8 HBM controllers, and the gold frame is the official parameters: 288GB HBM3E, 8 stacks, up to 8TB/s.

The next-generation Rubin will carry even more. You could say: HBM is already half of the (cost) equation for AI chips.

Figure 5: Vera Rubin motherboard, two Rubin GPU packages (HBM4 is under the lid).

II. Why the GPU “is hungry”: chef and warehouse

To understand why HBM exists, you first need to understand the GPU’s pain points. If we imagine the GPU as a chef, then video memory is the warehouse, and the traces are the food-serving corridors. This chef can stir-fry ten thousand dishes every second—but if the dishes can’t be transported in, then no matter how many arms the chef has, there’s nothing to do.

Figure 6: GPU = chef, video memory = warehouse, traces = food-serving corridors. In the AI era, “the dishes” are hundreds of billions of model parameters.

For AI foundation models, the “dishes” are hundreds of billions of parameters, and every compute step requires moving them back and forth from video memory. So the deciding factor for a GPU shifts from “how fast you can compute” to “how fast you can feed.” The ability to feed data is bandwidth.

III. Bandwidth has only two roads: faster, or wider

Data travels along traces driven by voltage: voltage high is 1, voltage low is 0, switching billions of times per second. If you want to transmit faster, there are only two ways.

The first way is to raise the frequency—make voltage levels switch faster. GDDR7 is already so fast that one trace switches 28 billion times per second. But physical laws start charging: too fast and the signal distorts, and neighboring traces interfere with each other—like people shouting side by side, getting faster and faster until nobody can hear clearly.

The second way is to increase the channel width—don’t just chase speed, chase more lanes, more lanes, more lanes.

Figure 7: GDDR7 is like 32-lane back roads, while HBM is a highway with 1024 lanes.

That’s the only formula in this whole piece:

Figure 8: Bandwidth = frequency × channel width. The speed of each lane, multiplied by the number of lanes.

HBM chose the “wider” road: one stack has 1024 data lines—32 times that of a single GDDR7 chip. One GPU is paired with 8 stacks, for a total of 8192 lanes. It doesn’t matter if each lane is 3x slower—having 32x more lanes more than makes up for it. Each stack delivers 1.2TB/s, which is equivalent to transmitting 300 full-length movies per second.

IV. The disaster of 30,000 lines

Sounds perfect? For engineers, though, it’s a disaster. The 1024 data lines are just the beginning. You still need power lines, address lines, and clock lines—about 4000 lines to fan out from one stack, and with 8 stacks that becomes nearly 30,000 lines.

What does 30,000 lines mean? You have far fewer wires than that if you add up every wire in an entire building at home. And they all have to be squeezed into a package the size of a credit card.

Figure 9: Number of interconnect wires in a building < within a single package. The question then becomes: where do these lines get routed?

V. PCB limits: its “core” is a piece of cloth

First, look at how traditional PCBs “draw” traces. Many people don’t know that a PCB’s base is actually a piece of cloth—glass fiber woven fabric, soaked in epoxy resin, and pressed with a full sheet of copper foil.

Here, traces aren’t built up trace by trace like wires. Instead, they’re printed. In manufacturing: apply a photosensitive film, shine light, wash with chemicals—then the exposed copper areas are etched away, leaving behind the trace pattern.

But you have to know the limit of this process is line widths of only dozens of micrometers.

Figure 10: Glass fiber cloth + copper foil + photochemical etching. But the limit of this process is trace widths of dozens of micrometers—about half the width of a human hair. For 30,000 lines, that’s still too thick.

VI. Switching to a different “paper”: silicon interposer

But the limit of this process is trace widths of dozens of micrometers—about half the width of a human hair—still too thick for 30,000 lines. It simply can’t fit.

So what do you do? Replace the “paper” where the wiring is drawn—switch to silicon. On silicon wafers, you can use lithography to draw traces (which is why lithography tools are so important), down to under 1 micrometer—one-hundredth the width of a human hair. With the same area, you can fit hundreds of times more wiring.

This silicon specifically used to lay out wiring is called a silicon interposer. Both the GPU and HBM sit on it, and all 30,000 lines are routed inside this piece of silicon. The chips aren’t stacked directly on each other, but they share the same silicon—this packaging form is called 2.5D in the industry.

Figure 11: The GPU and HBM sit side by side on the silicon interposer; underneath is lithographically routed wiring as dense as hair strands. This is the foundation of the “roommate relationship” shown in Figure 3.

VII. Capacity: when the ground isn’t enough, build upward

With the lane problem solved, there’s a second problem: capacity. The 5090 uses 16 pieces of 2GB GDDR7, for a total of only 32GB. But for a big model, just the parameters alone can require hundreds of GB—off by an order of magnitude. Spread them out? You can’t fit that on the interposer.

Like big cities, when land is insufficient, you build upward—stack DRAM chips into a 12-story building.

Figure 12: DRAM die stacked layer by layer, with capacity multiplying. But then comes a new problem: how do you route “water and electricity” between floors?

VIII. TSV: elevator shafts inside the building

For such tall buildings, how do the different floors communicate? The answer is TSV (Through-Silicon Via). Drill through thousands of vertical “elevator shafts” on each layer of the chip. How fine are they? A diameter of 5 micrometers—one fourteenth of a human hair. With holes that tiny, there’s no drill bit in the world that can drill them; you can only “chew” them using plasma (a.k.a. chemical reagents).

But plasma isn’t obedient: as it chews downward, it also chews sideways, causing the hole to become a spherical cavity. Engineers’ solution is the Bosch process: a three-step loop you can remember as “chew, coat, and smash”—

1. Chew (etch): plasma chews a small hole downward;

2. Coat (protection): spray a layer of Teflon, like waterproofing the inner wall of the shaft;

3. Smash (ion bombardment): the vertical ion “rain” breaks open the insulation at the bottom so the next cycle can only chew downward.

4. Repeat this, layer by layer chewing—coating (protective film)—and punching through until reaching the target position

Figure 13: After hundreds of cycles of chew-coat-smash, a perfectly straight deep shaft is dug out. The scalloped ridges on the shaft wall are the tooth marks left by the repeated cycles.

After the shaft is made: first deposit a layer of insulating film to separate copper and silicon via vapor deposition; then submerge the whole wafer into a copper sulfate solution and electroplate to fill the vias with copper. Finally, grind away about 90% of the wafer from the backside, leaving only 30 micrometers—about one-third the thickness of a sheet of printer paper. The copper pillars become exposed from the backside: the communication between floors is finally connected.

Figure 14: Backside thinning to 30 micrometers (right side: comparison to printer paper thickness). Each layer of chip has to go through the same process.

IX. Two schools of “building the tower”

Now weld the 12-story “floor slabs” together. Between layers, you connect them with tiny solder balls—each smaller than a pollen grain; you need thousands of them per layer, and none of them can be misaligned. As for how to solder and stack them, the industry splits into two camps:

After the whole stack is welded, you fill it with epoxy resin—for better heat dissipation and higher yield; or you do it layer by layer with film deposition plus pressing.

Figure 15: Left, SK Hynix—weld the entire stack first, then pour in an epoxy “concrete” batch (MR-MUF); right, Samsung/Micron—lay a layer of adhesive film, press a layer of floor slab (TC-NCF).

Hynix’s “concrete” has clearly better thermal conductivity. The taller the building and the more concentrated the heat, the more valuable the thermal conductivity of the interlayer material becomes. Just with this one batch of adhesive, Hynix won a major order from NVIDIA and took the top chair in the HBM market.

X. Finished product: what you pay the bill for

Stack 12 layers: one building is 36GB. Wrap 8 buildings around the GPU: one chip is 288GB. With 4 chips on one board, you reach 1152GB—over 1TB—back to the board in Figure 1 that Jensen Huang is holding.

What’s the cost? To make 1GB of HBM, you need to consume about 3GB worth of normal memory wafer production capacity: HBM chips are larger (TSVs take up area), the stacking yield compounds into a discount, and the process also occupies production lines. Factories all over the world are building AI buildings, so standard memory modules run out of feed.

Figure 16: With the same factory capacity, 1GB HBM ≈ 3GB DDR5. This is the complete mechanism behind this round of memory price increases.

So when you buy memory sticks and pay extra, you’re basically paying rent for AI.