① Computing density is extraordinarily high

In the latest GPUs and AI accelerators:

・Tens of billions to a hundred billion transistors

・Power consumption in the hundreds of watts

→ A chip alone cannot make it work.

② Memory bandwidth determines performance

AI processing handles:

・Large amounts of data

・Instantaneously

・In parallel

→ If DRAM is placed far away, performance cannot be achieved.

③ Simultaneous optimization of power, signals, and heat is required

・Unstable power supply → malfunctions

・Disturbed signals → errors

・Heat cannot escape → clock speeds drop

These are problems that can only be solved through packaging.

Packages for AI chips must simultaneously deliver:

・An ultra-high number of I/Os

・Ultra-high-speed interfaces

・Current supply on the order of hundreds of amperes

・Heat dissipation capacity of hundreds of watts

・High reliability

This cannot be achieved simply by extending conventional SoCs.

NVIDIA

・Integrates GPU + HBM in a 2.5D package

・Makes full use of CoWoS

・Adopts large interposers

Performance is largely determined by packaging.

AMD

・Introduced chiplet architecture early

・Separates compute, I/O, and memory

・Connects with HBM through 2.5D

Emphasizes the balance between yield and performance.

Apple

・Uses advanced fan-out in mobile SoCs

・Emphasizes high density and low power consumption

・Co-designs the package and SoC

World-class power efficiency.

By placing HBM close by:

・Bandwidth improves by orders of magnitude

・Power consumption decreases

・Latency is shortened

In AI computing, not being able to use HBM = not being able to compete.

In AI chips:

・Hundreds of amperes flow instantaneously

・Extremely sensitive to voltage fluctuations

Countermeasures:

・Strengthening power planes

・Optimizing decoupling

・Backside power delivery

・More advanced PI design

AI chips have:

・High heat density

・Local hot spots

Conventional heat spreaders have reached their limits.

Direction of evolution:

・Vapor chambers

・Liquid cooling

・Cooling directly attached to the die

The package is becoming a cooling device.

AI chips involve:

・Ultra-large dies

・Multiple HBM stacks

・Advanced packaging

→ Several hundred thousand yen or more per package.

・KGD management

・Package yield

・No rework possible

The difficulty of volume production is extremely high.

・Even higher bandwidth from HBM4 onward

・Integration with optical I/O

・Full-scale introduction of 3D stacking

・Integrated design of cooling and power

Competition in AI performance will shift completely to competition in packaging technology.

・Packaging determines AI chip performance

・Placing HBM close by is an essential condition

・Simultaneous optimization of power, heat, and signals is key

・Barriers to entry in advanced packaging are extremely high

・The winners will be the companies that master packaging technology

1.Why is HBM indispensable for AI chips?

2.Why is power design particularly difficult for AI chips?

3.Why has packaging technology become the main battleground in AI chip competition?

 

Column Supervisor: Koji Kakumoto (President, OTIS Co., Ltd.)

After studying abroad and working in planning and development at a trading company, he joined OTIS Co., Ltd. in 2011. While primarily working in the Corporate Planning Department, he has also served concurrently in manufacturing and technical divisions, and since 2018 has served as President, working to drive business growth and strengthen the organization.

 

This article is a general technical explanation intended for educational purposes and does not refer to any specific company, product, or technology.