■ What Generates Heat in AI Servers?
The GPU attracts the most attention as a heat source in AI servers.
However, the GPU is not the only thing generating heat.
Inside a server there are many heat sources, such as
● GPUs
● CPUs
● HBM
● Power supply circuits
● VRMs
● Network ICs
● Optical communication components
● SSDs
● Fans
In other words, an AI server is not a single huge heat source, but
a device in which many heat sources are gathered in a narrow space.
This difference is important.
Even if only the GPU is sufficiently cooled, the server as a whole will not operate stably if the power supply or network components become hot.
■ Why Do GPUs Generate So Much Heat?
AI repeats huge numbers of matrix operations.
That is why an enormous number of computations take place inside the GPU.
As the amount of processing increases, power consumption also increases.
Using power generates heat.
Furthermore, as semiconductors become higher in performance,
more circuits are integrated into a smaller area.
As a result,
not just simple heat generation, but heat generation density becomes higher.
This is a major difficulty in cooling AI semiconductors.
■ Heat Generation and Heat Generation Density Are Different
For example, even with the same 500 W of heat,
the difficulty of cooling differs greatly between 500 W generated across a large metal plate
and 500 W generated from a small semiconductor chip.
When heat concentrates in a small area, temperatures rise locally.
This heat must be moved in a short time to
● TIM
● Heat spreaders
● Vapor chambers
● Cold plates
In other words, in the heat problems of AI servers, it is necessary to look not only at
how many watts are generated, but at
from how many mm² that heat comes.
■ HBM Also Becomes Part of the Heat Problem
In AI semiconductors, not only the GPU but also HBM becomes important.
AI computation requires large amounts of data to be supplied to the GPU at high speed.
That is why HBM is placed near the GPU.
This is advantageous for performance.
Thermally, however, it means placing another heat source next to a large heat source.
Heat from the GPU.
Heat from the HBM.
Heat from peripheral circuits.
These affect each other within a narrow package.
In other words, in AI semiconductors, the thinking is shifting from
cooling a single chip
to
managing the temperature distribution of the entire package.
■ Hot Spots Matter More Than Average Temperature
In AI semiconductors, the entire chip does not necessarily generate heat uniformly.
Load concentrates on specific computing units.
Memory access concentrates.
Load is applied to power circuits.
Then hot spots occur.
For example, even if the average temperature of the entire chip is within the allowable range, only part of it may become very hot.
This local high temperature leads to
● Reduced operating frequency
● Reduced reliability
● Material degradation
● Joint degradation
In other words, in AI servers, being cool on average is not enough.
How to cool the hottest spot is what matters.
■ When It Gets Too Hot, It Lowers Performance to Protect Itself
When semiconductors get too hot, they sometimes perform control that lowers performance to prevent failure.
Lowering the clock.
Limiting power.
Slowing processing speed.
In other words,
insufficient cooling limits computing performance itself.
Even with a high-performance GPU installed, its performance cannot continue to be used if the heat cannot be handled.
From this arises the idea that cooling capacity is part of computing capacity.
■ The Heat Path in AI Servers
To understand the heat problems of AI servers,
it is necessary to see where heat moves from and to.
For example, in the case of a GPU, there is a long heat path:
GPU interior
↓
Package
↓
TIM
↓
Heat spreader
↓
TIM
↓
Heat sink or cold plate
↓
Air or coolant
↓
Outside the data center
If thermal resistance is large at even one point, that point becomes the bottleneck for cooling performance.
■ Even the Highest-Performance Cooling Equipment Is Meaningless If Heat Does Not Reach It
For example, suppose a very high-performance liquid cooling cold plate is used.
The coolant temperature is sufficiently low.
The flow rate is also sufficient.
But the TIM between the GPU and the cold plate is lifting.
There are air bubbles.
Pressure is insufficient.
Then
heat does not reach the cold plate.
In other words, the liquid cooling equipment has spare capacity, yet only the GPU becomes hot.
In AI servers, such interface problems are extremely important.
■ TIM Is a Small but Important Component of AI Servers
TIM is very thin.
Seen from the cooling equipment as a whole, it is an inconspicuous component.
But its role is large.
The surfaces of semiconductors and heat spreaders have fine irregularities even if they look flat.
When solids are put in contact as they are, tiny air layers remain between them.
TIM is used to fill that space and increase the actual contact area.
Because heat generation density is high in AI semiconductors,
the TIM’s
● Thermal resistance
● Thickness
● Compression ratio
● Conformability to surfaces
● Long-term stability
are important.
■ TIM Cannot Be Chosen on High Thermal Conductivity Alone
Using a TIM with high thermal conductivity makes it cool.
It is not that simple.
In practice,
● It is too thick
● It is too hard to conform to the surface
● Assembly pressure is insufficient
● Air bubbles get in
● It pumps out
● Its condition changes with long-term use
may occur.
In other words, TIM must be evaluated not only by material specifications, but
in the state actually incorporated.
■ In AI Servers Too, a 0.1 mm Lift Becomes a Problem
Even if cooling components become larger, contact with the heat source is made at a narrow interface.
TIM.
Insulating materials.
Adhesives.
If a 0.1 mm lift forms between them, air gets in.
The air layer increases local thermal resistance.
As a result,
a 0.1 mm gap may ruin the performance of cooling equipment tens of centimeters in size.
This is why the last 1 mm of heat matters.
■ Spreading Heat Sideways with Graphite
In AI semiconductors, the idea of not only releasing heat upward but also spreading it sideways is important.
Graphite has high thermal conductivity in the in-plane direction.
It can be used to spread local heat to
● Surrounding metal components
● Enclosures
● Cooling components
For example, it is the idea of
Local heat around the GPU
↓
Graphite
→→→ Spreading to the surroundings
However, it is meaningless if there is no heat discharge destination where the heat is spread.
■ Copper and Aluminum Are Also Used for Heat Spreading
Copper and aluminum are important materials in thermal management for AI servers.
Copper in particular has high thermal conductivity and is used in
● Heat spreaders
● Heat sinks
● Metal foils
● Cold plates
On the other hand, metals also conduct electricity.
That is why, in places where you want heat to pass but not electricity, insulation design is needed.
■ AI Servers Need Insulation and Heat Dissipation at the Same Time
In AI servers handling high power,
● GPUs
● VRMs
● Power supply circuits
● Metal cooling components
are close together.
That is why
thermal conduction,
insulation,
and fixing
must be achieved at the same time.
For example, a structure such as
Heat source
↓
Insulating TIM
↓
Metal component
can be considered.
However, making the insulating layer thicker may increase thermal resistance.
Making it thinner leaves no margin in insulation reliability.
In other words, in AI servers, achieving both heat dissipation performance and insulation performance is important.
■ Air Cooling Requires Large Amounts of Air
In conventional servers, air cooling with fans was central.
Air cooling is also important in AI servers.
However, as heat generation increases, it becomes necessary to
● Raise fan speed
● Add fans
● Enlarge heat sinks
● Widen airflow paths
As a result, the burden of
● Power consumption
● Noise
● Size
● Maintenance
increases.
In other words, the further the density of AI servers advances, the greater the price of making it work with air cooling alone.
■ Why Liquid Cooling Becomes Important in AI Servers
That is where liquid cooling becomes important.
Liquids carry more heat more easily than air.
Furthermore, if the cold plate is placed near the GPU,
heat can be recovered near the heat source.
A typical structure is
GPU
↓
TIM
↓
Cold plate
↓
Coolant
Liquid cooling is a good match for the AI server characteristics of high heat generation and high density.
■ Liquid Cooling Brings New Component Challenges
Switching to liquid cooling does not solve everything.
New challenges arise:
● Sealing
● Gaskets
● Insulation
● Leakage
● Corrosion
● Piping
● Coolant compatibility
In other words, the shift of AI servers to liquid cooling does not eliminate the market for thermal management components.
Rather, it may increase the importance of composite components: thermally conductive materials + insulating materials + seal materials.
■ Seal Components Become Important in Liquid Cooling
Cold plates.
Piping.
Fittings.
These require structures that do not let liquid leak.
If seal materials or gaskets
● Shift
● Differ in thickness
● Trap foreign matter
● Are insufficiently compressed
the risk of leakage increases.
In other words, in liquid-cooled AI servers, not only accuracy in passing heat, but also accuracy in stopping liquid is important.
■ Vapor Chambers Also Become Important
Vapor chambers spread local heat over a wide area.
For small, high-heat sources such as AI semiconductors,
Heat source
↓
TIM
↓
Vapor chamber
↓
Heat sink
makes it possible to make effective use of the cooling area.
In other words, in the heat problems of AI servers, multiple technologies are combined:
● Receiving heat with TIM
● Spreading it with vapor chambers
● Discarding it with air or liquid cooling
■ 3D Integration Makes the Heat Problems of AI Servers Even Harder
In AI semiconductors, high-density integration such as HBM and chiplets is advancing.
If 3D integration advances further in the future, heat sources move inside.
Even if the external cooling equipment is high-performance, it cannot cool if heat cannot be extracted from the inside to the surface.
That is why
● Thin TIM
● Heat-spreading materials
● High thermal conductivity layers
● Fine insulating materials
become important.
■ Summary
As we have seen, the heat problems of AI servers are not just about a single semiconductor, the GPU.
Not only heat generation but heat generation density is high.
High-heat components such as HBM are placed close together.
Not the average temperature, but internal hot spots govern performance.
These create the difficulties unique to AI semiconductors.
And to get that heat out, a heat path combining multiple technologies is needed: receiving it with TIM, spreading it with graphite and metals, and carrying it with vapor chambers or liquid cooling as needed.
No matter how high-performance the cooling equipment, if the TIM is lifting even slightly, heat stops before it.
Furthermore, as 3D integration advances, the heat sources themselves may move inside the package, beyond what external cooling equipment alone can handle.
In Part 2, after looking at the many heat sources other than the GPU, the energy problem of the data center as a whole, and the volume production and automation challenges of reproducing this heat path on a scale of tens of thousands of units, we describe what role OTIS plays in this area.
Column Supervisor: Koji Kakumoto (President, OTIS Co., Ltd.)
After studying abroad and working in planning and development at a trading company, he joined OTIS Co., Ltd. in 2011. While primarily working in the Corporate Planning Department, he has also served concurrently in manufacturing and technical divisions, and since 2018 has served as President, working to drive business growth and strengthen the organization.
This article is a general technical explanation intended for educational purposes and does not refer to any specific company, product, or technology.
