Table of Contents
BIZON Z9000 – 8 TITAN RTX, 2080 Ti GPU deep learning server with liquid cooling. Review, benchmarks, noise level, temperatures
By Mark Stevens
June 3, 2019
water cooling
gpu server
benchmarks
AI / Deep learning
AI, Deep Learning
Benchmarks
Water-cooling
Most of you guys have seen tons of liquid-cooled builds from top YouTubers like Linus Tech Tips or JayzTwoCents and you may think nothing will shock you anymore and you have already seen the craziest liquid cooling builds ever.
Until now.
Today we will show you something that will blow your mind and this thing is sick!
We are introducing the world’s first dual Xeon RACKMOUNT SERVER with 8 x TITAN RTX GPUs and custom liquid cooling system!
And this is the craziest and most expensive liquid-cooled computer you have ever seen!
Before we start, let me tell you what brings us to this crazy project.
Most of the people may think that liquid cooling is something made for gamers only. It just looks great because of the look of the liquid. That’s why gamers like it.
It’s true when you are dealing with one or two GPUs. The amount of heat dissipated by the GPUs will be reasonable for air cooling.
But what about multi-GPU configurations? 4, 8 or more GPUs?
We have 4 GPU liquid-cooled desktop workstation. More details »
But today’s article is about 8-10 GPU rackmount server.
The main question people ask is why liquid cooling is so expensive and why I have to overpay for it. We are here to answer you.
There are three main benefits of liquid cooling.
1. Low noise.
2. Low thermals.
3. Higher performance.
The latest generation of NVIDIA RTX GPUs in multi GPU configurations dissipate so much heat that you need liquid cooling to keep GPU within optimal temperature range.
Blower fans solve the problem for RTX 2080 Ti, but there is no blower fan for TITAN RTX.
TITAN RTX offers the best performance at a reasonable price ($2500). TITAN RTX performance is comparable to NVIDIA Tesla V100 ($7000).
Sounds good right? Then why buy Tesla V100 for that crazy money instead of using TITAN RTX and build the fastest GPU server in the world for a reasonable price?
Let’s build the server with stock air-cooled TITAN RTXs and see what happens.
We are installing the GPUs, and technically they fit. But there is a huge problem we are going to show you.
NVIDIA TITAN RTX’s cooling system design blocks them from using multi-GPU configurations. As you can see in the photo, when you install one TITAN RTX next to another, the fans are completely blocked. No clearance between them. I am pretty sure the GPU will not be able to dissipate the heat.
Let’s run the benchmarks and see what happens.
Test Bench:
Air-cooled BIZON G7000 server:
4 x TITAN RTX
Liquid cooled BIZON Z9000 server:
GPUs: 4 x TITAN RTX + 4 x RTX 2080 Ti
Note: We will activate 4 TITAN RTX only when running the benchmarks to match the air cooling build.
Our server has two settings for the fan speed. Let’s start from normal fan speed.
Benchmark specs:
- Model: Resnet 50
- Tensorflow 1.13
- Data: Synthetic
- 1000 iterations
- Batch size: 64
- FP32
Temperature test:
Performance:
Time:
Max fan speed:
Temperature test:
Performance:
Noise level:
Results:
Temperature:
As I have expected after a few minutes, temperatures immediately go up to critical 85C-90C.
This is the “red line” for the GPUs. Once the GPUs reach that level, they automatically activate thermal throttling to protect itself from damage.
Thermal throttling:
When your cooling solution can no longer dissipate heat fast enough to keep temperatures within a safe range, your graphics card starts to dump performance to keep the temperatures low.
Performance:
On the normal fan speed we can see up to 5X lower performance due to overheating.
Even on maximum fan speed we can see up to 3X lower performance due to overheating.
Moreover, when the GPU is permanently working under high temperatures, it will damage the GPUs and significantly decrease the lifespan.
Noise level:
Noise level at normal fan speed is acceptable.
At max fan speed mode, we have seen 91 dB noise level and this is totally unacceptable. Noise level above 85 dB is harmful. You need to wear sound reduction ear muffs for construction workers to stand near the server. Even if you think to place the server in the server room, you will hear it from outside of the server room pretty clear and coworkers will complain.
Note:
Sound is measured in decibels (dB). A whisper is about 30 dB, normal conversation is about 60 dB, and a motorcycle engine running is about 95 dB. Noise above 85 dB over may start to damage your hearing. Loud noise above 120 dB can cause immediate harm to your ears.
The National Institute for Occupational Safety and Health (NIOSH) in the US recommends that the equivalent noise level that a worker is exposed to should be limited to 85 dB(A) for an eight-hour day to minimize the risk of hearing damage.
Source
The server comes with 8 x 8000 RPM server fans, and they are not enough to keep the TITANs cool. Because there is just no clearance between the GPUs. And that’s where liquid cooling became a must.
Despite those facts, we can see a few companies on the market selling TITAN RTX servers without liquid cooling.
Now let’s compare with liquid cooling.
Temperature test:
Performance:
Noise level:
Time:
Look at these temperatures!
46C-50C is just perfect! This is low for any GPU.
Noise level is within the normal range and we can talk with no problem.
We can even place this server in the office, lab or living room.
No thermal throttling. No overheating. 100% stable performance.
Performance is very stable (we tested for 48 hours).
LuxMark is an OpenCL cross-platform benchmark tool.
Score:
Rendering (Luxmark) – BIZON Z9000 Liquid cooled server
Temperature test:
Score:
4 x TITAN RTX + 4 x RTX 2080 Ti:
Temperature test:
Score:
Liquid cooled NVIDIA TITAN RTX:
- Idle noise level: up to 40 db
- Idle temperature: 27-29C
- Max load noise level (stress test): up to 50 db
- Max load temperature (GPUs): 45-55C
- Performance: 100%. No overheating –> No performance drop. 24/7 stable.
Air cooled NVIDIA TITAN RTX (normal fan speed):
- Idle noise level: up to 70 db
- Idle temperature: 40C
- Max load noise level (deep learning benchmark): up to 80 db
- Max load temperature (GPUs): 86-88C
- Performance (deep learning): 6X times lower (150 vs. 900)
- Performance (rendering / luxmark): 2X times lower (92000 vs. 190000)
Air cooled NVIDIA TITAN RTX (max fan speed):
- Idle noise level: 92 db (warning!)
- Idle temperature: 40C
- Max load noise level (deep learning benchmark): 92 db (warning!)
- Max load temperature (GPUs): 86-88C
- Performance (deep learning): 3X times lower (300 vs. 900)
- Performance (rendering / luxmark): 2X times lower (92000 vs. 190000)
Yes!
Temperature test:
Hashrate:
There are various uses for the 8-10 GPU server:
AI research, machine learning, various computation tasks, speech and image recognition, computer vision, rendering, or even gaming.
The benchmarks of BIZON Z9000 liquid cooled server with 8 TITANs are much better than we may expect.
Top of the line GPUs like TITAN RTXes are extremely hot and require liquid cooling in multi-GPU configurations.
Liquid cooling is the best solution for rapid heat removal due to its unmatched thermal performance. By using liquid cooling we get rid of the loud fans. The liquid cooling will provide 24/7 stability, low noise, and long-life for components.
Noise level of BIZON liquid cooling system is up to 50% lower versus air cooling: 50 dB vs. 90 dB
Temperatures are up to 50% lower as well: 50C vs 90C.
Performance is up to 3 times faster with liquid cooling versus air cooling. GPUs do not overheat and do not activate thermal throttling.
Your investment in the liquid cooling system will pay for itself in a few months.
It’s been an incredible project and BIZON team did a great job.
BIZON liquid cooled experts and engineers spent months for research, development, and building the system and custom components. We have tested hundreds of different configurations, components, materials, coolants, pumps, radiators to find the optimal solution to provide the lowest noise level, temperatures, and performance.
Since its a commercial solution there are two liquid cooling systems for better stability.
Each CPU and 4 GPUs are handled by its dedicated radiator and pump.
That’s why you can see two colors.
Liquid cooling loop includes cooling for dual CPUs and all the graphics cards. So the server is perfect not just for GPU related uses but for CPU uses as well. Moreover, it can handle the GPU and CPU tasks at the same time.
Unlike most commercial solutions that look ugly and boring,
this beast looks great. It’s not just a regular piece of black metal. It’s a piece of art. It’s like a Bugatti in the liquid cooling world!
For more details visit the BIZON Z9000 product page. You can place the order online and customize the system.
Thank you for reading!