Best GPU for deep learning in 2022: RTX 3090 vs. RTX 3080 Ti vs A6000 vs A5000 vs A100 benchmarks (FP32, FP16) – Updated –

January, 19, 2022

Introduction

NVIDIA is the industry leader in deep learning and artificial intelligence, with its RTX 30-series and Professional RTX A-Series of GPUs designed specifically for these tasks. Featuring incredible performance and power efficiency, NVIDIA's 30-series is perfect for data scientists, researchers, and developers who want to get started in AI. With support for advanced features like Tensor Cores and Unified Memory, NVIDIA's 30-series delivers the performance you need to succeed in today's increasingly complex world of AI.

Using deep learning benchmarks, we will be comparing the performance of the most popular GPUs for deep learning in 2022: NVIDIA's RTX 3090, A100, A6000, A5000, and A4000.

 

Methodology

  • We used TensorFlow's standard "tf_cnn_benchmarks.py" benchmark script from the official GitHub (more details).
  • We ran tests on the following networks: ResNet-50, ResNet-152, Inception v3, Inception v4, VGG-16.
  • We compared FP16 to FP32 performance and used maxed batch sizes for each GPU.
  • We compared GPU scaling up to 4x GPUs!
To accurately compare benchmark data from multiple workstations, we maintained consistency by having the same driver and framework versions installed on each workstation. It is important to keep a controlled environment in order to have valid, comparable data.

 

Hardware

Test Bench:

bizon x5500BIZON X5500 (1-4x GPU deep learning desktop)
 
Tech specs:
  • CPU: 16-Core 3.90 GHz AMD RYZEN Threadripper Pro 3955WX
  • Overclocking: Stage #2 +200 MHz (up to +10% performance)
  • Cooling: Liquid Cooling System (CPU; extra stability and low noise)
  • Memory: 256 GB (8 x 32 GB) DDR4 3200 MHz
  • Operating System: BIZON Z–Stack (Ubuntu 20.04 (Bionic) with preinstalled deep learning frameworks)
  • HDD: 1TB PCIe SSD
  • Network: 10 GBIT
 
bizon zx5500BIZON ZX5500 (liquid cooled deep learning and GPU rendering workstation PC)
 
Tech specs:
  • CPU: 64-Core 3.5 GHz AMD RYZEN Threadripper Pro 3995WX
  • Overclocking: Stage #2 +200 MHz (up to + 10% performance)
  • Cooling: Custom water-cooling system (CPU + GPUs)
  • Memory: 256 GB (8 x 32 GB) DDR4 3200 MHz
  • Operating System: BIZON Z–Stack (Ubuntu 20.04 (Bionic) with preinstalled deep learning frameworks)
  • HDD: 1TB PCIe SSD
  • Network: 10 GBIT

 

Software

Deep Learning Models:
  • Resnet50
  • Resnet152
  • Inception V3
  • Inception V4
  • VGG16
Drivers and Batch Size:
  • Nvidia Driver: 470
  • CUDA: 11.2
  • TensorFlow: 2.40

Benchmarks












Note: Due to their 2.5 slot design, RTX 3090 GPUs can only be tested in 2-GPU configurations when air-cooled. Water-cooling is required for 4-GPU configurations.


Conclusion

Our Recommendation: NVIDIA RTX 3090, 24 GB
Price: $2300

RTX 3090Academic discounts are available.
Notes: Water cooling required for 4 x RTX 3090 configurations.  
NVIDIA's RTX 3090 is the best GPU for deep learning and AI. It has exceptional performance and features that make it perfect for powering the latest generation of neural networks. Whether you're a data scientist, researcher, or developer, the RTX 3090 will help you take your projects to the next level.
The RTX 3090 has the best of both worlds: excellent performance and price. The RTX 3090 is the only GPU model in the 30-series capable of scaling with an NVLink bridge. When used as a pair with an NVLink bridge, one effectively has 48 GB of memory to train large models. A problem some may encounter with the RTX 3090 is cooling, mainly in multi-GPU configurations. Due to its massive TDP of 350W and because the RTX 3090 does not have blower-style fans, it will almost immediately activate thermal throttling and then shut off at 90°C.
We have seen an up to 60% (!) performance drop due to overheating.
Liquid cooling is the best solution; providing 24/7 stability, low noise, and greater hardware longevity. Plus, any water-cooled GPU is guaranteed to run at its maximum possible performance. As per our tests, a water-cooled RTX 3090 will stay within a safe range of 50-60°C vs 90°C when air-cooled (90°C is the red zone where the GPU will stop working and shutdown). Noise is another important point to mention. 2x or 4x air-cooled GPUs are pretty noisy, especially with blower-style fans. Keeping the workstation in a lab or office is impossible - not to mention servers. The noise level is so high that it’s almost impossible to carry a conversation while they are running. Without proper hearing protection, the noise level may be too high for some to bear. Liquid cooling resolves this noise issue in desktops and servers. Noise is 20% lower compared to air cooling (49 dB for liquid cooling vs. 62 dB for air cooling on maximum load). One could place a workstation or even a server with such massive computing power in an office or lab. BIZON has designed an enterprise-class custom liquid-cooling system for servers and workstations.
 
We offer a wide range of deep learning workstations and GPU optimized servers.
Recommended workstations: Recommended NVIDIA GPU servers:

 

NVIDIA A100, 80 GB
Price: $12199

RTX A100Academic discounts are available.
A100 is the world's most advanced deep learning accelerator. It delivers the performance and flexibility you need to build intelligent machines that can see, hear, speak, and understand your world. Powered by the latest NVIDIA Ampere architecture, the A100 delivers up to 5x more training performance than previous-generation GPUs. Plus, it supports a wide range of AI applications and frameworks, making it the perfect choice for any deep learning deployment.
 
We offer a wide range of deep learning workstations and GPU optimized servers.
Recommended workstations: Recommended NVIDIA GPU servers:

 

NVIDIA A6000, 48 GB
Price: $4650

RTX A6000Academic discounts are available.
The NVIDIA A6000 GPU offers the perfect blend of performance and price, making it the ideal choice for professionals. With its advanced CUDA architecture and 48GB of GDDR6 memory, the A6000 delivers stunning performance. Training on RTX A6000 can be run with the max batch sizes.
 
We offer a wide range of deep learning workstations and GPU optimized servers.
Recommended workstations: Recommended NVIDIA GPU servers:

 

NVIDIA A5000, 24 GB
Price: $3100

RTX A5000Academic discounts are available.
NVIDIA's A5000 GPU is the perfect balance of performance and affordability. NVIDIA A5000 can speed up your training times and improve your results. This powerful tool is perfect for data scientists, developers, and researchers who want to take their work to the next level.
 
We offer a wide range of deep learning workstations and GPU optimized servers.
Recommended workstations: Recommended NVIDIA GPU servers:

 

NVIDIA A4000, 16 GB
Price: $1300

RTX A4000Academic discounts are available.
NVIDIA A4000 is a powerful and efficient graphics card that delivers great AI performance. Featuring low power consumption, this card is perfect choice for customers who wants to get the most out of their systems. RTX A4000 has a single-slot design, you can get up to 7 GPUs in a workstation PC.
 
We offer a wide range of deep learning workstations and GPU optimized servers.
Recommended workstations: Recommended NVIDIA GPU servers:
You can find more NVIDIA RTX A6000 vs RTX A5000 vs RTX A4000 vs RTX 3090 GPU Deep Learning Benchmarks here.

 

Overall Recommendations

For most users, NVIDIA RTX 3090 or NVIDIA A5000 will provide the best bang for their buck.
Working with a large batch size allows models to train faster and more accurately, saving a lot of time.
With the latest generation, this is only possible with the A6000 or RTX 3090.
Using FP16 allows models to fit in GPUs with insufficient VRAM.
24 GB of VRAM on the RTX 3090 is more than enough for most use cases, allowing space for almost any model and large batch size.
Back to News List
Need Help? We're here to help.

Unsure what to get? Have technical questions?
Contact us and we'll help you design a custom system which will meet your needs.

Explore Products