A network supporting thousands of fast and efficient computing resources cannot create any congestion points while an application is trying to create large models consisting of billions of data points.
Current GPUs run at 400 Gigabits per second (Gbps), connecting two GPUs across a switch would require an 800Gbps non-blocking connection.
Scaling a single pair of GPUs to thousands demonstrates the need of having a high-performance network fabric. The network needs to become an asset, be transparent, and work as a highly efficient communication backbone infrastructure to support this effort.