InfiniBand vs. Ethernet
As interconnection technologies, InfiniBand and Ethernet can be said to have their own characteristics and differences, and it is impossible to generalize which one is better. They continue to develop and evolve in different fields of application, and have become two indispensable interconnection technologies in our network world.
InfiniBand vs. Ethernet Network: What are they?
InfiniBand Network
The difference between InfiniBand and Ethernet is very different from a design point of view. As a technology for interconnecting networks, InfiniBand is widely used in supercomputer clusters due to its high reliability, low latency and high bandwidth. In addition, it is the preferred network interconnection technology for GPU servers as artificial intelligence advances.
To achieve a raw data rate of 10Gbits/sec over 4X cables, the InfiniBand standard allows the transmission of Single Data Rate (SDR) signals at a base rate of 2.5Gbits/sec per lane. A single channel can be expanded to 5Gbits/sec and 10Gbits/sec respectively, and the potential maximum data rate is 40Gbits/sec over 4X cables and 120Gbits/sec over 12X cables, enabling InfiniBand networks to have double data rate (DDR) and quadruple data rate (QDR) signal.
Ethernet Network
Since its introduction on September 30, 1980, the Ethernet standard has become the most widely used communication protocol in the LAN. Unlike InfiniBand, Ethernet was designed with the following main goals in mind: How can information flow easily between multiple systems? It’s a typical network designed with distribution and compatibility. The traditional Ethernet mainly uses TCP/IP to build the network, and it has gradually developed into RoCE so far.
In general, Ethernet networks are primarily used to connect multiple computers or other devices such as printers, scanners, etc. to a local area network. It can not only connect the Ethernet network to the wired network through the optical fiber cable, but also realize the Ethernet network in the wireless network through wireless networking technology. Fast Ethernet, Gigabit Ethernet, 10 Gigabit Ethernet, and Switched Ethernet are all major types of Ethernet.
InfiniBand vs. Ethernet: What Are the Differences Between Them?
The bottleneck of cluster data transmission in high-performance computing scenarios is the original design goal of InfiniBand, and it has become an interconnection standard in line with the requirements of the times. Therefore, InfiniBand and Ethernet have many differences, mainly in terms of bandwidth, latency, network reliability, and network technology.
Bandwidth
Since the birth of InfiniBand, the development of InfiniBand network has been faster than that of Ethernet for a long time. The main reason is that InfiniBand is applied to the interconnection between servers in high-performance computing and reduces the CPU load. However, Ethernet is more oriented to terminal device interconnection, and there is not too high a demand for bandwidth.
For high-speed network traffic exceeding 10G, if all packets are unpacked, it will consume a lot of resources. The first generation of SDR InfiniBand operates at a rate of 10Gbps, allowing for high-speed network transmission to offload the CPU and increase network utilization in addition to increasing data transmission bandwidth and decreasing CPU load.
Network Latency
InfiniBand and Ethernet also behave very differently when it comes to network latency. Ethernet switches typically employ store-and-forward and MAC table lookup addressing as Layer 2 technologies in the network transport model. The processing flow of Ethernet switches is longer than that of InfiniBand switches, because complex services such as IP, MPLS, and QinQ must be considered.
Layer 2 processing, on the other hand, is very straightforward for InfiniBand switches. The 16-bit LID is the only one that can be used to search for the forwarding path information. In parallel, the Cut-Through technology is utilized to significantly reduce the forwarding delay to less than 100 ns, which is significantly faster than the Ethernet switch.
Network Reliability
Since packet loss and retransmission have a significant impact on the overall performance of high-performance computing, a highly reliable network protocol is needed to ensure the lossless characteristics of the network at the mechanism level and realize its high reliability features. With its own defined layer 1 to layer 4 formats, InfiniBand is a complete network protocol. End-to-end flow control is the basis for the InfiniBand network’s packet sending and receiving, which can result in a lossless network.
Compared with InfiniBand, Ethernet network does not have a scheduling-based flow control mechanism. As a result, it is impossible to guarantee whether the peer end will be congested when sending packets. To be able to absorb the sudden increase of instantaneous traffic in the network, it is necessary to open up a cache space of tens of MB in the switch for temporarily storing these messages, which occupy the chip resources. This means that the chip area of an Ethernet switch with the same specification is significantly larger than that of an InfiniBand chip, which not only costs more but also consumes more power.
Networking Methods
In terms of networking mode, the InfiniBand network is simpler to manage than the Ethernet network. The idea of SDN is built into InfiniBand by design. A subnet manager will be present on each InfiniBand layer 2 network to configure the ID (LocalID) of the network’s nodes, uniformly calculate the forwarding path information through the control plane, and issue it to the InfiniBand exchange. To complete the network configuration, such a layer-2 network must be configured without any configuration.
Use the Ethernet networking mode to automatically generate MAC entries, and the IP must cooperate with the ARP protocol. Additionally, each server in the network must send packets on a regular basis to guarantee that entries are updated in real time. To divide the virtual network and limit its scale, it must therefore implement the Vlan mechanism. However, because the Ethernet network itself lacks an entry learning mechanism, it will result in a loop network. To prevent loops in the network forwarding path, protocols like STP must be implemented, which increases the complexity of network configuration.
Finding Your InfiniBand HDR Switches
Judging from the comparison between InfiniBand and Ethernet above, the advantages of InfiniBand networks are very prominent. If you’re looking to deploy your high-performance data center with InfiniBand switches, here is more information for you. The rapid iteration of the InfiniBand network, from SDR 10Gbps, DDR 20Gbps, QDR 40Gps, FDR56Gbps, EDR 100Gbps to today’s 200Gbps InfiniBand, all benefit from RDMA technology.
FS recently launched two 200G InfiniBand switches, which have a latency of less than 130ns, 16Tb/s aggregation switch throughput, high bandwidth of 40 HDR 200Gb/s ports, 80 HDR100 100Gb/s ports, and 274W power consumption. They use NVIDIA Quantum™ IC chips and superior process materials to enable higher performance and more reliable networks.
Conclusion
There are suitable application scenarios between InfiniBand and Ethernet. The CPU does not sacrifice more resources for network processing because of the significant increase in the rate brought by the InfiniBand network, which improves network utilization. This is one of the main reasons why the InfiniBand network will become the main network solution for the high-performance computing industry. 400bpsG NDR and 800Gbps XDR switches will also appear in the future. If there is no high requirement for communication delay between data center nodes, and flexible access and expansion are more important, then Ethernet networks can be selected for a long time.
Source: community.fs
[ratings]