RDMA

Most important thing to understand.

As we want to work with files a lot, with large volume of them, we need to reduce overhead as much as possible, and on Windows it is done using RDMA and SMB Direct. The higher the speed of network - the more important it is. Infiniband is actually designed for such work. And RDMA other Ethernet is called RoCE (RDMA over Converged Ethernet).

It must be supported by adapter, it's drivers, OS used and enabled properly.

If you want to keep CPU usage low and use very high IOPS capability of modern SSD drives, you need this.

This also mean that you need to stay away from relatively cheap (<$110) new copper based 10GBASE-T adapters, usually using Aquantia AQtion controller. They are fine things, but made for consumer market and do not have server grade drivers with necessary features and not designed for low CPU utilization.

Most popular option we will follow - used Mellanox ConnectX adapters that come from datacenters due to upgrades of equipment and including quite new servers with internal MB issues that being sold out for cheap.

RDMA also put requirements on OS versions and editions you can use. More on this in one of the next posts here.