Network packet steering of transmitted and received traffic for multi-core architectures (multi-core network packet steering) is needed in modern network computing environments, especially in data centers, where the high bandwidth and heavy loads would easily congest a single core's queue. For this reason, many techniques, both in hardware and in software, are used in order to distribute the incoming packets across the cores of the processor. On the traffic-receiving side, the most important techniques are RSS, aRFS, RPS and RFS. For transmission, XPS is important. Packets coming into the network interface card (NIC) are processed and loaded to the receiving queues managed by the cores (which are usually implemented as ring buffers within the kernel space). The main objective is being able to use all the cores available within the CPU to process incoming packets, while also improving latency and throughput.
Hardware techniques Hardware accelerated techniques like RSS and aRFS are used to route and load balance incoming packets across the multiple cores' queues of a processor. Those hardware supported methods achieve extremely low latencies and reduce the load on the CPU, as compared to the software based ones. However they require a specialized hardware integrated within the network interface controller (which, for example, is usually available on more advanced cards, like the SmartNIC).
RSS
Receive Side Scaling (RSS) is a hardware supported technique, leveraging an indirection table indexed by the last bits of the result provided by a hash function, taking as inputs the header fields of the packets. The hash function input is usually customizable and the header fields used can vary between use case and implementations. Some notable examples of header fields chosen as keys for the hash are the layer 3 IP source and destination addresses, the protocol and the layer 4 source and destination ports. In this way, packets corresponding to the same flow will be directed to the same receiving queue, without losing the original order, causing an out-of-order delivery. Moreover, all incoming flows will be load balanced across all the available cores thanks to the hash function properties. Another important feature introduced by the indirection table is the capability of changing the mapping of flows to the cores without having to change the hash function, but by simply updating the table entries.
aRFS
Accelerated Receive Flow Steering (aRFS) is another hardware supported technique, born with the idea of leveraging cache locality to improve performances by routing incoming packet flows to specific cores. Differently from RSS which is a fully independent hardware implementation, aRFS needs to interface with the software (the kernel) to properly function. RSS simply load balance incoming traffic across the cores; however if a packet flow is directed to the core i (as a result of the hash function) while the application needing the received packet is running on core j, many cache misses could be avoided by simply forcing i=j, so that packets are received exactly where they are needed and consumed. To do this aRFS doesn't forward packets directly from the result of the hash function, but using a configurable routing table (which can be filled and updated for instance by the scheduler through an API) packet flows can be steered to the specific consuming core.
Software techniques Software techniques like RPS and RFS employ one of the CPU cores to steer incoming packets across the other cores of the processor. This comes at the cost of introducing additional inter-processor interrupts (IPIs); however the number of hardware interrupts will not increase and potentially, by employing an interrupt aggregation technique, it could even be reduced. The benefits of a software solutions is the ease in implementation, without having to change any component (like the NIC) of the currently used architecture, but by simply deploying the proper kernel module. This benefit can be crucial especially in cases where the server machine can't be customized or accessed (like in cloud computing environment), even if the network performances could be reduced as compared the hardware supported ones.
RPS
Receive Packet Steering (RPS) is the RSS parallel implemented in software. All packets received by the NIC are load balanced between the cores' queues by implementing an hash function using as configurable key the header fields (like the layer 3 source and destination IP and layer 4 source and destination ports), in the same fashion as RSS does. Moreover, thanks to the hash properties, packets belonging to the same flow will always be steered to the same core. This is usually done in the kernel, right after the NIC driver. Having handled the network interrupt and before it can be processed, the packet is sent to the receiving queue of a core, which is then notified thanks to an inter process interrupt. RPS can be used in conjunction with RSS, in case the number of queues managed by the hardware is lower than the number of cores. In this case after having distributed across the RSS queues the incoming packets, a pool of cores can be assigned to each queue and RPS will be used to spread again the incoming flows across the specified pool.
RFS
Receive Flow Steering (RFS) upgrades RPS in the same direction as the aRFS hardware solution does. By routing packet flows to the same CPU core running the consuming application, cache locality can be improved and leveraged, avoiding many misses and reducing the latencies introduced by the retrieval of the data from the central memory. To do this, after having computed the hash of the header fields for the current packet, the result is used to index a lookup table. This table is managed by the scheduler, which updates its entries when the application processes are moved between the cores. The overall CPU load distribution is balanced as long as the applications in user-space are evenly distributed across the multiple cores.
… excerpt ends here. Continue reading the full article.
