[dpdk-users] Interrupt mode, queues, own event loop

[Stephen Hemminger]

On Fri, 04 Sep 2020 12:24:06 +0200
Budiský Jakub <ibudisky at fit.vutbr.cz> wrote:

> Hi,
> 
> I'm working on a project that involves packet bursts receiving; other 
> than that it's mostly idle. The DPDK was incorporated later on (when I 
> found out that Linux AF_XDP won't do the job) and I use my own C++ 
> implementation of an epoll-based event loop along with eventfd and 
> timerfd for communication / timeouts.
> 
> So I'm trying to use per-queue interrupts in my own event loop with 
> DPDK. Per-queue is quite important since I'm using the flow director for 
> load balancing and I'm relying on it. In the DPDK 18.11 (I believe) a 
> new function `rte_eth_dev_rx_intr_ctl_q_get_fd` was introduced just for 
> this purpose.
> 
> I'm currently using `uio_pci_generic` driver with Intel's 82599ES NIC 
> for debugging. For production I will switch to `vfio` due to the 
> application running in the userspace.
> 
> I've encountered two problems; the first being that I've expected the 
> DPDK to pass me eventfd file descriptors. While debugging I found out 
> that these are, in fact, /dev/uio0 files (I guess these are special 
> files created by the driver). I don't mind them "being different", but 
> this raises a few other issues: Is it safe to read them, i.e. does the 
> `ixgbe_pmd` driver rely on them in any way? Is there a way of 
> discriminating between a different types of file descriptor I may obtain 
> except looking at `/proc/self/fd/<fd_number>`? From the implementation 
> of `eal_intr_proc_rxtx_intr` it looks like the file descriptors will 
> differ for the `vfio` driver and I need to read a different amount of 
> data from them (4 Bytes for UIO vs. 8 Bytes for VFIO respectively, other 
> sizes may rise EINVAL).
> 
> The second problem is that I've got the same file descriptor for all the 
> queues, which means it may not be captured by the epoll in all relevant 
> threads. Is this behaviour intended? I recall seeing some limits 
> regarding the number of interrupt file descriptors but I believe it was 
> 15 for my NIC. I don't mind but I need to change the program's logic to 
> account for this. Can I read the file descriptor and find out which 
> queues do need to process incoming packets, or do I just wake them all 
> up? Does this differ (and if, how) between the `vfio` and 
> `uio_pci_generic` drivers?
> 
> I feel like I may have missed something, reading the 
> `linux/eal_interrupts.c` it indeed looks like some eventfd descriptors 
> are set up, but maybe this matters only if you use DPDK-encapsulated 
> event loop. Please let me know if I should call anything besides 
> `rte_eth_dev_rx_intr_ctl_q_get_fd` and the usual device configuration 
> functions.


The per-queue interrupt functionality for PCI devices is built
on top of MSI-X interrupts. The uio_pci_generic driver you are using
does not support MSI-X.

The way UIO driver works is to use the legacy INTx functionality,
and when an interrupt occurs the device driver in the kernel is called.
For the uio_pci_generic driver this is mapped to the device file descriptor.

For VFIO, you can have one interrupt per queue and it uses eventfd's
to create a per-queue channel.