<div dir="ltr"><div dir="ltr"><div>Hello,</div><div><br></div><div>I have found a bug in the ixgbe pmd, concerning this model of nic in particular:</div><div> -> Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T</div><div><br></div><div>I was able to detect this bug after enabling the use of Link state change interrupt in the card via the following flag:</div><div> - port->rte_conf.intr_conf.lsc = 1;</div><div><br></div><div>With
these interrupts enabled on this particular network card, there is a
chance on every link-up that all interrupts get disabled, as the
interrupt mask is unintentionally set to 0.</div><div>This happens
because there are two interrupts
that are triggered on a link status change on this model of nic. See related code here:</div><div><a href="https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_ethdev.c?id=c99e1db87798c2cd2c0ac94c7676e961eedffc87#n4552" target="_blank">https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_ethdev.c?id=c99e1db87798c2cd2c0ac94c7676e961eedffc87#n4552</a></div><div><br></div><div>Whenever an lsc interrupt is triggered, we enter this<code> ixgbe_dev_interrupt_get_status</code>() function.</div><div>Bit 20 (<code>IXGBE_EICR_LSC</code> ) and 25 (<code>IXGBE_EICR_GPI_SDP0_X550EM_x</code> ) of the Extendend Interrupt Cause Register (EICR) will be set to '1' on a link status change.</div><div>When the interrupt is handled in the <code>ixgbe_dev_interrupt_delayed_handler()</code> function, if the <code>IXGBE_FLAG_NEED_LINK_UPDATE</code> flag is raised (happens when EICR_LSC bit is raised in eicr), an alarm for a delayed handler will be programmed.</div><div><br></div><div>The following code poses issue:</div><div><a href="https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_ethdev.c?id=c99e1db87798c2cd2c0ac94c7676e961eedffc87#n4736" target="_blank">https://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_ethdev.c?id=c99e1db87798c2cd2c0ac94c7676e961eedffc87#n4736</a></div><div><pre><code>intr->mask = intr->mask_original;
intr->mask_original = 0;</code></pre></div><div>Sometimes, it's possible to see a first interrupt with bit <code>IXGBE_EICR_LSC</code> raised, quickly followed by a second interrupt with both bit <code>IXGBE_EICR_LSC</code> and bit <code>IXGBE_EICR_GPI_SDP0_X550EM_x</code> being raised.</div><div>In this case, the flag <code>IXGBE_FLAG_NEED_LINK_UPDATE</code> will be raised in both cases and the delayed handler will be programmed to be ran twice.</div><div>This will lead to <code>intr->mask_original</code> being set to 0 after the first delayed_handler is done, and then <code>intr->mask</code> being also set to 0 after the second time it's run.</div><div>This
effectively disables all interrupts from working on the device, which
is particularly painful since we no longer poll for link state on our
side, having switched to interrupt-based link state handling.</div><div><br></div><div>My suggestion to solve this issue is to avoid creating an alarm for <code>ixgbe_dev_interrupt_delayed_handler()</code> if there is already one pending. I do this by checking for bit IXGBE_EIMS_LSC in<code> intr->mask</code>.<br></div><div>I no longer see this issue after pushing in my local repository. I will suggest a patch by responding to this mail.<br></div><div><br></div><div>I can show a way to reproduce bellow with test pmd. I have added some custom logs into the code for my own convenience: <br></div><div><br></div><div>=====================================</div><div><br></div><div> 1) start testpmd:</div><div><br></div><div>PCI-SLOT IFNAME MAC-ADDRESS KMOD MAX SPEED CURRENT SPEED DEVICE<br>0000:03:00.0
ntfp1 0c:c4:7a:75:8f:ec ixgbe 1x2.5 GT/s PCIe 1x2.5 GT/s PCIe
Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T<br>0000:03:00.1
ntfp2 0c:c4:7a:75:8f:ed ixgbe 1x2.5 GT/s PCIe 1x2.5 GT/s PCIe
Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T</div><div><br></div><div>sudo dpdk-hugepages.py --setup 2G;</div><div>dpdk-devbind --bind=vfio-pci 0000:03:00.0<br>dpdk-devbind --bind=vfio-pci 0000:03:00.1</div><div><br></div><div> dpdk-testpmd -a 0000:03:00.0 -a 0000:03:00.1 -- -i --rxq=2 --txq=2 --coremask=0x0c --total-num-mbufs=250000<br>EAL: Detected CPU lcores: 16<br>EAL: Detected NUMA nodes: 1<br>EAL: Detected static linkage of DPDK<br>EAL: Multi-process socket /var/run/dpdk/rte/mp_socket<br>EAL: Selected IOVA mode 'PA'<br>EAL: VFIO support initialized<br>EAL: Using IOMMU type 8 (No-IOMMU)<br>EAL: Probe PCI driver: net_ixgbe (8086:15ad) device: 0000:03:00.0 (socket -1)<br>EAL: Probe PCI driver: net_ixgbe (8086:15ad) device: 0000:03:00.1 (socket -1)<br>Interactive-mode selected<br>previous number of forwarding cores 1 - changed to number of configured cores 2<br>Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.<br>testpmd: create a new mbuf pool <mb_pool_0>: n=250000, size=2176, socket=0<br>testpmd: preferred mempool ops selected: ring_mp_mc<br>Configuring Port 0 (socket 0)<br>Port 0: 0C:C4:7A:75:8F:EC<br>Configuring Port 1 (socket 0)<br>Port 1: 0C:C4:7A:75:8F:ED<br>Checking link statuses...<br>Done</div><div><br></div><div><br></div><div> 2) To reproduce, simply repeatedly set link up and down. At one point there should be something like this:</div><div><br></div><div>testpmd> set link-up port 0</div><div>ixgbe_dev_interrupt_get_status(): <><><><>><>: eicr register: 100000 <---------------------- <code>IXGBE_EICR_LSC</code><br>ixgbe_dev_interrupt_get_status(): <><><><>><>: set IXGBE_FLAG_NEED_LINK_UPDATE<br>ixgbe_dev_interrupt_get_status(): <><><><>><>: current flags: flags = 1<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (1): flags = 1<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (2): flags = 1<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (3): flags = 1<br>ixgbe_dev_interrupt_action(): <><><><>><>: for dev: 0x557c0cd844c0<br>ixgbe_dev_interrupt_action():
<><><><>><>: saved intr: mask_original
= 36700160 <------------ original interrupt mask<br>ixgbe_dev_interrupt_action():
<><><><>><>: current intr: mask =
35651584 <------------ interrupt mask with lsc
disabled<br>ixgbe_dev_interrupt_get_status(): <><><><>><>: eicr register: 2100000 <---------------------- <code>IXGBE_EICR_LSC and <code>IXGBE_EICR_GPI_SDP0_X550EM_x</code></code><br>ixgbe_dev_interrupt_get_status(): <><><><>><>: set IXGBE_FLAG_NEED_LINK_UPDATE<br>ixgbe_dev_interrupt_get_status(): <><><><>><>: current flags: flags = 5<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (1): flags = 5<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (2): flags = 5<br>ixgbe_dev_interrupt_action(): <><><><>><>: current flags (3): flags = 1<br>ixgbe_dev_interrupt_action(): <><><><>><>: for dev: 0x557c0cd844c0<br>ixgbe_dev_interrupt_action():
<><><><>><>: saved intr: mask_original
= 35651584 <------------ original mask is lost here<br>ixgbe_dev_interrupt_action(): <><><><>><>: current intr: mask = 35651584<br><br>Port 0: link state change event<br>ixgbe_dev_interrupt_delayed_handler(): <><><><>><>: mask restored: 35651584<br>testpmd> set link-down port 0ixgbe_dev_interrupt_delayed_handler(): <><><><>><>: mask restored: 0 <--- all interrupts are disabled<br></div><div><br></div><div> 3) At this point, link state can no longer be updated:</div><div><br></div><div>testpmd> show port info 0<br><br>********************* Infos for port 0 *********************<br>MAC address: 0C:C4:7A:75:8F:EC<br>Device name: 0000:03:00.0<br>Driver name: net_ixgbe<br>Firmware-version: 0x800001cf<br>Devargs: <br>Connect to socket: 0<br>memory allocation on the socket: 0<br>Link status: up <----------------------<br>Link speed: 10 Gbps<br>Link duplex: full-duplex<br>Autoneg status: On<br>MTU: 1500</div><div><br></div><div>testpmd> set link-down port 0<br>testpmd> show port info <br><br>********************* Infos for port 0 *********************<br>MAC address: 0C:C4:7A:75:8F:EC<br>Device name: 0000:03:00.0<br>Driver name: net_ixgbe<br>Firmware-version: 0x800001cf<br>Devargs: <br>Connect to socket: 0<br>memory allocation on the socket: 0<br>Link status: up <------------- should not be up after link down.<br>Link speed: 10 Gbps<br>Link duplex: full-duplex<br>Autoneg status: On<br>MTU: 1500<div><br><br></div></div></div></div>