[PATCH v2] bus/pci: retry VFIO device reset on EAGAIN

[Thanushree Sreerama]

VFIO_DEVICE_RESET is implemented in the kernel with
pci_try_reset_function(), which uses pci_dev_trylock() and returns
-EAGAIN when the PCI device lock is already held.

That lock is taken by other kernel paths, in particular sysfs
uevent_show(). After Linux restored device_lock() in uevent_show()
(revert of "driver core: Fix uevent_show() vs driver detach race"),
udev or any reader of /sys/.../uevent can race with DPDK VFIO
setup, and the ioctl returns errno 11 (EAGAIN).

Without a retry, pci_rte_vfio_setup_device() fails, EAL skips the
PCI device, and the application cannot use it:

  EAL: Unable to reset device! Error: 11 (Resource temporarily unavailable)
  EAL: 0000:f4:02.3 setup device failed
  EAL: Requested device 0000:f4:02.3 cannot be used

The device lock is held only for the sysfs/uevent critical section,
typically well under 20 ms. Retry the ioctl up to 5 times with a
20 ms delay (100 ms in total). That was enough for the transient
failure we observed; a persistent EAGAIN still fails device setup.

Fixes: f25f8f367644 ("bus/pci: check VFIO reset ioctl error")
Cc: stable at dpdk.org

Signed-off-by: Thanushree Sreerama <thanushree.sreerama at nokia.com>
---
Cc: jpf at zurich.ibm.com
Cc: Anatoly Burakov <anatoly.burakov at intel.com>

Changes since v1:
- Rebased on main
- Drop Change-Id
- Document EAGAIN condition, functional impact, and retry values
- Fail setup if retries are exhausted
- Match pci_vfio.c declaration style

 drivers/bus/pci/linux/pci_vfio.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c249..3ec5af8ed 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -420,6 +420,10 @@ pci_vfio_is_ioport_bar(const struct rte_pci_device *dev, int vfio_dev_fd,
 static int
 pci_rte_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
 {
+	const int max_retries = 5;
+	const int retry_delay_ms = 20;
+	int i, ret;
+
 	if (pci_vfio_setup_interrupts(dev, vfio_dev_fd) != 0) {
 		PCI_LOG(ERR, "Error setting up interrupts!");
 		return -1;
@@ -439,12 +443,20 @@ pci_rte_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
 	 * Reset the device. If the device is not capable of resetting,
 	 * then it updates errno as EINVAL.
 	 */
-	if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
-		PCI_LOG(ERR, "Unable to reset device! Error: %d (%s)", errno, strerror(errno));
-		return -1;
+	for (i = 0; i < max_retries; i++) {
+		errno = 0;
+		ret = ioctl(vfio_dev_fd, VFIO_DEVICE_RESET);
+		if (ret == 0 || errno == EINVAL)
+			return 0;
+		if (errno != EAGAIN || i + 1 == max_retries)
+			break;
+		PCI_LOG(DEBUG, "Device reset EAGAIN, retry %d/%d after %d ms",
+			i + 1, max_retries, retry_delay_ms);
+		usleep(retry_delay_ms * 1000);
 	}
 
-	return 0;
+	PCI_LOG(ERR, "Unable to reset device! Error: %d (%s)", errno, strerror(errno));
+	return -1;
 }
 
 static int
-- 
2.45.0