<div dir="ltr">Hi, Attaching the logs of EAL when trying to run a configuration with virtio_user port when IOMMU is disabled and enable_unsafe_iommu is enabled. As you can see it forces IOVA as PA but the viritui_user needs IOVA as VA. I am also attaching the output of dmesg. I am not sure which kernel logs you wanted... if there is anything else please let me know.. Regarding the ESXI logs, they are HUGE so I will send to you on a separate email.</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 21, 2023 at 1:14 PM Ferruh Yigit <<a href="mailto:ferruh.yigit@amd.com">ferruh.yigit@amd.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 7/21/2023 12:39 PM, Igor de Paula wrote: > I am trying to use virtio_user for an interface with the > kernel: <a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html</a> <<a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html</a>> > I think this requires IOVA as va. > I am not sure if virtio-user has IOVA as VA requirement, cc'ed Chenbo, he may know better. Meanwhile can you give a try to 'enable_unsafe_noiommu_mode' and '--iova-mode=pa'? > It does work with Intel host and IOMMU > enabled. Part of the negotiation when setting it up is getting the IOMMU > number so I thought it has to have IOMMU. > Yes, issue looks like related to the IOMMU, and it may be either related to HW support, or ESXi iommu driver support, we will check using below information you provided. > I tried disabling IOMMU and enabling enable_unsafe_noiommu flag but > again, that didn't work. > ESXI version - VMware ESXi, 7.0.0, 16324942 > AMD: AMD EPYC 7452 32-Core Processo > > On an Intel host which worked: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz > Regarding the logs I will try and attach it soon. > Thanks for info, waiting for logs. > > On Fri, Jul 21, 2023 at 12:21 PM Ferruh Yigit <<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a> > <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a>>> wrote: > > Hi Igor, > > VM doesn't have IOMMU, and vmxnet3 requires PA mode, for this can you > please try with: > - enable 'enable_unsafe_noiommu_mode' flag > - Force PA mode via '--iova-mode=pa' eal parameter > > > Also to be able to figure out AMD IOMMU support level, can you please > provide: > - AMD part number > - ESXi/hypervisor version > - ESXi & VM kernel logs > > > Thanks, > Ferruh > > On 7/20/2023 5:21 PM, Jochen Behrens wrote: > > +Ronak from the ESX team > > > > > > > > > > > > In our usage, we do set amd_iommu=off in the boot command line from > > grub. (Or intel_iommu=off for Intel processors.) > > > > > > > > Jochen > > > > > > > > *From: *Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a> > <mailto:<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>> > > *Date: *Thursday, July 20, 2023 at 6:00 AM > > *To: *Igor de Paula <<a href="mailto:igordptx@gmail.com" target="_blank">igordptx@gmail.com</a> <mailto:<a href="mailto:igordptx@gmail.com" target="_blank">igordptx@gmail.com</a>>> > > *Cc: *<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> <mailto:<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>> <<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a> > <mailto:<a href="mailto:users@dpdk.org" target="_blank">users@dpdk.org</a>>>, Jochen Behrens > > <<a href="mailto:jbehrens@vmware.com" target="_blank">jbehrens@vmware.com</a> <mailto:<a href="mailto:jbehrens@vmware.com" target="_blank">jbehrens@vmware.com</a>>>, Nipun Gupta > <<a href="mailto:nipun.gupta@amd.com" target="_blank">nipun.gupta@amd.com</a> <mailto:<a href="mailto:nipun.gupta@amd.com" target="_blank">nipun.gupta@amd.com</a>>>, Nikhil Agarwal > > <<a href="mailto:nikhil.agarwal@amd.com" target="_blank">nikhil.agarwal@amd.com</a> <mailto:<a href="mailto:nikhil.agarwal@amd.com" target="_blank">nikhil.agarwal@amd.com</a>>>, Ferruh > Yigit <<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a> <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a>>> > > *Subject: *Re: help > > > > !! External Email > > > > +Cc some AMD maintainers, they can have an idea about IOMMU settings. > > > > > > 20/07/2023 14:44, Igor de Paula: > >> I have enabled it in the host and in the BIOS for AMD... > >> In the Bios I changed to amd_iommu=on and in the host it's the > same for > >> either. > >> > >> On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon > <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a> <mailto:<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>> wrote: > >> > >> > 20/07/2023 11:35, Igor de Paula: > >> > > The weird thing is that it only happens when I am using a > host with an > >> > AMD > >> > > processor. It doesn't happen when I use a host with an Intel > processor. > >> > > >> > So it's probably a matter of BIOS settings for the IOMMU? > >> > > >> > > >> > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon > <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a> <mailto:<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>> > >> > > wrote: > >> > > > >> > > > +Cc the vmxnet3 maintainer. > >> > > > > >> > > > Please Jochen, do you have an idea what's wrong below? > >> > > > > >> > > > > >> > > > 20/07/2023 11:25, Igor de Paula: > >> > > > > This is because it can't negotiate the IOMMU type with > any port. > >> > > > > > >> > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon > <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a> <mailto:<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>> > >> > > > >> > > > wrote: > >> > > > > > >> > > > > > Hello, > >> > > > > > > >> > > > > > The first error is "Cause: Error: number of ports must > be even" > >> > > > > > > >> > > > > > > >> > > > > > 03/05/2023 18:13, Igor de Paula: > >> > > > > > > I am running a VM inside a VMWARE server (vSphere). > >> > > > > > > My goal it to set up DPDK with two HW ports, and set up a > >> > > > virtio_user to > >> > > > > > > interact with the kernel stack. > >> > > > > > > In another app I have it working but instead of > virtio_user I am > >> > > > running > >> > > > > > > KNI, it works in IOVA-PA mode. > >> > > > > > > I am looking to replace the KNI. > >> > > > > > > > >> > > > > > > When I try to set up virtio_user port as in the doc: > >> > > > > > > > >> > > > > > > >> > > > > >> > > <a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path</a> <<a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path</a>> <<a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path</a> <<a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path</a>>> > >> > > > > > > I get a error it can't run in PA mode. > >> > > > > > > > >> > > > > > > > >> > > > > > > When I try to run as VA mode from a parameter, I get the > >> > following > >> > > > > > errors: > >> > > > > > > EAL: lib.eal log level changed from info to debug > >> > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > >> > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > >> > > > > > > EAL: Support maximum 128 logical core(s) by > configuration. > >> > > > > > > EAL: Detected 2 lcore(s) > >> > > > > > > EAL: Detected 1 NUMA nodes > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > >> > > > > > > EAL: Checking presence of .so 'librte_eal.so' > >> > > > > > > EAL: Detected static linkage of DPDK > >> > > > > > > EAL: Ask a virtual area of 0x7000 bytes > >> > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000) > >> > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > >> > > > > > > EAL: DPAA Bus not present. Skipping. > >> > > > > > > EAL: VFIO PCI modules not loaded > >> > > > > > > EAL: Selected IOVA mode 'VA' > >> > > > > > > EAL: Probing VFIO support... > >> > > > > > > EAL: IOMMU type 1 (Type 1) is supported > >> > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > >> > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > >> > > > > > > EAL: VFIO support initialized > >> > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > >> > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000) > >> > > > > > > EAL: Setting up physically contiguous memory... > >> > > > > > > EAL: Setting maximum number of open files to 1048576 > >> > > > > > > EAL: Detected memory type: socket_id:0 > hugepage_sz:1073741824 > >> > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > >> > > > > > hugepage_sz:1073741824 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > 0x100000kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x140000000 (size = > 0x2000000000) > >> > > > > > > EAL: VA reserved for memseg list at 0x140000000, size > 2000000000 > >> > > > > > > EAL: Ask a virtual area of 0x2000 bytes > >> > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000) > >> > > > > > > EAL: Memseg list allocated at socket 0, page size > 0x100000kB > >> > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > >> > > > > > > EAL: Virtual area found at 0x2180000000 (size = > 0x2000000000) > >> > > > > > > EAL: VA reserved for memseg list at 0x2180000000, > size 2000000000 > >> > > > > > > EAL: TSC frequency is ~2350000 KHz > >> > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > >> > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > >> > > > > > > EAL: Trying to obtain current memory policy. > >> > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > >> > > > > > > EAL: Restoring previous memory policy: 0 > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is > 'VA', not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is > 'VA', not > >> > > > > > initializing > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > EAL: Bus (pci) probe failed. > >> > > > > > > EAL: lib.telemetry log level changed from disabled to > warning > >> > > > > > > EAL: Error - exiting with code: 1 > >> > > > > > > Cause: Error: number of ports must be even > >> > > > > > > EAL: request: mp_malloc_sync > >> > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > For some reason the HW ports won't setup. From what I > understand > >> > > > > > > net_vmxnet3 should work with VA mode. > >> > > > > > > I enabled I/OMUU for the VM. > >> > > > > > > The weird thing even when enabled, I still have the > >> > > > > > > enable_unsafe_noiommu_mode flag on. > >> > > > > > > And because it's on the this: > >> > > > > > > > >> > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev); > >> > > > > > > > >> > > > > > > return PA mode, and it fails. > >> > > > > > > > >> > > > > > > When I disable it by modifying > >> > > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get > >> > another > >> > > > > > error. > >> > > > > > > The error is that it doesn't find a suitable IOMMU type: > >> > > > > > > Just putting the relevant message: > >> > > > > > > > >> > > > > > > > >> > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > >> > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No > such > >> > device) > >> > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > >> > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > >> > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > >> > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No > such device) > >> > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No > such > >> > device) > >> > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > >> > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > >> > > > > > > > >> > > > > > > > >> > > > > > > I can see where it fails: > >> > > > > > > > >> > > > > > > > >> > > > > > > const struct vfio_iommu_type * > >> > > > > > > vfio_set_iommu_type(int vfio_container_fd) > >> > > > > > > { > >> > > > > > > unsigned idx; > >> > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) { > >> > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx]; > >> > > > > > > > >> > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, > >> > > > > > > t->type_id); > >> > > > > > > if (!ret) { > >> > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > >> > > > > > > t->type_id, t->name); > >> > > > > > > return t; > >> > > > > > > } > >> > > > > > > /* not an error, there may be more supported IOMMU > types */ > >> > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, > error " > >> > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > >> > > > > > > strerror(errno)); > >> > > > > > > } > >> > > > > > > /* if we didn't find a suitable IOMMU type, fail */ > >> > > > > > > return NULL; > >> > > > > > > } > >> > > > > > > > >> > > > > > > > >> > > > > > > The ioctl returns -1. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > I tried many things. > >> > > > > > > Enabling/disabling IOMMU. > >> > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > >> > > > > > > Running in PA mode. > >> > > > > > > Any help will be much appreciated. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > > > > > > !! External Email: This email originated from outside of the > > organization. Do not click links or open attachments unless you > > recognize the sender. > > > </blockquote></div>