<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> +Ronak from the ESX team<o:p></o:p> <o:p> </o:p> <o:p> </o:p> In our usage, we do set amd_iommu=off in the boot command line from grub. (Or intel_iommu=off for Intel processors.)<o:p></o:p> <o:p> </o:p> Jochen<o:p></o:p> <o:p> </o:p> <div id="mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Thomas Monjalon <thomas@monjalon.net> Date: Thursday, July 20, 2023 at 6:00 AM To: Igor de Paula <igordptx@gmail.com> Cc: users@dpdk.org <users@dpdk.org>, Jochen Behrens <jbehrens@vmware.com>, Nipun Gupta <nipun.gupta@amd.com>, Nikhil Agarwal <nikhil.agarwal@amd.com>, Ferruh Yigit <ferruh.yigit@amd.com> Subject: Re: help<o:p></o:p> </div> <div> !! External Email +Cc some AMD maintainers, they can have an idea about IOMMU settings. 20/07/2023 14:44, Igor de Paula: > I have enabled it in the host and in the BIOS for AMD... > In the Bios I changed to amd_iommu=on and in the host it's the same for > either. > > On Thu, Jul 20, 2023 at 1:31 PM Thomas Monjalon <thomas@monjalon.net> wrote: > > > 20/07/2023 11:35, Igor de Paula: > > > The weird thing is that it only happens when I am using a host with an > > AMD > > > processor. It doesn't happen when I use a host with an Intel processor. > > > > So it's probably a matter of BIOS settings for the IOMMU? > > > > > > > On Thu, Jul 20, 2023 at 10:32 AM Thomas Monjalon <thomas@monjalon.net> > > > wrote: > > > > > > > +Cc the vmxnet3 maintainer. > > > > > > > > Please Jochen, do you have an idea what's wrong below? > > > > > > > > > > > > 20/07/2023 11:25, Igor de Paula: > > > > > This is because it can't negotiate the IOMMU type with any port. > > > > > > > > > > On Thu, Jul 20, 2023 at 5:08 AM Thomas Monjalon <thomas@monjalon.net > > > > > > > wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > The first error is "Cause: Error: number of ports must be even" > > > > > > > > > > > > > > > > > > 03/05/2023 18:13, Igor de Paula: > > > > > > > I am running a VM inside a VMWARE server (vSphere). > > > > > > > My goal it to set up DPDK with two HW ports, and set up a > > > > virtio_user to > > > > > > > interact with the kernel stack. > > > > > > > In another app I have it working but instead of virtio_user I am > > > > running > > > > > > > KNI, it works in IOVA-PA mode. > > > > > > > I am looking to replace the KNI. > > > > > > > > > > > > > > When I try to set up virtio_user port as in the doc: > > > > > > > > > > > > > > > > > > > <a href="https://doc.dpdk.org/guides/howto/virtio_user_as_exception_path.html#virtio-user-as-exception-path">https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fhowto%2Fvirtio_user_as_exception_path.html%23virtio-user-as-exception-path&data=05%7C01%7Cjbehrens%40vmware.com%7Cfbb1f6df362e4537d62608db8921432f%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638254548229277606%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3NuWkbtQx2q8NJ2CzArNfuDjP%2FYIVv1kMc%2Bo4IiHlmM%3D&reserved=0</a> > > > > > > > I get a error it can't run in PA mode. > > > > > > > > > > > > > > > > > > > > > When I try to run as VA mode from a parameter, I get the > > following > > > > > > errors: > > > > > > > EAL: lib.eal log level changed from info to debug > > > > > > > EAL: Detected lcore 0 as core 0 on socket 0 > > > > > > > EAL: Detected lcore 1 as core 0 on socket 0 > > > > > > > EAL: Support maximum 128 logical core(s) by configuration. > > > > > > > EAL: Detected 2 lcore(s) > > > > > > > EAL: Detected 1 NUMA nodes > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21.3' > > > > > > > EAL: Checking presence of .so 'librte_eal.so.21' > > > > > > > EAL: Checking presence of .so 'librte_eal.so' > > > > > > > EAL: Detected static linkage of DPDK > > > > > > > EAL: Ask a virtual area of 0x7000 bytes > > > > > > > EAL: Virtual area found at 0x100000000 (size = 0x7000) > > > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > > > EAL: DPAA Bus not present. Skipping. > > > > > > > EAL: VFIO PCI modules not loaded > > > > > > > EAL: Selected IOVA mode 'VA' > > > > > > > EAL: Probing VFIO support... > > > > > > > EAL: IOMMU type 1 (Type 1) is supported > > > > > > > EAL: IOMMU type 7 (sPAPR) is not supported > > > > > > > EAL: IOMMU type 8 (No-IOMMU) is supported > > > > > > > EAL: VFIO support initialized > > > > > > > EAL: Ask a virtual area of 0x5b000 bytes > > > > > > > EAL: Virtual area found at 0x100007000 (size = 0x5b000) > > > > > > > EAL: Setting up physically contiguous memory... > > > > > > > EAL: Setting maximum number of open files to 1048576 > > > > > > > EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 > > > > > > > EAL: Creating 2 segment lists: n_segs:128 socket_id:0 > > > > > > hugepage_sz:1073741824 > > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > > EAL: Virtual area found at 0x100062000 (size = 0x2000) > > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > > > > EAL: Virtual area found at 0x140000000 (size = 0x2000000000) > > > > > > > EAL: VA reserved for memseg list at 0x140000000, size 2000000000 > > > > > > > EAL: Ask a virtual area of 0x2000 bytes > > > > > > > EAL: Virtual area found at 0x2140000000 (size = 0x2000) > > > > > > > EAL: Memseg list allocated at socket 0, page size 0x100000kB > > > > > > > EAL: Ask a virtual area of 0x2000000000 bytes > > > > > > > EAL: Virtual area found at 0x2180000000 (size = 0x2000000000) > > > > > > > EAL: VA reserved for memseg list at 0x2180000000, size 2000000000 > > > > > > > EAL: TSC frequency is ~2350000 KHz > > > > > > > EAL: Main lcore 0 is ready (tid=7f8ad790ec00;cpuset=[0]) > > > > > > > EAL: lcore 1 is ready (tid=7f8ad6907400;cpuset=[1]) > > > > > > > EAL: Trying to obtain current memory policy. > > > > > > > EAL: Setting policy MPOL_PREFERRED for socket 0 > > > > > > > EAL: Restoring previous memory policy: 0 > > > > > > > EAL: request: mp_malloc_sync > > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > > > > initializing > > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > > EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not > > > > > > initializing > > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > > > > > > > EAL: Bus (pci) probe failed. > > > > > > > EAL: lib.telemetry log level changed from disabled to warning > > > > > > > EAL: Error - exiting with code: 1 > > > > > > > Cause: Error: number of ports must be even > > > > > > > EAL: request: mp_malloc_sync > > > > > > > EAL: Heap on socket 0 was shrunk by 1024MB > > > > > > > > > > > > > > > > > > > > > > > > > > > > For some reason the HW ports won't setup. From what I understand > > > > > > > net_vmxnet3 should work with VA mode. > > > > > > > I enabled I/OMUU for the VM. > > > > > > > The weird thing even when enabled, I still have the > > > > > > > enable_unsafe_noiommu_mode flag on. > > > > > > > And because it's on the this: > > > > > > > > > > > > > > dev_iova_mode = pci_device_iova_mode(dr, dev); > > > > > > > > > > > > > > return PA mode, and it fails. > > > > > > > > > > > > > > When I disable it by modifying > > > > > > > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode, I get > > another > > > > > > error. > > > > > > > The error is that it doesn't find a suitable IOMMU type: > > > > > > > Just putting the relevant message: > > > > > > > > > > > > > > > > > > > > > EAL: Heap on socket 0 was expanded by 1024MB > > > > > > > EAL: PCI device 0000:0b:00.0 on NUMA socket -1 > > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such > > device) > > > > > > > EAL: 0000:0b:00.0 failed to select IOMMU type > > > > > > > EAL: Requested device 0000:0b:00.0 cannot be used > > > > > > > EAL: PCI device 0000:13:00.0 on NUMA socket -1 > > > > > > > EAL: probe driver: 15ad:7b0 net_vmxnet3 > > > > > > > EAL: Set IOMMU type 1 (Type 1) failed, error 19 (No such device) > > > > > > > EAL: Set IOMMU type 7 (sPAPR) failed, error 19 (No such device) > > > > > > > EAL: Set IOMMU type 8 (No-IOMMU) failed, error 19 (No such > > device) > > > > > > > EAL: 0000:13:00.0 failed to select IOMMU type > > > > > > > EAL: Requested device 0000:13:00.0 cannot be used > > > > > > > > > > > > > > > > > > > > > I can see where it fails: > > > > > > > > > > > > > > > > > > > > > const struct vfio_iommu_type * > > > > > > > vfio_set_iommu_type(int vfio_container_fd) > > > > > > > { > > > > > > > unsigned idx; > > > > > > > for (idx = 0; idx < RTE_DIM(iommu_types); idx++) { > > > > > > > const struct vfio_iommu_type *t = &iommu_types[idx]; > > > > > > > > > > > > > > int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, > > > > > > > t->type_id); > > > > > > > if (!ret) { > > > > > > > RTE_LOG(INFO, EAL, "Using IOMMU type %d (%s)\n", > > > > > > > t->type_id, t->name); > > > > > > > return t; > > > > > > > } > > > > > > > /* not an error, there may be more supported IOMMU types */ > > > > > > > RTE_LOG(DEBUG, EAL, "Set IOMMU type %d (%s) failed, error " > > > > > > > "%i (%s)\n", t->type_id, t->name, errno, > > > > > > > strerror(errno)); > > > > > > > } > > > > > > > /* if we didn't find a suitable IOMMU type, fail */ > > > > > > > return NULL; > > > > > > > } > > > > > > > > > > > > > > > > > > > > > The ioctl returns -1. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I tried many things. > > > > > > > Enabling/disabling IOMMU. > > > > > > > Modifying the enable_unsafe_noiommu_mode flag. > > > > > > > Running in PA mode. > > > > > > > Any help will be much appreciated. > > > > > > > > > > > > > !! External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.<o:p></o:p> </div> </div> </div> </div> </body> </html>