[dpdk-users] [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC association to Numa node

[Bruce Richardson]

Moving to users list, since that is more relevant than dev.

On Thu, Feb 08, 2018 at 07:36:30PM +0100, Victor Huertas wrote:
> Bruce,
> 
> 
> My requirements are not that much (500 Mbps and 1 Gbps desirable).
> 
> 
> Thanks for your references links I will have a look at them.
> 
> 
> Regarding the NIC detection in the DPDK app by the DPDK EAL initialization
> after successfully having loaded the vfio-pci, it happens something strange.
> 
> The nb_ports = rte_eth_dev_count(); is always returning 0.
> 
> 
> Therefore it shows an error telling that "port 0 is not present on the
> board".
> 
> 
> The EAL seems to detect the VFIOs as the only two logs that shows when
> initializing regarding VFIO are:
> 
> 
> EAL: Probing VFIO support...
> 
> EAL: VFIO support initialized
> 
> 
> And nothing else... shouldn't the EAL detect the two NICs I associated to
> the VFIO? very strange...
> 
> 

Yes, it should. Can you please send full output on startup.
Also, can you try using igb_uio module instead of vfio and see if that
works for you.

Regards,
/Bruce

> Regards,
> 
> 
> PD: I have changed the email address account in order to avoid sending
> these disturbing disclaimers. Sorry.
> 
> 
> -----Mensaje original-----
> De: Bruce Richardson [mailto:bruce.richardson at intel.com]
> Enviado el: jueves, 08 de febrero de 2018 17:42
> Para: Huertas García, Víctor
> CC: dev at dpdk.org
> Asunto: Re: [dpdk-dev] Nehalem Intel Xeon X5506 architecture and PCIe NIC
> association to Numa node
> 
> 
> 
> On Thu, Feb 08, 2018 at 04:27:36PM +0000, Huertas García, Víctor wrote:
> 
> >
> 
> > Hi all,
> 
> >
> 
> > After having tried many ways to make the PCIe NIC card appear associated
> to a numa node, I haven't been able to do it.
> 
> > That is, every time I try to look at which numa node belongs it always
> returns -1.
> 
> >
> 
> > $ cat /sys/bus/pci/devices/0000\:04\:00.1/numa_node
> 
> > -1
> 
> >
> 
> > $ cat /sys/bus/pci/devices/0000\:04\:00.0/numa_node
> 
> > -1
> 
> >
> 
> > Using lstopo, I confirm that all PCI cards are "outside" of  any Numa
> node.
> 
> >
> 
> > I have read in previous posted messages in dpdk-dev community that this
> is normal in Nehalem generation Xeon architecture and there is nothing I
> can do about it. Can somebody confirm this?
> 
> 
> 
> For that generation architecture, it is indeed expected. The NICs are not
> directly connected to any NUMA node.
> 
> 
> 
> > If so, what implications could this have on packet capture and
> performance?
> 
> 
> 
> Unsurprisingly, it's the case that newer platforms will perform better, as
> you are missing out on performance benefits from improved cores and also
> features like Intel® DDIO [1].
> 
> However, what I/O throughput are you hoping to get from your system?
> 
> Depending on your requirements, what you have may be enough. Some people
> use DPDK on lower-end platforms because that is all that they need. You may
> also find the chart on slide 6 of [2] of use to see how the max throughput
> of a platform has improved over time (and has improved further since that
> chart was published).
> 
> 
> 
> >
> 
> > Are the NICs available in my DPDK applications? Do I have to specifically
> "add" them by "-w 04:00.1 - w 04:00.0"?
> 
> 
> 
> Yes, your NICs will still be available, even without NUMA affinity, and no,
> you should not need to explicitly whitelist them - though you can if you
> want. So long as they are bound to a uio or vfio driver (e.g.
> 
> igb_uio or vfio-pci), they should be detected by DPDK EAL init and made
> available to your app.
> 
> 
> 
> > Is RSS supported and usable from the DPDK application?
> 
> 
> 
> Yes, at least for Intel NICs, and probably most other DPDK-supported NICs
> too.
> 
> 
> 
> >
> 
> > Thanks a lot for your attention
> 
> >
> 
> > Victor
> 
> >
> 
> 
> 
> /Bruce
> 
> 
> 
> [1] https://www.intel.com/content/www/us/en/io/data-direct-i-o-t
> echnology.html
> 
> [2] https://dpdksummit.com/Archive/pdf/2016Germany/DPDK-2016-
> DPDK_FD_IO_Introduction.pdf
> 
> 
> 
> PS: This is a public list, so email disclaimers are rather pointless.
> 
> It's best if they can be removed from mails sent here.
> 
> 
> 
> 
> -- 
> Victor