[dpdk-dev] [PATCH] doc: update mlx4 usage and dependencies

[Raghav Sethi]

Hi Adrien,

Quick update: So I just tried the setup again on a new machine with the
exact same hardware - installed OFED and compiled DPDK 2.0.0. Works like a
charm.

Still getting the issue on the old machine (also upgraded to 2.0.0 FWIW).
>From the error message it appears that the libraries are incorrectly
installed/configured. However, the OFED uninstall script removes the
libraries in question
(libmlx5-rdmav2.so, libibverbs/libmlx4-rdmav2.so,  libibverbs.so), I
checked! In fact, I diffed the libibverbs.so from both machines, and
they're exactly identical. So it seems either the OFED
installation/uninstallation is somehow flawed, or the error message from
testpmd is misleading.

Best,
Raghav

On Fri, Apr 3, 2015 at 12:52 PM Raghav Sethi <raghavs at cs.princeton.edu>
wrote:

> Hi Adrien,
>
> I've copied dev at dpdk.
>
> I re-installed Mellanox OFED, and then re-compiled DPDK without the flags
> for the static library. I'm running Ubuntu 14.04. However, I still get the
> same error message (below).
>
> I then tried re-installing the Mellanox libraries on dpdk.org, and
> compiled another copy of DPDK with them statically linked. This appears to
> work perfectly. I no longer get the permission issues I was getting before,
> possibly that was unrelated/fixed by OFED. Now both OFED, and OFED+
> dpdk.org libs statically linked have the same issue:
>
> EAL: PCI device 0000:06:00.0 on NUMA socket 0
> EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
> /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv, version
> IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
> libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
> /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
> version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
> reference
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
> EAL: No probed ethernet devices
>
> The OFED install was successful (and it apparently removes/overwrites old
> versions of the libraries). The installer also checks the firmware
> versions, and it says I'm on the latest. Loading the kernel modules is also
> successful, so I can't for the life of me figure out what this issue is now.
>
> Again, just to be clear, the full list of steps was:
> 1. Install OFED, run /etc/init.d/openibd restart
> 2. Compile DPDK with the CONFIG_RTE_LIBRTE_MLX4_PMD=y
> 3. Set up hugepages
> 4. Load kernel modules
> 5. Run testpmd
>
> Best,
> Raghav
>
> On Fri, Apr 3, 2015 at 3:34 AM Adrien Mazarguil <
> adrien.mazarguil at 6wind.com> wrote:
>
>> Hi Raghav,
>>
>> On Thu, Apr 02, 2015 at 10:05:57PM +0000, Raghav Sethi wrote:
>> > Hi Adrien,
>> >
>> > Thanks for your reply - I hope it's OK that I'm emailing you off-list.
>> I'm
>> > a grad student at Princeton working on a research project.
>>
>> It's fine, but let's discuss on the ML as much as possible next time so
>> the
>> community can benefit from it. Since very few people use this driver at
>> the
>> moment (DPDK 2.0 is not even out yet), I have my own questions for you.
>>
>> > Given your patch (which was very helpful, thanks), here is the series of
>> > all steps that I'm taking to run testpmd on a ConnectX-3 EN 40G
>> >
>> > 1. Compile and install the mellanox-specific libraries listed here:
>> > http://dpdk.org/download/mlx4/2015-03-26-DPDK-v2.0.0/
>>
>> This is fine, but did you use the provided Makefile to compile these
>> libraries statically? If so, no issues?
>>
>> Also you did not install Mellanox OFED? You should, because updated kernel
>> modules come from that distribution. It won't work properly (or at all)
>> without them.
>>
>> The libraries from DPDK.org are actually optional once you've installed
>> Mellanox OFED. They only improve performance. Remember to update firmwares
>> also.
>>
>> > 2. export EXTRA_CFLAGS and EXTRA_LDFLAGS
>> > 3. Create a new configuration which includes
>> > the CONFIG_RTE_LIBRTE_MLX4_PMD=y flag.
>> > 4. Compile DPDK 2.0 with the new configuration using setup.sh
>> > 5. Set up hugepages using setup.sh
>> > 6. Load the kernel modules ib_uverbs mlx4_en mlx4_core mlx4_ib
>> > 7. Run testpmd with the right PCI bus address
>> >
>> > However, I consistently get:
>> > sudo: unable to execute ./testpmd: Permission denied
>>
>> Now that doesn't make sense, isn't that file missing permissions or
>> something? Are you running RHEL with a restrictive SELinux policy?
>> Is sudo configured to allow you to run testpmd?
>>
>> In any case, I don't think it's related to librte_pmd_mlx4.
>>
>> > I tried taking the interface down with ifconfig because
>> > (./tools/dpdk_nic_bind.py --status shows the interface as active), but
>> that
>> > didn't change anything.
>>
>> No, you should leave the interfaces up, it's not supposed to disappear,
>> the
>> PMD will bring them back up anyway. Actually the Linux kernel interface
>> remains fully functional even while DPDK applications are running. You can
>> send packets out of it but as long as both the kernel and DPDK
>> applications
>> share the same MAC address, only DPDK applications will receive replies.
>> If
>> you change the MAC address on the kernel interface however, you'll be able
>> to use it normally.
>>
>> > I tried installing the mellanox libraries globally, but then I see these
>> > error messages:
>> >
>> > -------
>> > EAL: PCI device 0000:06:00.0 on NUMA socket 0
>> > EAL:   probe driver: 15b3:1003 librte_pmd_mlx4
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx5':
>> > /usr/lib/libibverbs/libmlx5-rdmav2.so: symbol ibv_exp_cmd_getenv,
>> version
>> > IBVERBS_1.1 not defined in file libibverbs.so.1 with link time reference
>> > libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
>> > /usr/lib/libibverbs/libmlx4-rdmav2.so: symbol ibv_exp_cmd_destroy_flow,
>> > version IBVERBS_1.1 not defined in file libibverbs.so.1 with link time
>> > reference
>> > libibverbs: Warning: no userspace device-specific driver found for
>> > /sys/class/infiniband_verbs/uverbs0
>> > PMD: librte_pmd_mlx4: cannot access device, is mlx4_ib loaded?
>> > EAL: No probed ethernet devices
>> > Interactive-mode selected
>> > EAL: Error - exiting with code: 1
>> >   Cause: Creation of mbuf pool for socket 0 failed
>> > --------
>>
>> That's because the libibverbs version you're using is not compatible with
>> the userspace drivers installed on your system (libibverbs uses dlopen()
>> to
>> load liblmx4). Even though you've installed both from DPDK.org, seems you
>> have another version installed by your distribution packages.
>>
>> If you intend to use dynamic libraries, you should uninstall libibverbs,
>> libmlx4 and related packages first.
>>
>> Regards,
>>
>> --
>> Adrien Mazarguil
>> 6WIND
>>
>