[dpdk-users] Building mlx5 with rdma-core/OFED

[Gerry Wan]

How did you install these packages? Are there extra environment variables I
need to set?
After running the OFED uninstall script,

apt-get install rdma-core
apt-get install libibverbs-dev

both work fine, but ninja install now fails on

[1/1756] Linking target drivers/librte_common_mlx5.so.21.1.
FAILED: drivers/librte_common_mlx5.so.21.1
cc  -o drivers/librte_common_mlx5.so.21.1 'drivers/a715181@
@rte_common_mlx5 at sha/meson-generated_.._rte_common_mlx5.pmd.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_devx_cmds.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_common.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_common_mp.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_common_mr.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_malloc.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_common_pci.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_mlx5_common_devx.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_nl.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_common_os.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_common_verbs.c.o'
'drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o'
-Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group
-Wl,-soname,librte_common_mlx5.so.21 -Wl,--no-as-needed -pthread -lm -ldl
-lnuma lib/librte_hash.so.21.1 lib/librte_eal.so.21.1
lib/librte_kvargs.so.21.1 lib/librte_telemetry.so.21.1
lib/librte_ring.so.21.1 lib/librte_rcu.so.21.1 lib/librte_pci.so.21.1
drivers/librte_bus_pci.so.21.1 lib/librte_net.so.21.1
lib/librte_mbuf.so.21.1 lib/librte_mempool.so.21.1
-Wl,--version-script=/home/gerryw/dpdk/dpdk-21.02/drivers/common/mlx5/version.map
/usr/lib/x86_64-linux-gnu/libmlx5.so
/usr/lib/x86_64-linux-gnu/libibverbs.so -Wl,--end-group
'-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/'
-Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/lib
-Wl,-rpath-link,/home/gerryw/dpdk/dpdk-21.02/build/drivers
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_create_flow_action_aso':
mlx5_glue.c:(.text+0x21f): undefined reference to
`mlx5dv_dr_action_create_aso'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_action_create_dest_array':
mlx5_glue.c:(.text+0x23b): undefined reference to
`mlx5dv_dr_action_create_dest_array'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_sampler':
mlx5_glue.c:(.text+0x25b): undefined reference to
`mlx5dv_dr_action_create_flow_sampler'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_alloc_pp':
mlx5_glue.c:(.text+0x28b): undefined reference to `mlx5dv_pp_alloc'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_default_miss':
mlx5_glue.c:(.text+0x4bb): undefined reference to
`mlx5dv_dr_action_create_default_miss'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_free_pp':
mlx5_glue.c:(.text+0x277): undefined reference to `mlx5dv_pp_free'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_reclaim_domain_memory':
mlx5_glue.c:(.text+0x2a7): undefined reference to
`mlx5dv_dr_domain_set_reclaim_device_memory'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_devx_port_query':
mlx5_glue.c:(.text+0x325): undefined reference to `mlx5dv_query_devx_port'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dv_create_flow_action_dest_devx_tir':
mlx5_glue.c:(.text+0x545): undefined reference to
`mlx5dv_dr_action_create_dest_devx_tir'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_pop_vlan':
mlx5_glue.c:(.text+0x655): undefined reference to
`mlx5dv_dr_action_create_pop_vlan'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_push_vlan':
mlx5_glue.c:(.text+0x665): undefined reference to
`mlx5dv_dr_action_create_push_vlan'
/usr/bin/ld: drivers/a715181@@tmp_rte_common_mlx5 at sta/common_mlx5_linux_mlx5_glue.c.o:
in function `mlx5_glue_dr_create_flow_action_dest_port':
mlx5_glue.c:(.text+0x685): undefined reference to
`mlx5dv_dr_action_create_dest_ib_port'
collect2: error: ld returned 1 exit status
[2/1756] Compiling C object 'drivers/a715181@@tmp_rte_mempool_octeontx at sta
/mempool_octeontx_octeontx_fpavf.c.o'.
ninja: build stopped: subcommand failed.


On Mon, Aug 2, 2021 at 6:24 AM PATRICK KEROULAS <
patrick.keroulas at radio-canada.ca> wrote:

>
> Hello Gerry,
> I went through this a while ago and had to properly remove everything
> related
> to OFED with the uninstaller. Then, install rdma-core and libibverbs-dev
> which
> includes infiniband/mlx5dv.h IIRC.
> Regards,
> Patrick
>
> On Sun, Aug 1, 2021 at 8:23 PM Gerry Wan <gerryw at stanford.edu> wrote:
>
>> I recently tried to reinstall MLX OFED 5.2-2.2.0.0 on my bare metal
>> machine
>> Ubuntu 20.04 (previously using the command ./mlnxofedinstall --dpdk
>> --upstream-libs --with-mft --with-kernel-mft). However, it now fails with
>> the message
>>
>> Checking SW Requirements...
>> Removing old packages...
>> Uninstalling the previous version of MLNX_OFED_LINUX
>> Installing new packages Installing mft-4.16.1...
>> Installing ofed-scripts-5.2...
>> Installing mstflint-4.15.0...
>> Installing mlnx-ofed-kernel-utils-5.2...
>> Installing mlnx-ofed-kernel-dkms-5.2...
>> Failed to install mlnx-ofed-kernel-dkms DEB
>> Collecting debug info...
>> See /tmp/MLNX_OFED_LINUX.9221.logs/mlnx-ofed-kernel-dkms.debinstall.log
>>
>> It appears that my kernel version got upgraded to 5.4.0-80-generic which I
>> think may be the issue. I've also tried to install using
>> --add-kernel-support --kernel 5.4.0-80-generic, with no luck.
>>
>> The DPDK website says that rdma-core is now recommended over OFED, but the
>> documentation for how to use it isn't very clear to me. I've tried to
>> install it via
>>
>> apt-get install rdma-core
>>
>> as well as running the quick start script listed here:
>> https://github.com/linux-rdma/rdma-core, but DPDK installation still
>> fails
>> with
>>
>> ../drivers/common/mlx5/linux/mlx5_glue.h:15:10: fatal error:
>> infiniband/mlx5dv.h: No such file or directory
>> 15 | #include <infiniband/mlx5dv.h>
>>
>> Any advice on getting the mlx5 drivers to work again would be helpful!
>>
>