[dpdk-dev] mlx5 under FreeBSD

Mit Matelske mit at pt.net
Wed Mar 6 22:01:39 CET 2019


> > 
> > dev.mlx5_core.1.hw.board_id: LNR3270110033
> > dev.mlx5_core.1.hw.fw_version: 12.17.2020
>
> Getting back to you on this one.
>
> From the FW it looks like you are trying to create a TIR (QP) which doesn't use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key is not reserved. 
> 
> Makes sense? 

Kinda, except I believe the DPDK code is trying to create a TIR with IBV_RX_HASH_FUNC_TOEPLITZ set and the RSS key defined:


   qp = mlx5_glue->create_qp_ex(priv->ctx,
       &(struct ibv_qp_init_attr_ex){
         .qp_type = IBV_QPT_RAW_PACKET,
         .comp_mask =
            IBV_QP_INIT_ATTR_PD |
            IBV_QP_INIT_ATTR_IND_TABLE |
            IBV_QP_INIT_ATTR_RX_HASH,
         .rx_hash_conf = (struct ibv_rx_hash_conf){
            .rx_hash_function =
               IBV_RX_HASH_FUNC_TOEPLITZ,
            .rx_hash_key_len = MLX5_RSS_HASH_KEY_LEN,
            .rx_hash_key = rss_hash_default_key,
            .rx_hash_fields_mask = 0,
            },
         .rwq_ind_tbl = ind_tbl->ind_table,
         .pd = priv->pd
       });


That call goes into create_qp in libmlx5 which eventually makes the following call:


  ret = ibv_cmd_create_qp_ex2(context, &qp->verbs_qp,
                   sizeof(qp->verbs_qp), attr,
                   &cmd_ex_rss.ibv_cmd, sizeof(cmd_ex_rss.ibv_cmd),
                   sizeof(cmd_ex_rss), &resp.ibv_resp,
                   sizeof(resp.ibv_resp), sizeof(resp));

Relevant variables from that call:

cmd_ex_rss.rx_hash_function: 1
cmd_ex_rss.rx_key_len: 40
cmd_ex_rss.rx_hash_key:
2C C6 81 D1 5B DB F4 F7 FC A2 
83 19 DB 1A 3E 94 6B 9E 38 D9 
2C 9C 03 D1 AD 99 44 A7 D9 56 
3D 59 06 3C 25 F3 FC 1F DC 2A 

All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.  

Any and all help is appreciated.  I'm a little out of my depths with this.

Thanks!

Mit Matelske

> 
> Thanks!
> 
> Mit Matelske
> 
> ----- Original Message -----
> From: "shahafs" <shahafs at mellanox.com>
> To: "Mit Matelske" <mit at pt.net>, "stephen"
> <stephen at networkplumber.org>
> Cc: "Thomas Monjalon" <thomas at monjalon.net>, "dev" <dev at dpdk.org>,
> "Yongseok Koh" <yskoh at mellanox.com>
> Sent: Sunday, March 3, 2019 2:43:41 AM
> Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> 
> Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > I was wondering if I could lean on the experts to help me out with an issue?
> > I'm finally testing this port but don't understand why the following
> > error is
> > happening:
> >
> > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > allocate QP for drop queue
> >
> > The driver is giving the following error corresponding with that DPDK
> > message:
> >
> > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > mlx5_cmd_check:701:(pid
> > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > parameter(0x3), syndrome (0x569ee6)
> 
> Hard to say why. I tried to look for the syndrome above on FW but couldn't
> find. Are you sure this is correct one?
> Which FW version you use?
> 
> >
> > The mlx5_core_create_tir function completes successfully for the en
> > driver, just not the ib driver when trying to build the hash tables.  Any
> ideas?
> >
> > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this
> > and everything looks fine according to my limited knowledge.  Not sure
> > why the card is balking on this action.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > ----- Original Message -----
> > From: "stephen" <stephen at networkplumber.org>
> > To: "Mit Matelske" <mit at pt.net>
> > Cc: "shahafs" <shahafs at mellanox.com>, "Thomas Monjalon"
> > <thomas at monjalon.net>, "dev" <dev at dpdk.org>, "Yongseok Koh"
> > <yskoh at mellanox.com>
> > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> >
> > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit at pt.net>
> > wrote:
> >
> > > >
> > > > Rather than building a complex shim, why not just have an OS
> > > > dependent section of the driver.  The bigger issue is that the
> > > > kernel side functionality in BSD is different or missing.  For
> > > > example TAP
> > API is different and not multi-queue.
> > >
> > > Once I started working on it, I had to go this route.  I haven't
> > > been able to but much time into it yet, but it is coming along:
> > >
> > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > /sys/class/infiniband/mlx5_0
> > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for
> > device "mlx5_0"
> > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > /sys/class/infiniband/mlx5_1
> > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected
> > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled
> > > due to old OFED/rdma-core version
> > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel
> > > offloading disabled due to old OFED/rdma-core version or firmware
> > > configuration
> > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > "0000:b3:00.0"
> > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > supported
> > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > table size is 512
> > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > supported
> > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration
> > > is not supported
> > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment
> > > padding is not supported
> > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR
> > > address space: 0x880000000
> > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > 00:90:0b:78:2a:ea
> > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > switch offloads will not be supported: cannot open libmnl socket:
> > > Address family not supported by protocol family
> > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > interface up
> > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port
> > > 0
> > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > allocate QP for drop queue Segmentation fault (core dumped)
> > >
> > > Working through mlx5_ethdev now...
> > >
> > > Mit
> >
> > Also if you are targeting Azure or Hyper-V, then don't bother porting
> > TAP/VDEV_NETVSC/Failsafe model because that is being replaced by
> > netvsc native PMD.  It would be easier to just get that working.
> > It would mean writing a  uio equivalent driver for FreeBSD.


More information about the dev mailing list