<div dir="ltr">Thank you for the advice on that. I have removed the reset, and I also fixed the rte_eth_dev_get_supported_ptypes(). I moved my rte_eth_dev_get_supported_ptypes() calls after the bypass interface was configured, and now they're returning expected values.<div><br></div><div>My next issue I'm running into is the bonding issue that I talked about in my initial post. It seems that when part of a kernel bond, the mlx5_core driver combines both ports into a single 'verbs' device named 'mlx5_bond_0'. On my old setup with igb_uio, it worked like this:</div><div><br></div><div>- At Linux boot, the 2x Intel ports were configured as a bond on the kernel via /etc/network/interfaces file.</div><div>- Before starting DPDK software, both ports bound to igb_uio driver.</div><div>- Software started, ports are setup by my software as kernel bypass.</div><div>- Enslaved the 2x 'new' bypass interfaces onto the same bond.</div><div>- Software reinjected LACP packets into kernel, to let kernel handle LACP protocol.</div><div><br></div><div>This behavior where both ports are 'combined' into a single verbs device is strange to me. How should I handle this? Is there any way to disable it and just have the 2 ports be separate interfaces?</div><div><ol class="gmail-scrollerInner_e2e187" aria-label="Messages in" role="list" tabindex="0" style="margin:0px;padding:0px;border:0px;font-family:"gg sans","Noto Sans","Helvetica Neue",Helvetica,Arial,sans-serif;font-size:16px;vertical-align:baseline;list-style:none;min-height:0px;overflow:hidden;color:rgb(0,0,0)"><li id="gmail-chat-messages-792873248794738688-1307041988356341932" class="gmail-messageListItem_d5deea" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:none"><div class="gmail-message_d5deea gmail-cozyMessage_d5deea gmail-wrapper_f9f2ca gmail-cozy_f9f2ca gmail-zalgo_f9f2ca" role="article" tabindex="-1" aria-labelledby="message-username-1307041700685807727 uid_1 message-content-1307041988356341932 uid_2 message-timestamp-1307041988356341932" style="margin:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;min-height:1.375rem;padding-right:48px"><div class="gmail-contents_f9f2ca" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px"></div></div></li><li id="gmail-chat-messages-792873248794738688-1307041976520278128" class="gmail-messageListItem_d5deea" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:none"><div class="gmail-message_d5deea gmail-cozyMessage_d5deea gmail-wrapper_f9f2ca gmail-cozy_f9f2ca gmail-zalgo_f9f2ca" role="article" tabindex="-1" aria-labelledby="message-username-1307041700685807727 uid_1 message-content-1307041976520278128 uid_2 message-timestamp-1307041976520278128" style="margin:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;min-height:1.375rem;padding-right:48px"><div class="gmail-contents_f9f2ca" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px"><div id="gmail-message-content-1307041976520278128" class="gmail-markup_f8f345 gmail-messageContent_f9f2ca" style="margin-top:0px;margin-right:0px;margin-bottom:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;font-size:1rem;vertical-align:baseline;outline:0px;line-height:1.375rem;overflow:hidden"><pre style="margin-top:6px;margin-bottom:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-size:0.75rem;vertical-align:baseline;box-sizing:border-box;border-radius:4px;line-height:1rem;max-width:90%;background-clip:border-box"><div class="gmail-codeContainer_f8f345" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px"><code class="gmail-scrollbarGhostHairline_c858ce gmail-scrollbar_c858ce gmail-hljs" style="margin:0px;padding:0.5em;font-weight:inherit;font-style:inherit;font-size:0.875rem;vertical-align:baseline;line-height:1.125rem;display:block;overflow-x:auto;border-radius:4px">root@DDoSMitigation:~/anubis/engine/bin# ibv_devinfo hca_id: mlx5_bond_0 transport: InfiniBand (0) fw_ver: 16.35.4030 node_guid: 506b:4b03:00b6:76ec sys_image_guid: 506b:4b03:00b6:76ec vendor_id: 0x02c9 vendor_part_id: 4119 hw_ver: 0x0 board_id: MT_0000000090 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet</code></div></pre></div></div><div class="gmail-buttonContainer_f9f2ca" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px"><div class="gmail-buttons_d5deea gmail-container_a3b500" role="group" aria-label="Message Actions" style="margin:0px;padding:0px 14px 0px 32px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;opacity:1"><div class="gmail-buttonsInner_d5deea gmail-popover_e986d9 gmail-popoverReactionHoverBar_e986d9 gmail-wrapper_f7e168" style="margin:0px;padding:2px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:grid;box-sizing:border-box;height:auto;border-radius:8px;overflow:visible"><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Click to react with thumbsup" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"><div class="gmail-icon_e986d9 gmail-buttonContent_e986d9" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;width:20px;height:20px;display:flex;object-fit:contain"><div class="emoji emoji_a3b500" role="img" aria-label="" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:bottom;outline:0px;object-fit:contain;width:20px;height:20px;background-image:url("/assets/cb58aef2857ab846f41d.svg");background-size:contain;background-repeat:no-repeat;background-position:50% 50%"></div></div></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Click to react with joy" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"><div class="gmail-icon_e986d9 gmail-buttonContent_e986d9" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;width:20px;height:20px;display:flex;object-fit:contain"><div class="emoji emoji_a3b500" role="img" aria-label="" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:bottom;outline:0px;object-fit:contain;width:20px;height:20px;background-image:url("/assets/af51e8d03e05a1c14355.svg");background-size:contain;background-repeat:no-repeat;background-position:50% 50%"></div></div></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Click to react with skull_crossbones" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"><div class="gmail-icon_e986d9 gmail-buttonContent_e986d9" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;width:20px;height:20px;display:flex;object-fit:contain"><div class="emoji emoji_a3b500" role="img" aria-label="" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:bottom;outline:0px;object-fit:contain;width:20px;height:20px;background-image:url("/assets/459e176cfd1c609c0ed5.svg");background-size:contain;background-repeat:no-repeat;background-position:50% 50%"></div></div></div><div class="gmail-separator_e986d9" style="margin:2px 4px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;width:1px;height:24px;border-radius:100px"></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Add Reaction" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Edit" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="Forward" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"></div><div class="gmail-hoverBarButton_e986d9 gmail-button_f7e168" aria-label="More" aria-expanded="false" role="button" tabindex="0" style="margin:0px;padding:2px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;display:flex;height:24px;min-width:24px;border-radius:6px"></div></div></div></div></div></li><li id="gmail-chat-messages-792873248794738688-1307041988356341932" class="gmail-messageListItem_d5deea" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:none"><div class="gmail-message_d5deea gmail-cozyMessage_d5deea gmail-wrapper_f9f2ca gmail-cozy_f9f2ca gmail-zalgo_f9f2ca" role="article" tabindex="-1" aria-labelledby="message-username-1307041700685807727 uid_1 message-content-1307041988356341932 uid_2 message-timestamp-1307041988356341932" style="margin:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px;min-height:1.375rem;padding-right:48px"><div class="gmail-contents_f9f2ca" style="margin:0px;padding:0px;border:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;outline:0px"><span class="gmail-latin24CompactTimeStamp_f9f2ca gmail-timestamp_f9f2ca gmail-timestampVisibleOnHover_f9f2ca gmail-alt_f9f2ca" style="margin-top:0px;margin-bottom:0px;margin-left:0px;padding:0px;border:0px;font-style:inherit;font-family:inherit;font-size:0.6875rem;vertical-align:baseline;outline:0px;display:inline-block;height:1.375rem;opacity:0;width:56px;line-height:1.375rem;text-align:right"><span class="gmail-separator_f9f2ca" aria-hidden="true" style="opacity:0;display:inline-block">[</span>12:58 PM<span class="gmail-separator_f9f2ca" aria-hidden="true" style="opacity:0;display:inline-block">]</span></span></div></div></li></ol><br class="gmail-Apple-interchange-newline"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 15, 2024 at 5:12 AM Yasuhiro Ohara <<a href="mailto:yasu1976@gmail.com" target="_blank">yasu1976@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> I think you should try a bit more, we are here to help.<br> <br> I second Thomas's opinion. Mellanox CX5 is a well-tested NIC on DPDK,<br> and I think you can make it work in only a few more steps.<br> <br> I've never tried rte_eth_dev_reset(), and now I suspect that ENOTSUPP<br> might have made the whole NIC functions stopped.<br> IIRC the ptype was working fine on CX5 in my app too.<br> Can you comment out the rte_eth_dev_reset() ?<br> (I think it's worth to try.)<br> <br> I don't think it is so uphill, but I don't disagree with you about<br> purchasing another<br> Intel 40G NICs. It'll also work perfectly fine.<br> <br> 2024年11月15日(金) 4:16 Thomas Monjalon <<a href="mailto:thomas@monjalon.net" target="_blank">thomas@monjalon.net</a>>:<br> ><br> > 14/11/2024 17:10, CJ Sculti:<br> > > I figured out the initial issue. For some reason, having both devices in a<br> > > bond on the kernel results in only 1 of the two ports being exposed as<br> > > 'verb' devices. Previously, ibv_devinfo returned only one port. After<br> > > removing both from the bond, ibv_devinfo returns both ports, and the DPDK<br> > > application successfully takes both in. I'm still having some weird<br> > > behavior trying to create a bypass interface with these ports though. I'm<br> > > using the same code that I've been using on my Intel NICs with igb_uio for<br> > > years, but seeing weird behavior. The ports are connected to our 40Gbps<br> > > Ethernet switch, and set to link_layer: Ethernet.<br> ><br> > You should be able to make it work with kernel bonding,<br> > but I'm not sure what's wrong to do that.<br> > And it looks not a priority for you. Let's focus on the other parts.<br> ><br> ><br> > > The first thing I noticed is that rte_eth_dev_reset() fails on these<br> > > interfaces with "ENOTSUP: hardware doesn't support reset".<br> ><br> > You don't need the reset procedure with mlx5,<br> > so you can make this code optional.<br> ><br> ><br> > > Secondly, when checking ptypes, I noticed my code says these NICs are<br> > > unable to support any sort of packet detection capability (code below, all<br> > > return false). The MLX5 docs do say that all of these ptypes used here are<br> > > supported by MLX5.<br> ><br> > The supported ptypes can be checked in mlx5_dev_supported_ptypes_get() code.<br> > I don't understand why it does not work for you.<br> ><br> ><br> > > I'm just picking up a project that was left off by an older dev. It hasn't<br> > > been touched in years, but has been working fine with our Intel NICs. All<br> > > I'm trying to do is update DPDK (which is done, updated from dpdk 19.05 to<br> > > DPDK 22.11, latest version with KNI support),<br> ><br> > You don't need KNI with mlx5.<br> > That's a big benefit of mlx5 design, it is natively bifurcated:<br> > <a href="https://doc.dpdk.org/guides/howto/flow_bifurcation.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/howto/flow_bifurcation.html</a><br> ><br> ><br> > > and get it to work with our Mellanox CX5 NICs.<br> > > This is my first time working with DPDK and I'm not very<br> > > familiar. Should I expect to be able to do this without having to make a<br> > > ton of code changes, or is this going to be an uphill battle for me? If<br> > > it's the latter, I will likely just go purchase Intel NICs and give up on<br> > > this.<br> ><br> > The NICs have difference that DPDK is trying to hide.<br> > If something is not compatible you may consider it as a bug or a limitation.<br> > I think you should try a bit more, we are here to help.<br> ><br> ><br> </blockquote></div>