<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body> <div class="moz-cite-prefix">+Chris Brezovec Hi Maxime, Chris from our team is attending the DPDK Summit in Dublin this week. If you have some time available, we'd appreciate it if he could meet with you to discuss the 32bit virtio issue we are seeing. Regards, Roger Melton On 9/6/23 2:57 PM, Dave Johnson (davejo) wrote: </div> <blockquote type="cite" cite="mid:MW4PR11MB65707E45F173E52B799E5DA4D3EFA@MW4PR11MB6570.namprd11.prod.outlook.com"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style>@font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;}@font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; font-size:11.0pt; font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;}p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin-top:0in; margin-right:0in; margin-bottom:0in; margin-left:.5in; font-size:10.0pt; font-family:"Calibri",sans-serif;}.MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;}div.WordSection1 {page:WordSection1;}ol {margin-bottom:0in;}ul {margin-bottom:0in;}</style> <div class="WordSection1"> <div id="mail-editor-reference-message-container"> <div> Hi Maxime, This email is regarding the following commit: <a href="https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657</a> <o:p> </o:p> A query had been sent previously on this topic (see below) indicating this commit appears to have broken the 32-bit testpmd app and impacted one of our products that runs as a 32-bit DPDK application. We consequently backed the commit out of our product but would prefer to get a fix for it. In the earlier exchange, you had asked if we were using virtio-pci or virtio-user (we are using virtio-pci) and asked for logs which Sampath provided. It’s been a while, so let me now if you need me to send resend those logs or need any other information. <o:p> </o:p> FWIW, I reproduced this using testpmd and noticed that this part of the change seems to be the interesting part (in drivers/net/virtio/virtqueue.h): <o:p> </o:p> /**<o:p></o:p> * Return the IOVA (or virtual address in case of virtio-user) of mbuf<o:p></o:p> * data buffer.<o:p></o:p> *<o:p></o:p> * The address is firstly casted to the word size (sizeof(uintptr_t))<o:p></o:p> * before casting it to uint64_t. This is to make it work with different<o:p></o:p> * combination of word size (64 bit and 32 bit) and virtio device<o:p></o:p> * (virtio-pci and virtio-user).<o:p></o:p> */<o:p></o:p> #define VIRTIO_MBUF_ADDR(mb, vq) \<o:p></o:p> ((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) + (vq)->mbuf_addr_offset))<o:p></o:p> <o:p> </o:p> If I revert just this part of the changeset (by re-using the VIRTIO_MBUF_ADDR to return buf_iova which matches what it had used previously), then 32-bit testpmd is able to receive traffic again: <o:p> </o:p> #define VIRTIO_MBUF_ADDR(mb, vq) (mb->buf_iova)<o:p></o:p> <o:p> </o:p> Looking at the address produced by each of these, I see the address is the same except that the casting results in the upper bits getting cleared: <o:p> </o:p> Address from patch (nonworking case) = 0x58e7c900<o:p></o:p> Address using buf_iova (working case) = 0x158e7c900<o:p></o:p> ::<o:p></o:p> Address from patch (nonworking case) = 0x58e7bfc0<o:p></o:p> Address using buf_iova (working case) = 0x158e7bfc0<o:p></o:p> ::<o:p></o:p> Address from patch (nonworking case) = 0x58e7b680<o:p></o:p> Address using buf_iova (working case) = 0x158e7b680<o:p></o:p> ::<o:p></o:p> <o:p> </o:p> Regards, Dave <o:p></o:p> <div id="mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Sampath Peechu (speechu) <a class="moz-txt-link-rfc2396E" href="mailto:speechu@cisco.com"> <speechu@cisco.com></a> Date: Monday, January 30, 2023 at 3:29 PM To: Maxime Coquelin <a class="moz-txt-link-rfc2396E" href="mailto:maxime.coquelin@redhat.com"> <maxime.coquelin@redhat.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:chenbo.xia@intel.com"> chenbo.xia@intel.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:chenbo.xia@intel.com"> <chenbo.xia@intel.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org"> dev@dpdk.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:dev@dpdk.org"><dev@dpdk.org></a> Cc: Roger Melton (rmelton) <a class="moz-txt-link-rfc2396E" href="mailto:rmelton@cisco.com"> <rmelton@cisco.com></a>, Malcolm Bumgardner (mbumgard) <a class="moz-txt-link-rfc2396E" href="mailto:mbumgard@cisco.com"> <mbumgard@cisco.com></a> Subject: Re: Commit broke 32-bit testpmd app<o:p></o:p> </div> Hi Maxime,<o:p></o:p> <o:p></o:p> Could you please let us know if you got a chance to look at the debugs logs I provided?<o:p></o:p> <o:p></o:p> Thanks,<o:p></o:p> Sampath<o:p></o:p> <o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Sampath Peechu (speechu) <a class="moz-txt-link-rfc2396E" href="mailto:speechu@cisco.com"> <speechu@cisco.com></a> Date: Tuesday, December 6, 2022 at 1:08 PM To: Maxime Coquelin <a class="moz-txt-link-rfc2396E" href="mailto:maxime.coquelin@redhat.com"> <maxime.coquelin@redhat.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:chenbo.xia@intel.com"> chenbo.xia@intel.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:chenbo.xia@intel.com"> <chenbo.xia@intel.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org"> dev@dpdk.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:dev@dpdk.org"><dev@dpdk.org></a> Cc: Roger Melton (rmelton) <a class="moz-txt-link-rfc2396E" href="mailto:rmelton@cisco.com"> <rmelton@cisco.com></a> Subject: Re: Commit broke 32-bit testpmd app<o:p></o:p> </div> Hi Maxime,<o:p></o:p> <o:p></o:p> Did you get a chance to look into this?<o:p></o:p> <o:p></o:p> Please let me know if you need anything else.<o:p></o:p> <o:p></o:p> Thanks,<o:p></o:p> Sampath<o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Sampath Peechu (speechu) <a class="moz-txt-link-rfc2396E" href="mailto:speechu@cisco.com"> <speechu@cisco.com></a> Date: Wednesday, November 23, 2022 at 5:15 PM To: Maxime Coquelin <a class="moz-txt-link-rfc2396E" href="mailto:maxime.coquelin@redhat.com"> <maxime.coquelin@redhat.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:chenbo.xia@intel.com"> chenbo.xia@intel.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:chenbo.xia@intel.com"> <chenbo.xia@intel.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org"> dev@dpdk.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:dev@dpdk.org"><dev@dpdk.org></a> Cc: Roger Melton (rmelton) <a class="moz-txt-link-rfc2396E" href="mailto:rmelton@cisco.com"> <rmelton@cisco.com></a> Subject: Re: Commit broke 32-bit testpmd app<o:p></o:p> </div> Hi Maxime,<o:p></o:p> <o:p></o:p> I’m attaching the following for reference.<o:p></o:p> <o:p></o:p> <ul style="margin-top:0in" type="disc"> <li class="MsoListParagraph" style="margin-left:0in;mso-list:l1 level1 lfo3">Instructions for Centos8 test setup<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l1 level1 lfo3">Diffs between the working and non-working versions (working version has the problem commit backed out)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l1 level1 lfo3">Working logs (stats show that ping packets from neighbor VM can be seen with both 64-bit and 32-bit apps)<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l1 level1 lfo3">Non-working logs (stats show that ping packets from neighbor VM are seen with 64-bit app but NOT seen with 32-bit app)<o:p></o:p></li></ul> <o:p></o:p> <o:p></o:p> ============================ <o:p></o:p> <o:p></o:p> $ sudo ./usertools/dpdk-devbind.py --status<o:p></o:p> <o:p></o:p> Network devices using DPDK-compatible driver<o:p></o:p> ============================================<o:p></o:p> 0000:07:00.0 'Virtio network device 1041' drv=igb_uio unused=<o:p></o:p> 0000:08:00.0 'Virtio network device 1041' drv=igb_uio unused=<o:p></o:p> <o:p></o:p> Network devices using kernel driver<o:p></o:p> ===================================<o:p></o:p> 0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=igb_uio *Active*<o:p></o:p> <o:p></o:p> …<o:p></o:p> ===========================<o:p></o:p> <o:p></o:p> <o:p></o:p> Thanks,<o:p></o:p> Sampath<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Maxime Coquelin <a class="moz-txt-link-rfc2396E" href="mailto:maxime.coquelin@redhat.com"> <maxime.coquelin@redhat.com></a> Date: Tuesday, November 22, 2022 at 4:24 AM To: Sampath Peechu (speechu) <a class="moz-txt-link-rfc2396E" href="mailto:speechu@cisco.com"> <speechu@cisco.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:chenbo.xia@intel.com"> chenbo.xia@intel.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:chenbo.xia@intel.com"> <chenbo.xia@intel.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org"> dev@dpdk.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:dev@dpdk.org"><dev@dpdk.org></a> Cc: Roger Melton (rmelton) <a class="moz-txt-link-rfc2396E" href="mailto:rmelton@cisco.com"> <rmelton@cisco.com></a> Subject: Re: Commit broke 32-bit testpmd app<o:p></o:p> </div> <div> Hi, In my initial reply (see below), I also asked if you had logs to share. And wondered whether it happens with Virtio PCI or Virtio-user? Regards, Maxime On 11/16/22 00:30, Sampath Peechu (speechu) wrote: > ++ <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org">dev@dpdk.org</a> <<a href="mailto:dev@dpdk.org" moz-do-not-send="true">mailto:dev@dpdk.org</a>> > > *From: *Maxime Coquelin <a class="moz-txt-link-rfc2396E" href="mailto:maxime.coquelin@redhat.com"> <maxime.coquelin@redhat.com></a> > *Date: *Tuesday, November 15, 2022 at 3:19 AM > *To: *Sampath Peechu (speechu) <a class="moz-txt-link-rfc2396E" href="mailto:speechu@cisco.com"> <speechu@cisco.com></a>, <a class="moz-txt-link-abbreviated" href="mailto:chenbo.xia@intel.com"> chenbo.xia@intel.com</a> > <a class="moz-txt-link-rfc2396E" href="mailto:chenbo.xia@intel.com"><chenbo.xia@intel.com></a> > *Cc: *Roger Melton (rmelton) <a class="moz-txt-link-rfc2396E" href="mailto:rmelton@cisco.com"> <rmelton@cisco.com></a> > *Subject: *Re: Commit broke 32-bit testpmd app > > Hi Sampath, > > > Please add <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org">dev@dpdk.org</a>, the upstream mailing list, if this is related > to the upstream DPDK project.If it is using RHEL DPDK package, please > use the appropriate support channels. > > On 11/14/22 23:55, Sampath Peechu (speechu) wrote: > > Hi Virtio Maintainers team, > > > > This email is regarding the following commit. > > > > > <a href="https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657" moz-do-not-send="true" class="moz-txt-link-freetext"> https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657</a> <<a href="https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657</a>> <<a class="moz-txt-link-freetext" href="https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657">https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657</a> <<a href="https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/DPDK/dpdk/commit/ba55c94a7ebc386d2288d6578ed57aad6cb92657</a>>> > > > > The above commit appears to have broken the 32-bit testpmd app (and > > consequently impacted one of our products that runs as a 32-bit DPDK > > app). The 64-bit testpmd app does not appear to be impacted though. > > We'll need some logs to understand what is going on. > Does it happen with virtio-pci or virtio-user? > > Regards, > Maxime > > > With the commit in place, we didn’t see any packets going through at > > all. After backing out the commit and rebuilding the 32-bit testpmd app > > in our test setup, we were able to pass traffic as expected. > > > > Could you please let us know if this is a known issue? And if there is a > > fix available for it? > > > > Thank you, > > > > Sampath Peechu > > > > Cisco Systems > > > <o:p></o:p> </div> </div> </div> </div> </div> </div> </blockquote> </body> </html>