<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word"> <div class="WordSection1"> <o:p></o:p> Maxime,<o:p></o:p> <o:p> </o:p> Do you think you might be able to look at the info below (probably got lost in the many emails this last week / weekend).<o:p></o:p> <o:p> </o:p> Kind regards,<o:p></o:p> -ChrisB<o:p></o:p> <o:p> </o:p> <div id="mail-editor-reference-message-container"> <div> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Chris Brezovec (cbrezove) <cbrezove@cisco.com> Date: Saturday, November 2, 2024 at 12:00 PM To: dev@dpdk.org <dev@dpdk.org>, maxime.coquelin@redhat.com <maxime.coquelin@redhat.com> Cc: Roger Melton (rmelton) <rmelton@cisco.com>, Walt Robinson (walrobin) <walrobin@cisco.com> Subject: Re: 32-bit virtio failing on DPDK v23.11.1 (and tags)<o:p></o:p> </div> <div> <o:p></o:p> Hi Maxime / team,<o:p></o:p> <o:p></o:p> I have been going through the 12+ virtio commits between the last known working version and the first place we noticed this being broken. It does appear to be a change in this commit: <a href="https://github.com/DPDK/dpdk/commit/a632f0f64ffba3553a18bdb51a670c1b603c0ce6">https://github.com/DPDK/dpdk/commit/a632f0f64ffba3553a18bdb51a670c1b603c0ce6</a><o:p></o:p> <o:p></o:p> I focused on the virtio_alloc_queue_headers() and virtio_free_queue_headers() functions. I think I have narrowed it down to to the hdr_mem setting. The following changes seem to be working in my test environment (which is a little limited).<o:p></o:p> <o:p></o:p> I was hoping you could look at these changes and hopefully help get a fix in for 24.11.<o:p></o:p> <o:p></o:p> Regards,<o:p></o:p> -ChrisB<o:p></o:p> <o:p></o:p> ---<o:p></o:p> drivers/net/virtio/virtqueue.c | 2 +-<o:p></o:p> 1 file changed, 1 insertion(+), 1 deletion(-)<o:p></o:p> <o:p></o:p> diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c<o:p></o:p> index 6f419665f1..fc7f7a9c55 100644<o:p></o:p> --- a/drivers/net/virtio/virtqueue.c<o:p></o:p> +++ b/drivers/net/virtio/virtqueue.c<o:p></o:p> @@ -344,7 +344,7 @@ virtio_alloc_queue_headers(struct virtqueue *vq, int numa_node, const char *name<o:p></o:p> if (vq->hw->use_va)<o:p></o:p> *hdr_mem = (uintptr_t)(*hdr_mz)->addr;<o:p></o:p> else<o:p></o:p> - *hdr_mem = (uintptr_t)(*hdr_mz)->iova;<o:p></o:p> + *hdr_mem = (*hdr_mz)->iova;<o:p></o:p> return 0;<o:p></o:p> }<o:p></o:p> -- <o:p></o:p> 2.35.6<o:p></o:p> <o:p></o:p> <div id="mail-editor-reference-message-container"> <div> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Chris Brezovec (cbrezove) <cbrezove@cisco.com> Date: Tuesday, September 3, 2024 at 10:43 AM To: dev@dpdk.org <dev@dpdk.org>, maxime.coquelin@redhat.com <maxime.coquelin@redhat.com> Cc: Roger Melton (rmelton) <rmelton@cisco.com>, Walt Robinson (walrobin) <walrobin@cisco.com> Subject: Re: 32-bit virtio failing on DPDK v23.11.1 (and tags)<o:p></o:p> </div> <div> Hi Maxime / others,<o:p></o:p> <o:p></o:p> I am just following up to see if you have had any chance to look at what I previously sent and had any ideas regarding the issue.<o:p></o:p> <o:p></o:p> Thanks in advance!<o:p></o:p> -ChrisB<o:p></o:p> <o:p></o:p> <div id="mail-editor-reference-message-container"> <div> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Chris Brezovec (cbrezove) <cbrezove@cisco.com> Date: Wednesday, August 28, 2024 at 5:27 PM To: dev@dpdk.org <dev@dpdk.org>, maxime.coquelin@redhat.com <maxime.coquelin@redhat.com> Cc: common-dpio-core-team(mailer list) <common-dpio-core-team@cisco.com> Subject: 32-bit virtio failing on DPDK v23.11.1 (and tags)<o:p></o:p> </div> <div> HI Maxime,<o:p></o:p> <o:p></o:p> My name is Chris Brezovec, we met and talked about some 32 bit virtio issues we were seeing at Cisco during the DPDK summit last year. There was also a back and forth between you and Dave Johnson at Cisco last September regarding the same issue. I have attached some of the email chain from that conversation that resulted in this commit being made to dpdk v23.11 (<a href="https://github.com/DPDK/dpdk/commit/8c41645be010ec7fa0df4f6c3790b167945154b4">https://github.com/DPDK/dpdk/commit/8c41645be010ec7fa0df4f6c3790b167945154b4</a>).<o:p></o:p> <o:p></o:p> We recently picked up the v23.11.1 DPDK release and saw that 32 bit virtio is not working again, but 64-bit virtio is working. We are noticing CVQ timeouts - PMD receives no response from host and this leads to failure of the port to start. We were able to recreate this issue using testpmd. We have done some tracing through the virtio changes made during the development of the v23.xx DPDK release, and believe we have identified the following rework commit to have caused a failure (<a href="https://github.com/DPDK/dpdk/commit/a632f0f64ffba3553a18bdb51a670c1b603c0ce6">https://github.com/DPDK/dpdk/commit/a632f0f64ffba3553a18bdb51a670c1b603c0ce6</a>).<o:p></o:p> <o:p></o:p> We have also tested v23.07, v23.11, v23.11.2-rc2, v24.07 and they all seem to see the same issue when running in 32-bit mode using testpmd.<o:p></o:p> <o:p></o:p> We were hoping you might be able to take a quick look at the two commits and see if there might be something obvious missing in the refactor work that might have caused this issue. I am thinking there might a location or two in the code that should be using the VIRTIO_MBUF_ADDR() or similar macro that might have been missed.<o:p></o:p> <o:p></o:p> Regards,<o:p></o:p> ChrisB<o:p></o:p> <o:p></o:p> This is some of the testpmd output seen on v23.11.2-rc2:<o:p></o:p> <o:p></o:p> LD_LIBRARY_PATH=/home/rmelton/scratch/dpdk-v23.11.2-rc2.git/build/lib /home/rmelton/scratch/dpdk-v23.11.2-rc2.git/build/app/dpdk-testpmd -l 2-3 -a 0000:07:00.0 --log-level pmd.net.iavf.*,8 --log-level lib.eal.*,8 --log-level=lib.eal:info --log-level=lib.eal:debug --log-level=lib.ethdev:info --log-level=lib.ethdev:debug --log-level=lib.virtio:warning --log-level=lib.virtio:info --log-level=lib.virtio:debug --log-level=pmd.*:debug --iova-mode=pa -- -i<o:p></o:p> <o:p></o:p> — snip —<o:p></o:p> <o:p></o:p> virtio_send_command(): vq->vq_desc_head_idx = 0, status = 255, vq->hw->cvq = 0x76d9acc0 vq = 0x76d9ac80<o:p></o:p> virtio_send_command_split(): vq->vq_queue_index = 2<o:p></o:p> virtio_send_command_split(): vq->vq_free_cnt=64<o:p></o:p> vq->vq_desc_head_idx=0<o:p></o:p> virtio_dev_promiscuous_disable(): Failed to disable promisc<o:p></o:p> Failed to disable promiscuous mode for device (port 0): Resource temporarily unavailable<o:p></o:p> Error during restoring configuration for device (port 0): Resource temporarily unavailable<o:p></o:p> virtio_dev_stop(): stop<o:p></o:p> Fail to start port 0: Resource temporarily unavailable<o:p></o:p> Done<o:p></o:p> virtio_send_command(): vq->vq_desc_head_idx = 0, status = 255, vq->hw->cvq = 0x76d9acc0 vq = 0x76d9ac80<o:p></o:p> virtio_send_command_split(): vq->vq_queue_index = 2<o:p></o:p> virtio_send_command_split(): vq->vq_free_cnt=64<o:p></o:p> vq->vq_desc_head_idx=0<o:p></o:p> virtio_dev_promiscuous_enable(): Failed to enable promisc<o:p></o:p> Error during enabling promiscuous mode for port 0: Resource temporarily unavailable - ignore <o:p></o:p> <o:p></o:p> <o:p></o:p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </body> </html>