Implementing a simple TAP PMD to dpdk-vhost structure

[Maxime Coquelin]

On 9/7/23 08:22, Nicolson Ken (ニコルソン ケン) wrote:
> Hi David,
> 
> Hmm, maybe the issue is at my end - to configure QEMU, I followed the tutorial at https://www.redhat.com/en/blog/hands-vhost-user-warm-welcome-dpdk
> 
>> Are you seeing the vhost port getting initialised in testpmd output?
> 
> Yes, I get a lot of VHOST_CONFIG messages, ending with:
> VHOST_CONFIG: (/tmp/sock0) virtio is now ready for processing.
> Rx csum will be done in SW, may impact performance.
> 
> ***
> UPDATE: I've been reading more of the manual while doing the troubleshooting below, and I think I've found a major issue while checking the Guest OS.
> According to https://doc.dpdk.org/guides-21.11/linux_gsg/linux_drivers.html
> 
> dmesg | tail
> ...
> [ 1297.875090] vfio-pci: probe of 0000:31:00.0 failed with error -22
> 
> I get the above in the Guest when trying to do devbind, and I also get this on both Host and Guest:
> 
> cat /boot/config-$(uname -r) | grep NOIOMMU
> CONFIG_VFIO_NOIOMMU=y
> 
> Should that actually be "N"? Does "is not set" equal to no IOMMU? I should follow the grubby settings given on that RedHat page, I think. If that is off, then I would guess that that could very well be a source of all my issues.

CONFIG_VFIO_NOIOMMU=y is valid but it just build noiommu support, what
you need is to enable it at probe time:

# modprobe vfio enable_unsafe_noiommu_mode=y
# cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
Y

But I understood you wanted to inject packets in the guest kernel. If
this is true it should not be necessary, the Virtio device  has to be
bound to Kernel Virtio-net driver in the guest, not to VFIO/Virtio PMD.

Maxime

> 
> Back to the previous contents:
> ***
> 
> Now, I try your example - in another terminal on the Host, I'll use "ping -v -I dtap0 -6 fe80::5054:ff:fe01:7d00", which is the address of the interface created in the Guest OS by QEMU (hmm, my <mac address=> seems to have been ignored), after I do "sudo ip link set enp9s0 up" there.
> 
> First, ping output (if I don't use "-I dtap0", nothing happens):
> $ ping -v -I dtap0 -6 fe80::5054:ff:fe01:7d00
> ping: Warning: source address might be selected on device other than: dtap0
> PING fe80::5054:ff:fe01:7d00(fe80::5054:ff:fe01:7d00) from :: dtap0: 56 data bytes
>  From fe80::20d4:2dff:fe67:3768%dtap0 icmp_seq=1 Destination unreachable: Address unreachable
>  From fe80::20d4:2dff:fe67:3768%dtap0 icmp_seq=2 Destination unreachable: Address unreachable
>  From fe80::20d4:2dff:fe67:3768%dtap0 icmp_seq=3 Destination unreachable: Address unreachable
> ^C
> --- fe80::5054:ff:fe01:7d00 ping statistics ---
> 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3055ms
> 
> testpmd> set verbose 3
> Change verbose level from 0 to 3
> testpmd> start
> io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native
> Logical Core 1 (socket 0) forwards packets on 2 streams:
>    RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
>    RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
> 
> Then for each for the 6 attempted pings I see:
> 
> port 0/queue 0: received 1 packets
>    src=22:D4:2D:67:37:68 - dst=33:33:FF:01:7D:00 - pool=mb_pool_0 - type=0x86dd - length=86 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV6  - sw ptype: L2_ETHER L3_IPV6  - l2_len=14 - l3_len=40 - Receive queue=0x0
>    ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
> port 1/queue 0: sent 1 packets
>    src=22:D4:2D:67:37:68 - dst=33:33:FF:01:7D:00 - pool=mb_pool_0 - type=0x86dd - length=86 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV6  - sw ptype: L2_ETHER L3_IPV6  - l2_len=14 - l3_len=40 - Send queue=0x0
>    ol_flags: RTE_MBUF_F_TX_L4_NO_CKSUM
> 
> Thanks for the help,
> Ken
> 
> -----Original Message-----
> From: David Marchand <david.marchand at redhat.com>
> Sent: Wednesday, September 6, 2023 5:41 PM
> To: Nicolson Ken (ニコルソン ケン) <ken.nicolson at jp.panasonic.com>
> Cc: users at dpdk.org; Maxime Coquelin <maxime.coquelin at redhat.com>; Xia, Chenbo <chenbo.xia at intel.com>
> Subject: Re: Implementing a simple TAP PMD to dpdk-vhost structure
> 
> On Wed, Sep 6, 2023 at 9:53 AM Nicolson Ken (ニコルソン ケン)
> <ken.nicolson at jp.panasonic.com> wrote:
>>> Alternatively, did you consider using testpmd with the vhost pmd instead ?
>>
>> I've tried that before, but as far as I can see from net/vhost/rte_eth_host.c it uses rte_vost_enqueue/dequeue_burst() to basically act as a loopback for the Guest OS. I use:
>>
>> $ sudo dpdk-testpmd -l 0-3 -n 4 --vdev 'net_tap0' --vdev
>> 'net_vhost1,iface=/tmp/sock0,client=1' -- -i
>>
>> But if I feed data in using "tcpreplay -I dtap0 ...", "show port stats all" shows everything going into the TAP but nothing is forward to vhost.
> 
> Well, pinging from a dtap0 netdev in the host to a virtio-net netdev in a guest works me.
> 
> testpmd> set verbose 3
> Change verbose level from 0 to 3
> testpmd> start
> io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native Logical Core 1 (socket 1) forwards packets on 2 streams:
>    RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
>    RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
> 
> port 0/queue 0: received 16 packets
>    src=26:9B:E2:29:7E:C6 - dst=33:33:00:00:00:16 - pool=mb_pool_0 - type=0x86dd - length=90 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV6_EXT
> - sw ptype: L2_ETHER L3_IPV6_EXT  - l2_len=14 - l3_len=48 - Receive
> queue=0x0
>    ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
>    src=26:9B:E2:29:7E:C6 - dst=33:33:00:00:00:16 - pool=mb_pool_0 - type=0x86dd - length=90 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV6_EXT
> - sw ptype: L2_ETHER L3_IPV6_EXT  - l2_len=14 - l3_len=48 - Receive
> queue=0x0
>    ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
>    src=26:9B:E2:29:7E:C6 - dst=33:33:FF:29:7E:C6 - pool=mb_pool_0 - type=0x86dd - length=86 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV6  - sw
> ptype: L2_ETHER L3_IPV6  - l2_len=14 - l3_len=40 - Receive queue=0x0
>    ol_flags: RTE_MBUF_F_RX_L4_CKSUM_UNKNOWN RTE_MBUF_F_RX_IP_CKSUM_UNKNOWN RTE_MBUF_F_RX_OUTER_L4_CKSUM_UNKNOWN
> 
> 
> Are you seeing the vhost port getting initialised in testpmd output?
> How are you sure that nothing is forwarded?
> 
> 
> --
> David Marchand
>