[dpdk-dev] Vhost user no connection vm2vm

Tetsuya Mukawa mukawa at igel.co.jp
Fri May 22 11:58:52 CEST 2015


Hi Maciej,

I guess it's nice to make sure guest memory is actually allocated by
hugepages.
So please check like below.

$ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages
$ ./start_qemu.sh
$ cat /sys/kernel/mm/hugepage/xxxxx/free_hugepages

If qemu guest allocates memory from hugepages, 2nd cat command will
indicate it.

Thanks,
Tetsuya


On 2015/05/22 18:28, Maciej Grochowski wrote:
> "Do you use some command I suggest before,
> In case of you miss the previous mail, just copy it again:"
>
> -Yes but it didn't help me ;/
>
> I will describe step by step to esure that configuration is made by right
> way
>
>
> I started vhost:
>
> ./build/app/vhost-switch -c f -n 4  --huge-dir /mnt/huge --socket-mem 3712
> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9
>
> Now I run two vm machines, with followed configuration
>
> VM1   __  __  VM2
> eth0 >  \/  > eth0
> eth1 >__/\__> eth1
>
> So I will connect VM1.eth0 with VM2.eth1 and VM1.eth1 with VM2.eth0
> Because it is test env and I didn't have other network connection on vhost
> I will create two networks 192.168.0.x and 192.168.1.x
>  VM1.eth0 with VM2.eth1 will be placed in 192.168.0.x and VM1.eth1 with
> VM2.eth0 in 192.168.1.x
>
> ## I started first VM1 as follow
> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm2 -cpu host
> -smp 1 \
> -hda /home/ubuntu/esi_ee/qemu/debian_min_1.qcow2 -m 256 -mem-path /mnt/huge
> -mem-prealloc \
> -chardev
> socket,id=char3,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \
> -netdev type=vhost-user,id=hostnet3,chardev=char3 \
> -device
> virtio-net-pci,netdev=hostnet3,id=net3,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
> \
> -chardev
> socket,id=char4,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \
> -netdev type=vhost-user,id=hostnet4,chardev=char4 \
> -device
> virtio-net-pci,netdev=hostnet4,id=net4,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
> ## qemu give followed output
> qemu-system-x86_64: -netdev type=vhost-user,id=hostnet3,chardev=char3:
> chardev "char3" went up
> qemu-system-x86_64: -netdev type=vhost-user,id=hostnet4,chardev=char4:
> chardev "char4" went up
>
> ## second VM2
> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu host
> -smp 1 \
> -hda /home/ubuntu/esi_ee/qemu/debian_min_2.qcow2 -m 256 -mem-path /mnt/huge
> -mem-prealloc \
> -chardev
> socket,id=char1,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \
> -netdev type=vhost-user,id=hostnet1,chardev=char1 \
> -device
> virtio-net-pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
> \
> -chardev
> socket,id=char2,path=/home/ubuntu/esi_ee/dpdk/examples/vhost/usvhost \
> -netdev type=vhost-user,id=hostnet2,chardev=char2 \
> -device
> virtio-net-pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off
> ## second output
> qemu-system-x86_64: -netdev type=vhost-user,id=hostnet1,chardev=char1:
> chardev "char1" went up
> qemu-system-x86_64: -netdev type=vhost-user,id=hostnet2,chardev=char2:
> chardev "char2" went up
>
>
>
> After that I had MAC conflict between VM2 and VM1
>
> VM1: -ifconfig -a
> eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
>           inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> eth1      Link encap:Ethernet  HWaddr 52:54:00:12:34:57
>           BROADCAST MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
>
> VM2: -ifconfig -a
> eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56
>           inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> eth1      Link encap:Ethernet  HWaddr 52:54:00:12:34:57
>           BROADCAST MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> In KNI example I had something similar and also no packet flow and solution
> was to change MAC addresses
>
> #VM1
> /etc/init.d/networking stop
> ifconfig eth0 hw ether 00:01:04:00:01:00
> ifconfig eth1 hw ether 00:01:04:00:01:01
> /etc/init.d/networking start
> ifconfig eth0
> ifconfig eth1
>
> #VM2
> /etc/init.d/networking stop
> ifconfig eth0 hw ether 00:01:04:00:02:00
> ifconfig eth1 hw ether 00:01:04:00:02:01
> /etc/init.d/networking start
> ifconfig eth0
> ifconfig eth1
>
> Then I make a configuration that You show:
>
> #VM1
> ip addr add 192.168.0.100/24 dev eth0
> ip addr add 192.168.1.100/24 dev eth1
> ip neigh add 192.168.0.200 lladdr 00:01:04:00:02:01 dev eth0
> ip link set dev eth0 up
> ip neigh add 192.168.1.200 lladdr 00:01:04:00:02:00 dev eth1
> ip link set dev eth1 up
>
> eth0      Link encap:Ethernet  HWaddr 00:01:04:00:01:00
>           inet addr:192.168.0.100  Bcast:0.0.0.0  Mask:255.255.255.0
>           inet6 addr: fe80::201:4ff:fe00:100/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> eth1      Link encap:Ethernet  HWaddr 00:01:04:00:01:01
>           inet addr:192.168.1.100  Bcast:0.0.0.0  Mask:255.255.255.0
>           inet6 addr: fe80::201:4ff:fe00:101/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
>
> #VM2
> ip addr add 192.168.1.200/24 dev eth0
> ip addr add 192.168.0.200/24 dev eth1
> ip neigh add 192.168.1.100 lladdr 00:01:04:00:01:01 dev eth0
> ip link set dev eth0 up
> ip neigh add 192.168.0.100 lladdr 00:01:04:00:01:00 dev eth1
> ip link set dev eth1 up
>
> eth0      Link encap:Ethernet  HWaddr 00:01:04:00:02:00
>           inet addr:192.168.1.200  Bcast:0.0.0.0  Mask:255.255.255.0
>           inet6 addr: fe80::201:4ff:fe00:200/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> eth1      Link encap:Ethernet  HWaddr 00:01:04:00:02:01
>           inet addr:192.168.0.200  Bcast:0.0.0.0  Mask:255.255.255.0
>           inet6 addr: fe80::201:4ff:fe00:201/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> After that:
>
> VM1.eth0 ip=192.168.0.100-MAC=00:01:04:00:01:00  is connected to VM2.eth1
> ip=192.168.0.200-MAC=00:01:04:00:02:01
> VM1.eth1 ip=192.168.1.100-MAC=00:01:04:00:01:01  is connected to VM2.eth0
> ip=192.168.1.200-MAC=00:01:04:00:02:00
>
> That show my arp tables:
>
> #VM1
> arp -a
> ? (192.168.0.200) at 00:01:04:00:02:01 [ether] PERM on eth0
> ? (192.168.1.200) at 00:01:04:00:02:00 [ether] PERM on eth1
>
>
> #VM2
> arp -a
> ? (192.168.0.100) at 00:01:04:00:01:00 [ether] PERM on eth1
> ? (192.168.1.100) at 00:01:04:00:01:01 [ether] PERM on eth0
>
>
> #After this configuration I trying to ping from VM1 VM2 (both IP)
>
> root at debian-amd64:~# ping -I eth0 192.168.0.200
> PING 192.168.0.200 (192.168.0.200) from 192.168.0.100 eth0: 56(84) bytes of
> data.
> ^C
> --- 192.168.0.200 ping statistics ---
> 5 packets transmitted, 0 received, 100% packet loss, time 4032ms
>
> root at debian-amd64:~# ping 192.168.0.200
> PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data.
> ^C
> --- 192.168.0.200 ping statistics ---
> 5 packets transmitted, 0 received, 100% packet loss, time 3999ms
>
> root at debian-amd64:~# ping -I eth1 192.168.1.200
> PING 192.168.1.200 (192.168.1.200) from 192.168.1.100 eth1: 56(84) bytes of
> data.
> ^C
> --- 192.168.1.200 ping statistics ---
> 6 packets transmitted, 0 received, 100% packet loss, time 5040ms
>
> root at debian-amd64:~# ping 192.168.1.200
> PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data.
> ^C
> --- 192.168.1.200 ping statistics ---
> 5 packets transmitted, 0 received, 100% packet loss, time 4030ms
>
> #and from VM2 VM1
> root at debian-amd64:~# ping 192.168.0.100
> PING 192.168.0.100 (192.168.0.100) 56(84) bytes of data.
> ^C
> --- 192.168.0.100 ping statistics ---
> 3 packets transmitted, 0 received, 100% packet loss, time 2015ms
>
> root at debian-amd64:~# ping -I eth1 192.168.0.100
> PING 192.168.0.100 (192.168.0.100) from 192.168.0.200 eth1: 56(84) bytes of
> data.
> ^C
> --- 192.168.0.100 ping statistics ---
> 5 packets transmitted, 0 received, 100% packet loss, time 4032ms
>
> root at debian-amd64:~# ping -I eth0 192.168.1.100
> PING 192.168.1.100 (192.168.1.100) from 192.168.1.200 eth0: 56(84) bytes of
> data.
> ^C
> --- 192.168.1.100 ping statistics ---
> 4 packets transmitted, 0 received, 100% packet loss, time 3024ms
>
> root at debian-amd64:~# ping 192.168.1.100
> PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
> ^C
> --- 192.168.1.100 ping statistics ---
> 8 packets transmitted, 0 received, 100% packet loss, time 7055ms
>
>
> Also stats from vhost:
> Device statistics ====================================
> Statistics for device 0 ------------------------------
> TX total:               0
> TX dropped:             0
> TX successful:          0
> RX total:               0
> RX dropped:             0
> RX successful:          0
> Statistics for device 1 ------------------------------
> TX total:               0
> TX dropped:             0
> TX successful:          0
> RX total:               0
> RX dropped:             0
> RX successful:          0
> Statistics for device 2 ------------------------------
> TX total:               0
> TX dropped:             0
> TX successful:          0
> RX total:               0
> RX dropped:             0
> RX successful:          0
> Statistics for device 3 ------------------------------
> TX total:               0
> TX dropped:             0
> TX successful:          0
> RX total:               0
> RX dropped:             0
> RX successful:          0
> ======================================================
>
> My way of thinking was: "In vhost there are several function for L2 that
> learn MAC-s and links them so why I see no received packets?"
>
> Maybe I'm doing some silly bug in network configuration but for me its
> looking like data flow issue especially that no function on the vhost side
> did not see any packages.
>
> On Fri, May 22, 2015 at 10:54 AM, Gaohaifeng (A) <gaohaifeng.gao at huawei.com>
> wrote:
>
>>  Hi
>>
>> What kernel version are You using on host/guest?
>>
>>>> I use ubuntu 12.04(3.11.0-15-generic) in host. In vm I use ubuntu 12.04
>> and ubuntu14.04 both, but the result is same.
>>
>>
>>
>> Do you use some command I suggest before, In case of you miss the previous
>> mail, just copy it again:
>>
>>>> I try it but the result is same
>>
>>
>>
>>
>> I use l2fwd in vm to do more test and found that virtio_xmit_pkts is
>> called and avail_idx is increasing in vm, but in host avail_idx(in
>> rte_vhost_dequeue_burst function) is always zero. It seems that the host
>> see the different mem area.
>>
>>
>>
>> Init Logs below:
>>
>> VHOST_CONFIG: (0) Mergeable RX buffers disabled
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
>>
>> VHOST_CONFIG: mapped region 0 fd:24 to 0x2aaaaac00000 sz:0xa0000 off:0x0
>>
>> VHOST_CONFIG: REGION: 0 GPA: (nil) QEMU VA: 0x2aaaaac00000 SIZE (655360)
>>
>> VHOST_CONFIG: mapped region 1 fd:26 to 0x2aaaaae00000 sz:0x40000000
>> off:0xc0000
>>
>> VHOST_CONFIG: REGION: 1 GPA: 0xc0000 QEMU VA: 0x2aaaaacc0000 SIZE
>> (1072955392)
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>>
>> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f1000
>>
>> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f2000
>>
>> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f3000
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>>
>> VHOST_CONFIG: vring kick idx:0 file:23
>>
>> VHOST_CONFIG: virtio isn't ready for processing.
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>>
>> VHOST_CONFIG: (0) mapped address desc: 0x2aaae62f4000
>>
>> VHOST_CONFIG: (0) mapped address avail: 0x2aaae62f5000
>>
>> VHOST_CONFIG: (0) mapped address used: 0x2aaae62f6000
>>
>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>>
>> VHOST_CONFIG: vring kick idx:1 file:28
>>
>> VHOST_CONFIG: virtio is now ready for processing.
>>
>>
>>
>>
>>
>>> Unfortunately not, I have the same issue in rte_vhost_dequeue_burst
>> function.
>>
>>
>>
>>> What kernel version are You using on host/guest? In my case on host I
>> had 3.13.0 and on guests old 3.2 debian.
>>
>>
>>
>>> I just looked deeper into virtio  back-end (vhost) but at first glace it
>> seems like nothing coming from virtio.
>>
>>
>>
>>> What I'm going to do today is to compile newest kernel for vhost and
>> guest and debug where packet flow stuck, I will report the result
>>
>>
>>
>>> On Thu, May 21, 2015 at 11:12 AM, Gaohaifeng (A) <
>> gaohaifeng.gao at huawei.com> wrote:
>>
>>> Hi Maciej
>>         >Did you solve your problem? I meet this problem as your case.
>> And I found avail_idx(in rte_vhost_dequeue_burst function) is always zero
>> although I do send packets in VM.
>>
>>> Thanks.
>>
>>
>>> Hello, I have strange issue with example/vhost app.
>>>
>>> I had compiled DPDK to run a vhost example app with followed flags
>>>
>>> CONFIG_RTE_LIBRTE_VHOST=y
>>> CONFIG_RTE_LIBRTE_VHOST_USER=y
>>> CONFIG_RTE_LIBRTE_VHOST_DEBUG=n
>>>
>>> then I run vhost app based on documentation:
>>>
>>>  ./build/app/vhost-switch -c f -n 4  --huge-dir /mnt/huge --socket-mem
>>> 3712
>>> -- -p 0x1 --dev-basename usvhost --vm2vm 1 --stats 9
>>>
>>> -I use this strange --socket-mem 3712 because of physical limit of
>>> memoryon device -with this vhost user I run two KVM machines with
>>> followed parameters
>>>
>>> kvm -nographic -boot c -machine pc-i440fx-1.4,accel=kvm -name vm1 -cpu
>>> host -smp 2 -hda /home/ubuntu/qemu/debian_squeeze2_amd64.qcow2 -m
>>> 1024 -mem-path /mnt/huge -mem-prealloc -chardev
>>> socket,id=char1,path=/home/ubuntu/dpdk/examples/vhost/usvhost
>>> -netdev type=vhost-user,id=hostnet1,chardev=char1
>>> -device virtio-net
>>> pci,netdev=hostnet1,id=net1,csum=off,gso=off,guest_tso4=off,guest_tso6
>>> =
>>> off,guest_ecn=off
>>> -chardev
>>> socket,id=char2,path=/home/ubuntu/dpdk/examples/vhost/usvhost
>>> -netdev type=vhost-user,id=hostnet2,chardev=char2
>>> -device
>>> virtio-net-
>>> pci,netdev=hostnet2,id=net2,csum=off,gso=off,guest_tso4=off,guest_tso6
>>> =
>>> off,guest_ecn=off
>>>
>>> After running KVM virtio correctly starting (below logs from vhost app)
>> ...
>>> VHOST_CONFIG: mapped region 0 fd:31 to 0x2aaabae00000 sz:0xa0000
>>> off:0x0
>>> VHOST_CONFIG: mapped region 1 fd:37 to 0x2aaabb000000 sz:0x10000000
>>> off:0xc0000
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>>> VHOST_CONFIG: vring kick idx:0 file:38
>>> VHOST_CONFIG: virtio isn't ready for processing.
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>>> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>>> VHOST_CONFIG: vring kick idx:1 file:39
>>> VHOST_CONFIG: virtio is now ready for processing.
>>> VHOST_DATA: (1) Device has been added to data core 2
>>>
>>> So everything looking good.
>>>
>>> Maybe it is something trivial but using options: --vm2vm 1 (or) 2
>>> --stats 9 it seems that I didn't have connection between VM2VM
>>> communication. I set manually IP for eth0 and eth1:
>>>
>>> on 1 VM
>>> ifconfig eth0 192.168.0.100 netmask 255.255.255.0 up ifconfig eth1
>>> 192.168.1.101 netmask 255.255.255.0 up
>>>
>>> on 2 VM
>>> ifconfig eth0 192.168.1.200 netmask 255.255.255.0 up ifconfig eth1
>>> 192.168.0.202 netmask 255.255.255.0 up
>>>
>>> I notice that in vhostapp are one directional rx/tx queue so I tryied
>>> to ping between VM1 to VM2 using both interfaces ping -I eth0
>>> 192.168.1.200 ping -I
>>> eth1 192.168.1.200 ping -I eth0 192.168.0.202 ping -I eth1
>>> 192.168.0.202
>>>
>>> on VM2 using tcpdump on both interfaces I didn't see any ICMP requests
>>> or traffic
>>>
>>> And I cant ping between any IP/interfaces, moreover stats show me that:
>>>
>>> Device statistics ====================================
>>> Statistics for device 0 ------------------------------
>>> TX total:               0
>>> TX dropped:             0
>>> TX successful:          0
>>> RX total:               0
>>> RX dropped:             0
>>> RX successful:          0
>>> Statistics for device 1 ------------------------------
>>> TX total:               0
>>> TX dropped:             0
>>> TX successful:          0
>>> RX total:               0
>>> RX dropped:             0
>>> RX successful:          0
>>> Statistics for device 2 ------------------------------
>>> TX total:               0
>>> TX dropped:             0
>>> TX successful:          0
>>> RX total:               0
>>> RX dropped:             0
>>> RX successful:          0
>>> Statistics for device 3 ------------------------------
>>> TX total:               0
>>> TX dropped:             0
>>> TX successful:          0
>>> RX total:               0
>>> RX dropped:             0
>>> RX successful:          0
>>> ======================================================
>>>
>>> So it seems like any packet didn't leave my VM.
>>> also arp table is empty on each VM.
>>
>>



More information about the dev mailing list