[dpdk-dev] virtio UIO / PMD issues in default Ubuntu Cloud Images

[Matthew Hall]

Hi Sergio,

Sorry for the delay, this project is part-time.

On Tue, Oct 14, 2014 at 12:16:07PM +0000, Gonzalez Monroy, Sergio wrote:
> Could you provide steps to reproduce your issue with virtio non-UIO PMD (virtio-net-pmd) using static combined DPDK lib?

I am getting two different kinds of problems. See below.

> librte_pmd_virtio.so does depend on DPDK, but ldd won't show it because it 
> was not build against DPDK libs which it makes sense if you are building 
> static lib DPDK.

The virtio non UIO PMD doesn't depend on DPDK even if everything's built as a 
.so . To me this seems buggy but maybe I missed something:

$ ldd librte_pmd_virtio.so
    linux-vdso.so.1 =>  (0x00007fff7adfe000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0d810c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa0d86ed000)

Now, about the problems...

1) This first problem happens using it with DPDK built as shared 
libraries. It gets stuck looping forever in this code if I build it as a 
shared library, and DPDK won't initialize:

(gdb)
850     while (vq->last_used == vq->vr.used->idx) {
(gdb)
851         usleep(100);
(gdb) bt
#0  virtio_send_command (vq=0x7fffed0ffdc8, ctrl=0x7fffffffe2b0, dlen=0x7fffffffe2a0, pkt_num=1) at virtio_user.c:851
#1  0x00007ffff193f73c in virtio_dev_promiscuous_disable (dev=0x7ffff7dd9120 <rte_eth_devices>) at virtio_user.c:914
#2  0x00007ffff7bd2f37 in rte_eth_promiscuous_disable (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:1055
#3  0x00007ffff7bd271b in rte_eth_dev_config_restore (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:788
#4  0x00007ffff7bd25b1 in rte_eth_dev_start (port_id=0 '\000') at /vagrant/external/dpdk/lib/librte_ether/rte_ethdev.c:828
#5  0x000000000042f2ea in main (argc=3, argv=0x7fffffffe668) at sdn_sensor.c:399
(gdb)

It appears to do this whether or not I use VBoxManage modifyvm <vm_name> 
--nicpromiscN allow-all or not on the VM. So it can't be caused just by VBox 
messing up the promisc settings.

2) The second problem concerns compiling against DPDK 1.7.1 (plug my minor 
clang compilation fixes) with these settings:

# Compile to share library
CONFIG_RTE_BUILD_SHARED_LIB=n
# Combine to one single library
CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"

Then virtio-net-pmd is compiled using make RTE_INCLUDE=../dpdk/build/include . 
Just as printed above, no runtime dependency on DPDK is there (as expected in 
the static case, but not what I believe should be expected in the dynamic 
case).

>From there, the app is launched. Then the following error appears, and no 
Ethernet ports can be detected since the builtin UIO PMD driver can't see the 
ports either, so the app exits.

EAL: open shared lib /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so
EAL: /vagrant/external/virtio-net-pmd/librte_pmd_virtio.so: undefined symbol: per_lcore__lcore_id

Running nm and nm -D shows this:

$ nm librte_pmd_virtio.so | fgrep -i per_lcore__lcore_id
U per_lcore__lcore_id

According to the nm manpage, "U" means "The symbol is undefined".

Since it didn't get added from a static lib, and our DPDK wasn't linked and 
loaded in dynamic mode where the linker could hook up the symbol, loading 
librte_pmd_virtio.so fails and no ports can be found.

Given all of this, I must re-ask the question I previously asked: who provided 
QA before claiming on the DPDK website that this driver works with VirtualBox, 
and precisely and reproducibly, how did they get it to run right? Because it's 
impossible to get anything to work on my machine using all the directions I've 
been able to find thus far.

Thanks,
Matthew.