[dpdk-dev] How to debug packet sends to virtual functions

Mats Liljegren liljegren.mats2 at gmail.com
Tue Feb 4 12:31:34 CET 2014

Hi Qinglai,

How did you configure the PCI passthrough in KVM? Did you use command
line parameter as described in the instructions, or do you use virsh
XML files, or maybe using virt-manager?

The steps that works best (i.e. can receive but not transmit) are:

1) sudo modprobe ixgbe max_vfs=2
2) sudo ip link set em1 vf 0 mac <phys mac>
3) sudo ifconfig em1 up
4) sudo modprobe pci-stub
5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id
6) echo "0000:02:10.0" | sudo tee
7) echo "0000:02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind
8) virsh create vm vm.xml

The guest then does:

5) sudo modprobe uio
6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio 0000:02:10.0
7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio
8) sudo <run DPDK app>

The virsh xml file has the following entry for the virtual function:
    <hostdev mode='subsystem' type='pci' managed='yes'>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0' />
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'

I actually have two virtual functions and two physical functions, but
I just repeat above for each.


On Tue, Feb 4, 2014 at 12:14 PM, jigsaw <jigsaw at gmail.com> wrote:
> Hi Mats,
> I didn't have any deviation. What I did is just loading ixgbe (with
> extra params for vf, as you mentioned in your first email),
> and DPDK is up and running in guest. Of course I also followed section
> 8.10 of the DPDK release notes.
> I can switch between DPDK and ixgebvf in guest at runtime and
> everything works fine.
> Sorry I can't help to debug coz I have only 82599EB at hand.
> -Qinglai
> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren
> <liljegren.mats2 at gmail.com> wrote:
>> Hi Qinglai,
>> Thanks for the response!
>> My previous attempt was with ixgbe loaded in the host. I also needed
>> to load ixgbevf, but this seems to be because of a short-coming in
>> libvirt. Maybe loading ixgbevf and then unbind it when running the
>> guest is what causes my problems. I could receive packets with this
>> setup, but not transmit. I used extra debug to syslog, and it showed
>> that packets was placed in the transmit queue, but these packets was
>> never sent.
>> I'll see if I can get this working without loading ixgbevf in the host.
>> What instructions did you follow to get this working? Did you do any
>> deviation from the instructions?
>> Regards
>> Mats
>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw at gmail.com> wrote:
>>> Hi Mats,
>>> I've tried vf with 82599EB and it works fine. As long as the VF is
>>> visible in guest, DPDK's VF driver should work just as ixgbevf, which
>>> shares more or less the same code.
>>> I don't understand why you would expect DPDK at guest to work as VF,
>>> while the host has no ixgbe loaded.
>>> To make further debug I'd suggest compile ixgbe driver with your own
>>> syslogs. At least you will be able to see the signalling between vf
>>> and ixgbe drivers.
>>> -Qinglai
>>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>>> <liljegren.mats2 at gmail.com> wrote:
>>>> This is my fourth mail in my desperate attempt to get DPDK running in
>>>> KVM and no comments so far, not even any "it works for me". Am I the
>>>> only one crazy enough to believe that this can be done?
>>>> Anyway, out of desperation I tried to get it running without having
>>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>>>> Virtual Machine Monitor", using the PMD version of the instructions.
>>>> I get as far as being able to see my four virtual functions in the
>>>> guest using "lspci". But starting the DPDK application gives me the
>>>> following error:
>>>> PMD:    The MAC address is not valid.
>>>>         The most likely cause of this error is that the VM host
>>>>         has not assigned a valid MAC address to this VF device.
>>>>         Please consult the DPDK Release Notes (FAQ section) for
>>>>         a possible solution to this problem.
>>>> This may be true, but without any kernel modules loaded, how am I
>>>> supposed to change any MAC addresses? Can this be done from within
>>>> DPDK?
>>>> As a side-note, I did try to load ixgbevf in the guest, but it
>>>> produced no interfaces. There was no error messages in the syslog
>>>> though.
>>>> Is it possible to get X540 working in a guest or should I switch hardware?
>>>> Since the instructions assumes I know the command line to KVM to start
>>>> my guest (which I do not), I cannot followed them precisely. I use
>>>> virsh and XML file, and maybe I've misunderstood how to translate the
>>>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>>>> but I've also tried <interface type='hostdev'>. Neither has been
>>>> working for me so far, though the <interface> version got me as far as
>>>> being able to receive packets at least, but not transmitting.
>>>> Regards
>>>> Mats
>>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>>>> <liljegren.mats2 at gmail.com> wrote:
>>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>>>> on both the host and the guest using ixgbevf driver, so apparently the
>>>>> cables are correctly attached.
>>>>> Using DPDK is still no-go. It can receive packets, but when sending
>>>>> the packets the function returns success, but the driver reports
>>>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>>>> received packets of course).
>>>>> What could cause this behavior?
>>>>> Regards
>>>>> Mats
>>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>>>> <liljegren.mats2 at gmail.com> wrote:
>>>>>> I have a follow-up on this:
>>>>>> ixgbe version 3.13.10-k
>>>>>> ixgbevf version 2.7.12-k
>>>>>> (These are what was provided by Ubuntu 13.10)
>>>>>> I tried the following sequence on the host, before starting the guest:
>>>>>> 1) sudo rmmod ixgbe
>>>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>>>> 4) sudo ifconfig em1_0  # This is the virtual function
>>>>>> 5) ping
>>>>>> I can see that the ping request reaches its target, and a reply is
>>>>>> sent back. But this reply is not received by the ping shell command.
>>>>>> Why?
>>>>>> Regards,
>>>>>> Mats
>>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>>>> <liljegren.mats2 at gmail.com> wrote:
>>>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>>>> have problems with packets being silently thrown away. I can receive
>>>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>>>> the correct port, using correct MAC address as source address. And
>>>>>>> still, the packets are lost.
>>>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>>>> of the cable (activity light not flashing, the receiving end running
>>>>>>> tcpdump does not see any traffic).
>>>>>>> I'm using two X540 with two virtual functions each. The application
>>>>>>> runs in a KVM/Qemu environmen.
>>>>>>> Any suggestions how to debug this?
>>>>>>> Regards,
>>>>>>> Mats

More information about the dev mailing list