<div dir="ltr"><div dir="ltr">Hi JJ,<div>Sorry for the late reply.</div><div>Here are the answers to your questions.<br>Still having the same issue.</div><div>Nothing is reaching to DPDK application after ping .<br>I tried to run l2fwd and found the same issue<br>on VM2 after running l2fwd I have two Mac addresses for port 0 : FA:16:3E:EF:2C:26 and Port 1, MAC address: FA:16:3E:0D:39:EC<br><br>then I added entry in to ARP table to associate IP address with these mac address on VM2 like :</div><div>arp -s x.y.t.z FA:16:3E:EF:2C:26 dev ens3(ens3 is bind with kernel module)</div><div>arp -s
x.y.t.r FA:16:3E:0D:39:EC dev ens3<br>After it I tried to ping
x.y.t.z assuming that this message will go to port 0 RX queue but not see any message in l2fwd .</div><div><br></div><div>Here is a log snapshot:<br>"iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14<br>done:<br>Port 0, MAC address: FA:16:3E:EF:2C:26<br><br>Initializing port 1... iavf_dev_init_vlan(): Failed to update vlan offload<br>iavf_dev_configure(): configure VLAN failed: -95<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 25<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26<br>iavf_dev_rx_queue_setup(): >><br>iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.<br>check_rx_vec_allow(): Vector Rx can be enabled on this rxq.<br>iavf_dev_tx_queue_setup(): >><br>check_tx_vec_allow(): Vector tx can be enabled on this txq.<br>iavf_dev_start(): >><br>iavf_set_rx_function(): request RXDID[22] in Queue[0] is not supported, set rx_pkt_burst as legacy for all queues<br>iavf_set_rx_function(): Using avx2 Vector Rx (port 1).<br>iavf_set_tx_function(): Using avx2 Vector Tx (port 1).<br>iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0]<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7<br>iavf_add_del_all_mac_addr(): add/rm mac:FA:16:3E:0D:39:EC<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10<br>iavf_dev_tx_queue_start(): >><br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8<br>iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event<br>iavf_dev_rx_queue_start(): >><br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8<br>iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14<br>iavf_dev_alarm_handler(): ICR01_ADMINQ is reported<br>iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14<br>done:<br>Port 1, MAC address: FA:16:3E:0D:39:EC<br><br>Skipping disabled port 2<br><br>Checking link statusdone<br>Port 0 Link up at 25 Gbps FDX Autoneg<br>Port 1 Link up at 25 Gbps FDX Autoneg<br>L2FWD: entering main loop on lcore 2<br>L2FWD: -- lcoreid=2 portid=1<br>L2FWD: entering main loop on lcore 1<br>L2FWD: -- lcoreid=1 portid=0" <br><br>and then log I added at RX queue to dump source and dest in Rx buffer:<br>L2FWD: ITE one ----nb_rx=1 portid=0<br>L2FWD: ITR ---2----src=134610945 dest=134219268</div><div>I added this login <br><span style="color:rgb(106,153,85);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre">
/* main processing loop */</span><div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div><span style="color:rgb(86,156,214)">static</span> <span style="color:rgb(86,156,214)">void</span></div><div><span style="color:rgb(220,220,170)">l2fwd_main_loop</span>(<span style="color:rgb(86,156,214)">void</span>)</div><div>{</div></div><br> in <br><span style="color:rgb(106,153,85);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;white-space:pre">/* Read packet from RX queues. 8< */</span> second loop </div><div><br></div></div> <span style="background-color:rgb(0,0,0)">f<font color="#f3f3f3">or (i = 0; i < qconf->n_rx_port; i++) {<br> portid = qconf->rx_port_list[i];<br> nb_rx = rte_eth_rx_burst(portid, 0,<br> pkts_burst, MAX_PKT_BURST);<br> port_statistics[portid].rx += nb_rx;<br><br> for (j = 0; j < nb_rx; j++) {<br> m = pkts_burst[j];<br> uint32_t ip_dst,ip_src;<br> struct rte_ipv4_hdr *ip_hdr;<br> rte_prefetch0(rte_pktmbuf_mtod(m, void *));<br> ip_hdr = rte_pktmbuf_mtod(m, struct rte_ipv4_hdr *);<br> ip_dst = rte_be_to_cpu_32(ip_hdr->dst_addr);<br> ip_src = rte_be_to_cpu_32(ip_hdr->src_addr);<br> RTE_LOG(INFO, L2FWD, "ITE one ----nb_rx=%u portid=%u\n", nb_rx,portid);<br> RTE_LOG(INFO, L2FWD, "ITR ---2----src=%u dest=%u\n",ip_src,ip_dst);<br> l2fwd_simple_forward(m, portid);<br> }<br> }</font></span><div>Please let me know why the ping message is not reaching dpdk even though on the source side</div><div><br></div><div>Thanks,</div><div>Dinesh </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 18, 2023 at 3:51 AM J.J. Martzki <<a href="mailto:mars14850@gmail.com">mars14850@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">From your picture, l3fwd is running in VM2, with a tap vdev. But<br>
what's the relationship between VM1 and VM2? Are they running on the<br>
same host?<br></blockquote><div><font color="#ff0000">Yes, They are running on the same host with 3 NICs two are bind with dpdk and one is with kernel module, These NICs are on separate subnets </font></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I assume:<br>
1. VM1 and VM2 are running on the same host.<br>
2. ens05 is a net device in VM1.<br>
3. VM1 has its own net device, let's call it ens06, and IP2 is on it.<br><font color="#ff0000">
Yes</font>:<br></blockquote>0<font color="#ff0000">000:00:05.0 'Ethernet Virtual Function 700 Series 154c' drv=uio_pci_generic unused=iavf,vfio-pci<br>0000:00:06.0 'Ethernet Virtual Function 700 Series 154c' drv=uio_pci_generic unused=iavf,vfio-pci<br><br>Network devices using kernel driver<br>===================================<br></font><div><font color="#ff0000">0000:00:03.0 'Virtio network device 1000' if=ens3 drv=virtio-pci unused=vfio-pci,uio_pci_generic *Active*</font></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Based on the assumptions above, you should make sure VM1 and VM2 can<br>
communicate with each other without l3fwd, just through the linux<br>
network stack.<br></blockquote><div>Yes . I tested with ens05 and ens03 and ping is working fine without dpdk.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
This ensures the link between these 2 VMs is good (ens05 in VM2 and<br>
ens06 in VM1 can reach each other directly). Maybe they are connected<br>
by a linux bridge on the host or something like that.<br>
<br>
And you l3fwd should attach to ens05, by vfio or uio. Just use<br>
dpdk-devbind.py to do this.<br>
After this, ens05 is no longer available in the kernel, and l3fwd can<br>
use it. As your command line '-p 0x3' described, l3fwd will use port 0<br>
and port 1.<br>
You can use l3fwd init log to make sure there are 2 ports available<br>
for l3fwd as we expected.<br>
<br>
BTW, I suggest you start from test-pmd first. test-pmd can connect<br>
ports ethier. But it supports interactive mode and many command lines.<br>
Such as display of port info like MAC you mentioned before.<br>
<br>
> Please share some link that has details descriptions about testing this flow :<br>
I didn't find a suitable article to describe this, maybe you can try:<br>
<a href="https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk" rel="noreferrer" target="_blank">https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk</a><br>
In this article, the DPDK application is ovs, just like your l3fwd.<br>
And it can attach many ports not only vhost ports but also physical<br>
ports, vdev ports like tap and so on.<br>
<br>
J.J. Martzki,<br>
Best wishes.<br>
<br>
<br>
Dinesh Kumar <<a href="mailto:raidinesh@utexas.edu" target="_blank">raidinesh@utexas.edu</a>> 于2023年4月17日周一 23:47写道：<br>
><br>
> Hi JJ,<br>
> Thank you for the clarification. I am new to this so trying to understand how it will work.<br>
><br>
> Please share some link that has details descriptions about testing this flow :<br>
><br>
> | VM1 | HOST<br>
> | VM2 |<br>
> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app<br>
><br>
> Sorry for my misunderstanding.<br>
><br>
> here is the correct flow I am working on now :<br>
> After running the below command from VM2, I am pinging IP address of VM1 vi network namespace ns0, but not able to see any message in l3fwd and ping is unreachable.<br>
> Thanks for your time and any help will be really appreciated.<br>
><br>
><br>
><br>
> On Fri, Apr 14, 2023 at 10:46 PM J.J. Martzki <<a href="mailto:mars14850@gmail.com" target="_blank">mars14850@gmail.com</a>> wrote:<br>
>><br>
>> > Is this process correct to assign IP and then device to DPDK?<br>
>> No, after binding the driver to vfio/uio, the IP configuration is no<br>
>> longer available.<br>
>> The IP is something belonging to an application, not a driver (DPDK).<br>
>><br>
>> > Is there any way to see ip and ethernet address after binding to DPDK?<br>
>> MAC is available for the rte_eth_dev, there's some API like<br>
>> rte_eth_macaddr_get() to get MAC<br>
>> from a specific port. l3fwd should have some CLI commands for getting<br>
>> MAC ethier, maybe<br>
>> you can read it's documents.<br>
>> IP is not available as I said before.<br>
>><br>
>> BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate<br>
>> VMs? It's not gonna work.<br>
>> AFAK, l3fwd switches packets from ports which attached to it. You<br>
>> can't use 2 separate l3fwd<br>
>> to connect 2 VMs.<br>
>><br>
>> If you are tring to connect 2 VMs, the classic topo is something like:<br>
>> | VM1 | HOST<br>
>> | VM2 |<br>
>> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app<br>
>><br>
>> J.J. Martzki,<br>
>> Best wishes.<br>
>><br>
>> Dinesh Kumar <<a href="mailto:raidinesh@utexas.edu" target="_blank">raidinesh@utexas.edu</a>> 于2023年4月14日周五 01:32写道：<br>
>> ><br>
>> > Hello Experts,<br>
>> ><br>
>> > I am trying to run the following scenario using l3fwd example in dpdk-stable-21.11.2 release .<br>
>> ><br>
>> > The flow is :<br>
>> ><br>
>> > VM1 VM2<br>
>> > ns0->tap0<IPV4 IP>->DPDK <---------------------> DPDK-> bind a device with IP <IPV4 IP ><br>
>> ><br>
>> ><br>
>> > The first question I have for VM2:<br>
>> > I assigned IP to the device and can see eth and static ip address via 'ip add ', then after binding it to dpdk, I can not see this information via ip add because it is not controlled by kernel module.<br>
>> > Is there any way to see ip and ethernet address after binding to DPDK?<br>
>> > Is this process correct to assign IP and then device to DPDK?<br>
>> ><br>
>> ><br>
>> > on VM1:<br>
>> > After starting the DPDK application, I can see a tap0 device created in kernel mode via 'ip add show'.Then I attached this interface with network namespace via:<br>
>> ><br>
>> > ip netns add ns0<br>
>> > ip link set tap0 netns ns0<br>
>> > ip netns exec ns0 ip addr add <IPV4 Address> dev tap0<br>
>> > ip netns exec ns0 ip link set tap0 up<br>
>> > ip netns exec ns0 ip link set lo up<br>
>> ><br>
>> > Still, I am not able to ping anything from this ns0.<br>
>> ><br>
>> > Do I need any other configuration to ping via ns0 to VM2?<br>
>> ><br>
>> > Sorry, in case I am asking any dumb questions, I am new to DPDK and trying to make the use case work.<br>
>> > Any help/pointer is really appreciated.<br>
>> ><br>
>> ><br>
</blockquote></div></div></div>