<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi Stephen, Thea,
<div><br>
</div>
<div>If you uses SRIOV, then containers behave essentially like VMs, packets will be exchanged through the PCIe bus and switched on the NIC ASIC, which as Stephen mentions, will identify MAC addresses as « itself » and packets do not physically get out of the
NIC . I’d argue these days it’s not as much of a problem. You can typically have a PCIe5 x16 ConnectX7 that has a bus BW of 500Gbps but has actually only one or two 100G ports, so you’ve got plenty of spare bandwidth for internal host exchange. We know NICs
are getting smart and take a broader role than pure « external » I/O.</div>
<div><br>
</div>
<div>Internal host networking without going through PCIe can be handled like VMs too : with virtio and the DPDK vhost driver. Memory copies are involved in that case.</div>
<div><br>
</div>
<div>I suspect for your matter at hand Thea, the easiest is to use SRIOV. Research-wise, a simple solution is to use —networking=host … </div>
<div><br>
</div>
<div>Eg this is working well but uses privileged container and lets the docker access all host network for fastclick :</div>
<div>sudo docker run -v /mnt/huge:/dev/hugepages -it --privileged --network host tbarbette/fastclick-dpdk:generic --dpdk -a $VF_PCIE_ADDR -- -e "FromDPDKDevice(0) -> Discard;" </div>
<div><br>
</div>
<div>The related sample Dockerfile can be found at : <a href="https://github.com/tbarbette/fastclick/blob/main/etc/Dockerfile.dpdk">https://github.com/tbarbette/fastclick/blob/main/etc/Dockerfile.dpdk</a></div>
<div><br>
</div>
<div>A problem also with DPDK-based dockers is that you generally don’t want to keep the -march=native, so personally I got that script to build a version of my docker image with many architectures : <a href="https://github.com/tbarbette/fastclick/blob/main/etc/docker-build.sh">https://github.com/tbarbette/fastclick/blob/main/etc/docker-build.sh</a> so
the user can use the image that targets their own arch.</div>
<div><br>
</div>
<div><br>
</div>
<div>May that be helpful,</div>
<div><br>
</div>
<div>Tom</div>
<div>
<div><br>
<blockquote type="cite">
<div>Le 20 nov. 2024 à 08:10, Kompella V, Purnima <Kompella.Purnima@commscope.com> a écrit :</div>
<br class="Apple-interchange-newline">
<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; letter-spacing: revert !important; orphans: auto; text-transform: revert !important; widows: auto; word-spacing: revert !important; -webkit-text-stroke-width: 0px; text-decoration: none; aspect-ratio: revert !important; background: revert !important; block-size: revert !important; bottom: revert !important; color: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; font-size: revert !important; height: revert !important; hyphens: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; padding: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; text-align: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; word-break: revert !important; writing-mode: revert !important; zoom: revert !important; border: 0px !important; display: table !important; width: 577px; table-layout: fixed !important; float: none !important; border-spacing: 0px !important;">
<tbody style="aspect-ratio: revert !important; background: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; font-size: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; padding: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; width: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important; display: block !important;">
<tr style="aspect-ratio: revert !important; background: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; display: revert !important; font-size: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; padding: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; width: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important;">
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="aspect-ratio: revert !important; background-image: revert !important; background-position: revert !important; background-size: revert !important; background-repeat: revert !important; background-attachment: revert !important; background-origin: revert !important; background-clip: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; display: revert !important; font-size: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important; padding: 7px 2px !important; background-color: rgb(166, 166, 166) !important; width: 0px !important;">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" style="aspect-ratio: revert !important; background-image: revert !important; background-position: revert !important; background-size: revert !important; background-repeat: revert !important; background-attachment: revert !important; background-origin: revert !important; background-clip: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; display: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important; width: 543px; background-color: rgb(234, 234, 234) !important; padding: 7px 5px 7px 15px !important; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif !important; font-size: 12px !important; font-weight: normal !important; color: rgb(33, 33, 33) !important; text-align: left !important; overflow-wrap: break-word !important;">
<div style="aspect-ratio: revert !important; background: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; display: revert !important; font-size: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; padding: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; width: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important;">
Vous n’obtenez pas souvent d’e-mail à partir de<span class="Apple-converted-space"> </span><a href="mailto:kompella.purnima@commscope.com" style="color: blue; text-decoration: underline;">kompella.purnima@commscope.com</a>.<span class="Apple-converted-space"> </span><a href="https://aka.ms/LearnAboutSenderIdentification" style="color: revert !important; text-decoration: underline; background: revert !important; content-visibility: revert !important; direction: revert !important; display: revert !important; font-size: revert !important; opacity: revert !important; rotate: revert !important; scale: revert !important; visibility: revert !important;">Pourquoi
c’est important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" style="aspect-ratio: revert !important; background-image: revert !important; background-position: revert !important; background-size: revert !important; background-repeat: revert !important; background-attachment: revert !important; background-origin: revert !important; background-clip: revert !important; block-size: revert !important; border: revert !important; bottom: revert !important; color-scheme: revert !important; content-visibility: revert !important; cursor: revert !important; direction: revert !important; display: revert !important; height: revert !important; hyphens: revert !important; letter-spacing: revert !important; line-height: revert !important; margin: revert !important; opacity: revert !important; order: revert !important; outline: revert !important; overflow: revert !important; position: revert !important; resize: revert !important; rotate: revert !important; scale: revert !important; tab-size: revert !important; table-layout: revert !important; text-indent: revert !important; text-orientation: revert !important; text-overflow: revert !important; text-shadow: revert !important; text-transform: revert !important; text-wrap: revert !important; top: revert !important; transition: revert !important; vertical-align: revert !important; visibility: revert !important; white-space: revert !important; word-break: revert !important; word-spacing: revert !important; writing-mode: revert !important; zoom: revert !important; width: 75px !important; background-color: rgb(234, 234, 234) !important; padding: 7px 5px !important; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif !important; font-size: 12px !important; font-weight: normal !important; color: rgb(33, 33, 33) !important; text-align: left !important; overflow-wrap: break-word !important;">
</td>
</tr>
</tbody>
</table>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">Hi Stephen,<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">A parallel question about packet-flow between VFs of the same PF when VFs are assigned to different containers on the same host
server<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;">
<span style="font-size: 12pt; color: rgb(31, 73, 125);">Create 2 SRIOV-VFs of a PF in the host and assign them to 2 containers (one VF per container)<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;">
<span style="font-size: 12pt; color: rgb(31, 73, 125);">send IP packet from container-1 to container-2 (SRC_MAC address in this ethernet frame = container1 VF’s MAC address, DST_MAC address = container2 VF’s MAC address),<o:p></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;">
<span style="font-size: 12pt; color: rgb(31, 73, 125);">container-1 sends packet by calling<span class="Apple-converted-space"> </span></span><span style="font-size: 10pt; font-family: Consolas; background: rgb(212, 212, 212);">rte_eth_tx_burst()</span><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p></o:p></span></div>
<div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 0.5in;">
<span style="font-size: 12pt; color: rgb(31, 73, 125);">container-2 is polling for packets from its VF by calling</span><span style="font-size: 10pt; font-family: Consolas; background: rgb(212, 212, 212);">rte_eth_rx_burst()</span><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">Will the packet in above scenario leave the host server, go the switch and then come back to the same host machine for entering
container-2 ?<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">Or, is the SRIOV in PF-NIC smart to identify that SRC_MAC and DST_MAC of the ethernet frame are its own VFs and hence it routes
the packet locally within the NIC (packet doesn’t reach the switch at all) ?<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">Regards,<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);">Purnima<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; color: rgb(31, 73, 125);"><o:p> </o:p></span></div>
<div>
<div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0in 0in;">
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="Apple-converted-space"> </span>Stephen Hemminger <stephen@networkplumber.org><span class="Apple-converted-space"> </span><br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday, November 20, 2024 3:34 AM<br>
<b>To:</b><span class="Apple-converted-space"> </span>Thea Corinne Rossman <thea.rossman@cs.stanford.edu><br>
<b>Cc:</b><span class="Apple-converted-space"> </span>users@dpdk.org<br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: Containernet (Docker/Container Networking) with DPDK?<o:p></o:p></div>
</div>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div>
<p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background: white;">
</p>
</div>
<div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div style="margin: 0in;">On Tue, 19 Nov 2024 13:39:38 -0800<o:p></o:p></div>
<div style="margin: 0in;">Thea Corinne Rossman <<a href="mailto:thea.rossman@cs.stanford.edu" style="color: blue; text-decoration: underline;">thea.rossman@cs.stanford.edu</a>> wrote:<o:p></o:p></div>
<div style="margin: 0in;"> <o:p></o:p></div>
<div style="margin: 0in;">> This is SO helpful -- thank you so much.<o:p></o:p></div>
<div style="margin: 0in;">><span class="Apple-converted-space"> </span><o:p></o:p></div>
<div style="margin: 0in;">> One follow-up question regarding NICs: can multiple containers on the same<o:p></o:p></div>
<div style="margin: 0in;">> host share the same PCI device? If I have a host NIC with (say) VFIO driver<o:p></o:p></div>
<div style="margin: 0in;">> binding, do I have to split it with some kind of SR-IOV so that each<o:p></o:p></div>
<div style="margin: 0in;">> container has its own "NIC" binding? Or, when running DPDK's "devbind"<o:p></o:p></div>
<div style="margin: 0in;">> script, can I set up each one with the same PCI address?<o:p></o:p></div>
<div style="margin: 0in;"> <o:p></o:p></div>
<div style="margin: 0in;"> <o:p></o:p></div>
<div style="margin: 0in;">Totally depends on what container system you are using.<o:p></o:p></div>
<div style="margin: 0in;">If you have two containers sharing same exact PCI device, chaos would ensue.<o:p></o:p></div>
<div style="margin: 0in;">You might be able to make two VF's on host and pass one to each container;<o:p></o:p></div>
<div style="margin: 0in;">that would make more sense. </div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>