<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hello all,</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I am looking for guidance on using the mlx5 PMD in <span style="font-weight: 600;">
switchdev mode</span> with the DPDK primary/secondary process model.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 14pt; color: rgb(0, 0, 0);">
<span style="font-weight: 700;">Environment</span></div>
<ul data-editing-info="{"applyListStyleFromLevel":true}" style="direction: ltr; text-align: left; margin-top: 0px; margin-bottom: 0px; list-style-type: disc; background-color: rgb(255, 255, 255);">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
DPDK 26.03</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
mlx5 NIC</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
NIC configured in switchdev mode</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
One DPDK primary process and one DPDK secondary process</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
Both applications use the same PCI allow-list containing the PF and VFs</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The PF and VFs remain bound to the mlx5 kernel driver</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
Both processes use the same DPDK multi-process configuration and shared-memory prefix</div>
</li></ul>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The primary successfully probes the PF and available representor ports with the 'representor' devarg. However, the secondary does not successfully probe the PF or representor devices, even though the PF and VFs are included in the allow-list. Consequently,
those devices are absent from the secondary’s ethdev list and do not appear when iterating with RTE_FOREACH_DEV after initialization.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The server we are running has other applications that require the NIC to be switchdev mode. The PF is the E-Switch master, while the secondary processes traffic through a VF ethdev. The main question is how the PF, VF, and representor ethdevs are expected to
be managed when some devices are visible only to the primary.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>Our current understanding is that:</b></div>
<ul data-editing-info="{"applyListStyleFromLevel":true}" style="direction: ltr; text-align: left; margin-top: 0px; margin-bottom: 0px; list-style-type: disc; background-color: rgb(255, 255, 255);">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The primary owns the PF and the E-Switch master.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The PF may not be independently attachable or visible to the secondary.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
PF-specific operations, such as creating or destroying E-Switch transfer rules, should be performed by the primary.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The secondary may process traffic through a VF ethdev, while control-plane operations involving the PF are handled by the primary.</div>
</li></ul>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Could someone confirm whether this is the expected architecture for mlx5 switchdev with DPDK primary/secondary processes?</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; line-height: 1.33333; background-color: rgb(255, 255, 255); margin: 2em 0px 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 14pt; color: rgb(0, 0, 0);">
<span style="font-weight: 700;">Primary/secondary hotplug behavior</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px 0px 1.25em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I reviewed the DPDK 26.03 hotplug implementation and confirmed that primary/secondary synchronization is performed for both hotplug add and remove operations.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
For hotplug add, the primary probes the device locally and then sends an attach request to all registered secondary processes. If communication fails or a secondary cannot attach the device, the primary rolls back the local attach and notifies the secondaries
of the rollback.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
For hotplug remove, the primary first sends a synchronous remove request through the EAL multiprocess channel to all registered secondaries. Only after that coordination does it proceed with the local bus-specific device removal. This generic synchronization
is part of the common EAL hotplug path and is not dependent on the bus or PMD.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The mlx5 PMD also performs its own multi-process coordination during device stop and close operations. Therefore, removing an mlx5 device may involve both mlx5-specific notifications and the generic EAL hotplug notification.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This creates a problem in our configuration:</div>
<ul data-editing-info="{"applyListStyleFromLevel":true}" style="direction: ltr; text-align: left; margin-top: 0px; margin-bottom: 0px; list-style-type: disc; background-color: rgb(255, 255, 255);">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The primary can probe the PF and create representors.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The secondary cannot probe or attach to the PF or representors.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
A primary hotplug add still attempts to attach the device in every registered secondary.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
A primary hotplug remove still attempts to remove the device from every registered secondary.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
The secondary cannot successfully mirror these operations because the device does not exist in its ethdev view.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); margin-top: 0.5em; margin-bottom: 0.5em;">
<div class="elementToProof" role="presentation" style="direction: ltr; margin: 0.5em 0px;">
As a result, the primary hotplug operation can fail or be rolled back because the secondary cannot attach to or remove the device.</div>
</li></ul>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This appears to be a mismatch between the generic primary/secondary hotplug model and the mlx5 switchdev model, where the PF and representors may be primary-only resources. Managing ethdevs in the primary application always has implications in secondary, so
how is this supposed to be handled in switchdev mode when secondary cannot even probe the PF ports?</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="font-weight: 700;"><br>
</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="font-weight: 700;"><br>
</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; line-height: 1.33333; background-color: rgb(255, 255, 255); margin: 2em 0px 1em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 14pt; color: rgb(0, 0, 0);">
<span style="font-weight: 700;">Telemetry issue</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px 0px 1.25em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
There is also a related issue involving the DPDK telemetry thread.</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;">The telemetry thread sends an
</span><span style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; font-weight: 600;"><code style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;">ethdev/info</code></span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;"> request
for port 0. Port 0 (PF E-switch master) has a valid MAC address in our application. However, in the failure path we observed,
</span><span style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; font-weight: 600;"><code style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;">format_ether_addr()</code></span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;"> is
called with a NULL pointer while formatting the telemetry response. The helper dereferences or formats the address pointer without first checking whether it is NULL, resulting in a segmentation fault.</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-size: 11pt; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;">Would it be appropriate for
</span><span style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; font-weight: 600;"><code style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;">format_ether_addr()</code></span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;"> to
handle a NULL pointer defensively—for example, by formatting the address as all zeroes—or should the caller guarantee that the pointer is always valid and return an error if it is not?</span></div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
A possible defensive behavior would be conceptually similar to:</div>
<pre class="elementToProof" style="background-color: rgb(255, 255, 255); margin: 1.71429em 0px; padding-top: 1.75rem; padding-bottom: 0.857143em; border-radius: 6px; max-width: 100%;"><div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; line-height: 1.5; white-space: pre-wrap; margin: 0.5em 0px; padding: 1em; border-radius: 0.375rem; font-family: Consolas, Monaco, "Andale Mono", "Ubuntu Mono", monospace; font-size: 11pt; color: rgb(0, 0, 0);"><code style="font-family: Consolas, Monaco, "Andale Mono", "Ubuntu Mono", monospace;">if (addr == NULL) {
/* Format as 00:00:00:00:00:00,
* or report an invalid address.
*/
}</code></div></pre>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I would appreciate feedback on:</div>
<ul style="direction: ltr; text-align: left; margin: 1.25em 0px; padding-left: 0px; list-style-position: initial; list-style-type: disc; background-color: rgb(255, 255, 255);">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); direction: ltr; margin-top: 0.5em; margin-bottom: 0.5em;">
Whether a NULL MAC-address pointer is expected to be possible in the <span style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; font-weight: 600;" role="presentation">
<code style="font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;">ethdev/info</code></span> telemetry path</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); direction: ltr; margin-top: 0.5em; margin-bottom: 0.5em;">
Whether a NULL check and all-zero MAC formatting would be acceptable upstream behavior</li></ul>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 0px 0px 1.25em; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="direction: ltr; text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1.25em 0px 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards,<br>
Dylan</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
</body>
</html>