<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hi Harry, </div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Can we call rte_eth_dev_rx_queue_stop() on a rx queue when a fast path is still polling the queue? The sequence on control and fast path cores would like:</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Control path:</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
rte_eth_dev_rx_queue_stop(rx_queue_id);</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
...waiting for draining of rx_queue...</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
rte_eth_dev_stop()</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
....</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Fast path:</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Keep calling rte_eth_rx_burst()</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
(I am expecting it will return 0 if queue is already drained and stopped)</div>
<div class="elementToProof" style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<p> </p>
<p><i>Thanks,</i></p>
<p>Changchun</p>
<p> </p>
</div>
<div id="appendonsend"></div>
<div style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Helvetica Neue", sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Van Haaren, Harry <harry.van.haaren@intel.com><br>
<b>Sent:</b> Friday, February 14, 2025 4:22 AM<br>
<b>To:</b> NAGENDRA BALAGANI <nagendra.balagani@oracle.com>; users@dpdk.org <users@dpdk.org><br>
<b>Subject:</b> [External] : Re: Query Regarding Race Condition Between Packet Reception and Device Stop in DPDK</span>
<div> </div>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> From: NAGENDRA BALAGANI <nagendra.balagani@oracle.com></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Sent: Friday, February 14, 2025 8:43 AM</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> To: users@dpdk.org <users@dpdk.org></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Subject: Query Regarding Race Condition Between Packet Reception and Device Stop in DPDK</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> </div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Hi Team,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Ni Nagendra,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> We are facing a race condition in our DPDK application where one thread is reading packets from queue using rte_eth_rx_burst() , while another thread is attempting to stop the device using rte_eth_dev_stop(). This is causing instability, as the reading thread
may still be accessing queues while the device is being stopped.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
This is as expected - it is not valid to stop a device while other cores are using it.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Could you please suggest the best way to mitigate this race condition without impacting fast path performance? We want to ensure safe synchronization while maintaining high throughput.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
There are many implementations possible, but the end result of them all is "ensure that the dataplane core is NOT polling a device that is stopping".</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
1) One implementation is using a "force_quit" boolean value (see dpdk/examples/l2fwd/main.c for example). This approach changes the lcore's "while (1)" polling loop, and turns it into a "while (!force_quit)". (Note some nuance around "volatile" keyword for
the boolean to ensure reloading on each iteration, but that's off topic).</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
2) Another more flexible/powerful implementation could be some form of message passing. For example imagine the dataplane thread and control plane (stopping ethdev) thread are capable of communicating by sending an "event" to eachother. When a "stop polling"
event is recieved by the dataplane thread, it disables polling just that eth device/queue, and responds with a "stopped polling" reply. On recieving the "stopped polling" event, the thread that wants to stop the eth device can now safely do so.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Both of these implementations will have no datapath performance impact:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
1) a single boolean value check (shared state cache-line, likely in the core's cache) per iteration of polling of the app is super lightweight</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
2) an "event ringbuffer" check (when empty, also shared-state, likely in cache) per iteration is also super light.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
General notes on the above:</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
There's even an option to only check the boolean/event-ringbuffer once every N iterations: this will cause even less overhead, but will increase the latency of event action/reply on the datapath thread. As almost always, it depends on what's important for your
use-case!</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The main difference between implementation 1) and 2) above can be captured by this phrase: "Do not communicate by sharing memory; instead, share memory by communicating.", which I read at the Rust docs here:
<a href="https://urldefense.com/v3/__https://doc.rust-lang.org/book/ch16-02-message-passing.html__;!!ACWV5N9M2RV99hQ!PJ2tOtZcFXixBAPvzdcZl6JNtpUlX739wECm2pM-nD2ETdGn8kg48hadNTgteBLeIbz73d_gFfqnBjyJTTc4vwVoAeQTAFU$" id="OWA820d6899-7c96-7c76-ffc2-4e0fb94a6a0b" class="OWAAutoLink" data-auth="NotApplicable">
https://doc.rust-lang.org/book/ch16-02-message-passing.html</a>. 1) literally shares memory (both threads access the force_quit value directly). 2) focusses on communicating: which enables avoiding the race condition in a more powerful/elegant way (and future
proof too - it allows adding new event types cleanly, which the force_quit bool value does not.) I like this design-mentality, as it is a good/high-performance/scalable way of interacting between threads, and scales to future needs too: so I recommend approach
2.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Looking forward to your insights.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
></div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Regards,</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
> Nagendra </div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards, -Harry</div>
</body>
</html>