<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hello Harry,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
I implemented a working echo server with your API.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
The code is here: <a href="https://github.com/getelson-at-mellanox/rdpdk/tree/safe-q" id="LPlnk" class="OWAAutoLink">
https://github.com/getelson-at-mellanox/rdpdk/tree/safe-q</a><br>
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Several changes:</div>
<ul data-editing-info="{"applyListStyleFromLevel":false,"unorderedStyleType":1}" style="margin-top: 0px; margin-bottom: 0px; list-style-type: disc;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">DPDK configuration is split to 3 mandatory steps:</div>
</li><ul data-editing-info="{"applyListStyleFromLevel":true}" style="margin-top: 0px; margin-bottom: 0px; list-style-type: circle;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">port configuration in <br>
Port::configure(&mut self, rxq_num: u16, txq_num: u16) -> Result<(), String></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">Rx queues configuration in <br>
Port::config_rxqs(&mut self, desc_num: u16, mempool: DpdkMempool) -> Result<(), String></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">Tx queues configuration in<br>
Port::config_txqs(&mut self, desc_num: u16) -> Result<(), String></div>
</li></ul>
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">In the IO thread, I renamed the `enable_polling()` to `activate()` for Rx/Tx symmetry.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">I renamed `port` and `q` struct members to `port_id`, `queue_id` </div>
</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Build steps:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<ol start="1" data-editing-info="{"applyListStyleFromLevel":false,"orderedStyleType":1}" style="margin-top: 0px; margin-bottom: 0px; list-style-type: decimal;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">Apply <a href="https://github.com/getelson-at-mellanox/rdpdk/blob/safe-q/dpdk-patches/0001-rust-export-missing-port-objects.patch" id="LPlnk">
https://github.com/getelson-at-mellanox/rdpdk/blob/safe-q/dpdk-patches/0001-rust-export-missing-port-objects.patch</a> to DPDK source.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">Install DPDK </div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<div class="elementToProof">Set PKG_CONFIG_PATH to DPDK installation</div>
</li></ol>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Activation:</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
# cargo run --example echo -- -a <port PCI address></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Regards,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Gregory</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Van Haaren, Harry <harry.van.haaren@intel.com><br>
<b>Sent:</b> Friday, April 18, 2025 14:40<br>
<b>To:</b> Gregory Etelson <getelson@nvidia.com><br>
<b>Cc:</b> dev@dpdk.org <dev@dpdk.org>; Richardson, Bruce <bruce.richardson@intel.com>; owen.hilyard@unh.edu <owen.hilyard@unh.edu><br>
<b>Subject:</b> Re: [PATCH] rust: RFC/demo of safe API for Dpdk Eal, Eth and Rxq</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">External email: Use caution opening links or attachments<br>
<br>
<br>
> From: Etelson, Gregory<br>
> Sent: Thursday, April 17, 2025 7:58 PM<br>
> To: Van Haaren, Harry<br>
> Cc: dev@dpdk.org; getelson@nvidia.com; Richardson, Bruce; owen.hilyard@unh.edu<br>
> Subject: Re: [PATCH] rust: RFC/demo of safe API for Dpdk Eal, Eth and Rxq<br>
><br>
> Hello Harry,<br>
><br>
> Thank you for sharing the API.<br>
> Please check out my comments below.<br>
<br>
Thanks for reading & discussion!<br>
<br>
<snip><br>
<br>
> > +<br>
> > + pub fn start(&mut self) -> (Vec<RxqHandle>, Vec<TxqHandle>) {<br>
> > + // call rte_eth_dev_start() here, then give ownership of Rxq/Txq to app<br>
><br>
> After a call to Port::start, Rx and Tx queues are detached from it's port.<br>
> With that model how rte_eth_dev_stop() and subsequent rte_eth_dev_start()<br>
> DPDK calls can be implemented ?<br>
<br>
Correct, the RxqHandle and TxqHandle don't have a "back reference" to the port.<br>
There are a number of ways to ensure eth_dev_stop() cannot be called without the<br>
Rxq/Txqs being "returned" to the Port instance first.<br>
<br>
Eg: Use an Arc<T>. The port instance "owns" the Arc<T>, which means it is going to keep<br>
the Arc alive. Now give each Rxq/Txq a clone of this Arc. When the Drop impl of the<br>
Rxq/Txq runs, it will decrement the Arc. So just letting the Rxq/Txq go out of scope<br>
will be enough to have the Port understand that handle is now gone.<br>
<br>
The port itself can use Arc::into_inner function[1], which returns Option<T>. If the<br>
Some(T) is returned, then all instances of RxqHandle/TxqHandle have been dropped,<br>
meaning it is safe to eth_dev_stop(), as it is impossible to poll RXQs if there's no Rxq :)<br>
[1] <a href="https://doc.rust-lang.org/std/sync/struct.Arc.html#method.into_inner">
https://doc.rust-lang.org/std/sync/struct.Arc.html#method.into_inner</a><br>
<br>
// Pseudo-code here:<br>
Dpdk::Eth::Port::stop(&mut self) -> Result<(), Error> {<br>
let handles_dropped = self.handle_arc.into_inner(); // returns "T" if its the only reference to the Arc<br>
if handles_dropped.is_none() {<br>
return Err("an Rxq or Txq handle remains alive, cannot safely stop this port");<br>
}<br>
}<br>
<br>
There's probably a few others, but that's "idiomatic Rust" solution.<br>
We'd have to pass the Arc from the RxqHandle into the Rxq instance itself too,<br>
but that's fine.<br>
<br>
<snip><br>
<br>
> > +fn main() {<br>
> > + let mut dpdk = dpdk::Eal::init().expect("dpdk must init ok");<br>
> > + let rx_mempool = dpdk::Mempool::new(4096);<br>
> > +<br>
> > + let mut ports = dpdk.take_eth_ports().expect("take eth ports ok");<br>
><br>
> Eal::take_eth_ports() resets EAL ports.<br>
<br>
I don't think it "resets" here. The "take eth ports" removes the Port instances from<br>
the dpdk::Eal struct, but there's no "reset" behaviour.<br>
<br>
> A call to rte_dev_probe() will ether fail, because Eal::eth_ports is None<br>
> or create another port-0, depending on implementation.<br>
<br>
I don't see how or why rte_dev_probe() would be called. The idea is not to allow Rust<br>
apps call DPDK C APIs "when they want". The safe Rust API provides the required abstraction.<br>
So its not possible to have another call to rte_dev_probe(), after the 1st time under eal_init().<br>
<br>
Similar topic: Hotplug. I have experience with designing C APIs around hotplug<br>
use-cases (Music/DJ software, from before my DPDK/networking days!). I think DPDK has<br>
an interesting "push hotplug" approach (aka, App makes a function call to "request" the device).<br>
Then on successful return, we can call rte_eth_dev_get_port_by_name() to get the u16 port_id,<br>
and build the Port instance from that. Outline API:<br>
<br>
enum EalHotplugDev {<br>
EthDev(Dpdk::Eth::Port), // enums can have contents in Rust :)<br>
CryptoDev(Dpdk::Crypto),<br>
// Etc<br>
}<br>
<br>
Eal::hotplug_add(bus: String, dev: String, args: String) -> Result<EalHotplugDev, Error> {<br>
// TODO: call rte_eal_hotplug_add()<br>
// TODO: identify how to know if its an Eth, Crypto, Dma, or other dev type?<br>
match (dev_type) {<br>
"eth" => {<br>
let port_id = rte_eth_dev_get_port_by_name(dev);<br>
EalHotplugDev::EthDev( Dpdk::Eth::Port::new(port_id) )<br>
}<br>
}<br>
}<br>
<br>
Applications could then do:<br>
let Ok(dev) = eal.hotplug_add("pci", "02:00.0", "dev_option=true") else {<br>
// failed to hotplug, log error?<br>
return;<br>
}<br>
match (dev) {<br>
EthDev => {<br>
// handle the dev here, e.g. configure & spawn thread to poll Rxq like before.<br>
}<br>
}<br>
<br>
I like having an outline of difficult to "bolt on" features (hotplug is typically hard to add later..)<br>
but I recommend we focus on getting core APIs and such running before more detail/time/implementation here.<br>
<br>
<br>
Regards, -Harry<br>
</div>
</span></font></div>
</body>
</html>