<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Stephen,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I’m thinking, perhaps the best path forward for now, is to continue using a private mbuf layout and carry the collateral internally — at least for now. The changes are getting well beyond the initial desired impacts and has potential make things become unstable.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I would love to see some sort of solution along these lines in the future so we can discontinue the use of a private patch. To be clear, the solution is not built around cnxk, cnxk is just impacted — we have several other platform builds that do not use cnxk
but we still carry the metadata in the mbuf itself for performance reasons.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
WRT the skip functions, we fit under the 384 max for wqe but there’s no room for additional growth beyond that with how it’s laid out. The other 2 are not as sensitive to the size but still have valid limits which are well beyond our use case. Today we carry
256 bytes which caps us on wqe.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
That said, we plan to discuss internally next week on how to move forward as we prep for our v25.11 upgrade work. I want to thank you and Morten, however, for your time and support on this work.</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-rt</div>
<div style="direction: ltr; font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div style="padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor;">
<div style="text-align: left; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Stephen Hemminger <stephen@networkplumber.org><br>
<b>Date: </b>Friday, October 9, 2026 at 12:14 PM<br>
<b>To: </b>Randy Tice (rtice) <rtice@cisco.com><br>
<b>Cc: </b>dev@dpdk.org <dev@dpdk.org>; Morten Brørup <mb@smartsharesystems.com>; Nithin Dabilpuram <ndabilpuram@marvell.com>; Harman Kalra <hkalra@marvell.com><br>
<b>Subject: </b>Re: [PATCH v4 0/1] mbuf: add runtime metadata dynamic-field storage<br>
<br>
</div>
</div>
<div id="mail-editor-reference-message-body">
<div class="ms-outlook-mobile-reference-message skipProofing">
<meta name="Generator" content="Microsoft Exchange Server">
</div>
<div class="PlainText" style="font-size: 11pt;">On Thu, 8 Oct 2026 16:15:26 +0000<br>
"Randy Tice (rtice)" <rtice@cisco.com> wrote:<br>
<br>
> I did a brief look at this and still not convinced about the exact use case.<br>
> What is the problem this is trying to solve and why can't it be done<br>
> by using existing API’s.<br>
><br>
> #RT:<br>
> Our implementation currently requires an additional 256 bytes of metadata per<br>
> mbuf. That does not fit in the existing dynamic-field storage, so today we<br>
> carry a private patch to extend struct rte_mbuf. The goal of this work is to<br>
> replace that private struct change with a supported upstream mechanism.<br>
><br>
> We did look at using mbuf private data first. It can work when the application<br>
> owns all mbuf pool creation, but it is not sufficient for our case without<br>
> another global/base reservation mechanism. Some mbuf pools are created by<br>
> drivers or libraries rather than directly by the application; the CNXK inline<br>
> IPsec/OOP meta pool is one example (NIX_INL_META_POOL, created through<br>
> cnxk_nix_inl_meta_pool_cb()). To make private data work there, we had to add<br>
> an EAL argument that reserved a base private size for all pktmbufs, including<br>
> PMD-created pools.<br>
><br>
> Private data also lacks a central layout registry. If multiple modules use<br>
> private data, they must coordinate offsets out of band to avoid overlaying<br>
> each other. The dynamic-field registry solves that coordination problem, but<br>
> the existing copied dynamic-field area is too small and has copy/clone<br>
> semantics that are wrong for this metadata.<br>
><br>
> That is why this version uses a globally configured per-mbuf metadata area<br>
> managed by the dynamic-field registry, with explicit metadata fields that are<br>
> not copied by generic copy/clone/attach paths. This direction came out of the<br>
> prior discussion with you, Morten, and me: avoid a Cisco-private mbuf struct<br>
> patch, avoid per-pool private-data layout coordination, and keep sizeof(struct<br>
> rte_mbuf) fixed.<br>
<br>
You uncovered a design flaw in the CNXK driver and the<br>
proposed solution is wider than it needs to be.<br>
<br>
All mbufs visible to application must come from mempools controlled by the application.<br>
The design of CNXK driver is wrong, it shouldn't be using a private hidden pool.<br>
It is ok for drivers to have hidden mempools that are used for non-visible things,<br>
an example is the packet capture mempool where the mbufs only go into capture stream.<br>
<br>
More long winded AI description:<br>
<br>
On Thu, 8 Oct 2026 16:15:26 +0000<br>
"Randy Tice (rtice)" <rtice@cisco.com> wrote:<br>
<br>
> We did look at using mbuf private data first. It can work when the<br>
> application owns all mbuf pool creation, but it is not sufficient for<br>
> our case without another global/base reservation mechanism. Some mbuf<br>
> pools are created by drivers or libraries rather than directly by the<br>
> application; the CNXK inline IPsec/OOP meta pool is one example<br>
<br>
That is the real bug. A driver should not hand the application mbufs<br>
from a pool the application did not create. The Rx queue API already<br>
takes the pool from the application, and rte_eth_rxconf can carry<br>
more than one (rx_mempools). If cnxk needs a meta pool whose mbufs<br>
reach the application, it should get it from the application, or at<br>
minimum create it with the same priv_size as the Rx queue pool.<br>
<br>
Internal pools are fine when the mbufs never cross the API, e.g.<br>
dumpcap or the bonding LACP pool.<br>
<br>
With that fixed, the existing private area does what you need. It is<br>
per pool, sized by the application, and is not copied by copy, clone<br>
or attach. Offset coordination inside it is the application's job,<br>
and if a registry is wanted it can be layered on top of priv without<br>
changing the mbuf layout.<br>
<br>
So my answer to option 1 vs 2 is neither. I don't want a global<br>
layout knob that puts a load in every inline helper and needs<br>
per-driver range checks, to work around one driver.<br>
<br>
What I would take:<br>
- cnxk: validate wqe_skip/later_skip/first_skip against priv_size<br>
and headroom. This is a bug today, independent of this series.<br>
- cnxk: meta pool comes from, or matches, the application pool.<br>
- ethdev: under RTE_ETHDEV_DEBUG_RX, check that received mbufs<br>
belong to a pool configured on that queue.<br>
<br>
Nithin, Harman: do mbufs from NIX_INL_META_POOL get returned to the<br>
application by rx_burst, or are they only consumed internally?<br>
</div>
</div>
</div>
</body>
</html>