# RCU for ethdev Rx/Tx callback removal (pdump, bpf_pkt)
2026-10-12, Stephen Hemminger
Proposal: replace the per-queue use counters in bpf_pkt and pdump with an RCU
read section inside the ethdev callback walker, using a membarrier-based
flavor so the reader needs no full barrier. The Rx/Tx path then has no
lock-prefixed instruction for callback protection, removal is safe with
MT_LOCKFREE Tx, and BPF programs and callback handles can be freed after a
stop.
## Today
`rte_eth_remove_rx_callback()` and `rte_eth_remove_tx_callback()` unlink the
callback and return. They do not wait for threads already inside it and do not
free it. The API text tells the caller to free it "after a short delay" or to
use RCU. Each in-tree user has its own workaround.
| User | Mechanism | Per-burst cost (x86-64) | Problem |
| --- | --- | --- | --- |
| bpf_pkt | Per-queue use counter, `rte_smp_mb()` at entry, recheck of `cbi->cb` | One `lock addl $0,(%rsp)`, two writes to the cbi cache line | Wrong with two threads on one queue. Callback handle never freed. |
| pdump | Per-queue use counter, release/acquire fences | None for the counter. Four lock-prefixed stats updates. Ring CAS when packets are captured. | Counter is unsound. Filter never destroyed. Callback handle never freed. |
### bpf_pkt
- `bpf_eth_cbi_inuse()` does `cbi->use++` then `rte_smp_mb()`. The callback
then rechecks `cbi->cb != NULL`.
- Unload sets `cb = NULL`, does `rte_smp_mb()`, and waits while `use` is odd.
- This is correct with one thread per queue.
- With `RTE_ETH_TX_OFFLOAD_MT_LOCKFREE` two threads can be in the callback for
one queue. Both increment `use` non-atomically. Two threads inside give an
even count, so unload proceeds and destroys the program under them.
- cnxk, octeontx, dpaa, dpaa2 and null advertise that offload.
- The comment above `bpf_eth_cbi_unload()` already says there is no safe
generic way to remove a callback while IO is active, and that the handle is
not freed for that reason.
### pdump
Two independent holes, each enough on its own.
First, there is no recheck after taking the counter. The list head is loaded
in `rte_eth_rx_burst()` before the callback runs:
```
datapath lcore control thread
-------------- --------------
cb = load(queue callback list)
rte_eth_remove_rx_callback()
pdump_cb_wait(): count even, return
reply sent; dumpcap frees ring, mempool
pdump_rx(): use_count++ (odd)
copy into freed ring and mempool
```
Second, `pdump_cb_hold()` is a store followed by a release fence, and
`pdump_cb_wait()` is an acquire fence followed by a load. That is the
store-buffer pattern and needs a full barrier on both sides. On x86 both
fences compile to nothing.
The window is a few instructions wide unless the reader is preempted inside
it.
## Goals and decisions
Goals:
- No lock-prefixed instruction or full barrier on the Rx/Tx path for callback
protection.
- No application changes. QSBR-style quiescent state calls are not acceptable.
- Correct with `MT_LOCKFREE` Tx.
- Correct for Rx/Tx from the EAL interrupt thread. Bonding mode 4 sends LACP
from an alarm callback.
- Free what is leaked today: the callback handle, and in pdump the filter.
Decisions already taken:
- Stop and unload cost is not a concern. A few extra syscalls are fine, a
sleep is not.
- Sync calls `membarrier()` twice, as liburcu does. It is not on the fast
path.
- Rx/Tx from an unregistered thread is an application bug, since
`rte_thread_register()` exists. The interrupt thread is the one exception
and is handled inside the rcu code.
- ethdev, pdump, bpf, eal and rcu may all change.
Out of scope for now:
- Fixing the cryptodev enqueue/dequeue callbacks. The findings are recorded
below.
- The multi-producer ring CAS in capture.
## Reader side
The reader marks itself in a per-thread slot with plain stores. Only a
compiler barrier follows the entry store. The hardware barrier is forced from
the writer side, as in liburcu's memb flavor.
Why not what DPDK has: `rte_rcu_qsbr` needs the application to report
quiescent states. Its `thread_online()`/`thread_offline()` pair works without
application changes, but online is a store plus a seq_cst fence. That is the
same lock per burst bpf_pkt pays now.
liburcu flavors for reference
([README](https://github.com/urcu/userspace-rcu/blob/master/README.md)):
| Flavor | Reader | Application change |
| --- | --- | --- |
| qsbr | Nothing | Periodic quiescent call |
| mb | Store plus full barrier | Register thread |
| memb | Store only, writer calls `membarrier()` | Register thread |
| bp | As memb, registers itself on first use | None |
| signal | As memb, via a signal | Removed upstream |
The proposal takes memb's barrier scheme and replaces thread registration with
fixed slots. liburcu is LGPL-2.1, so this is a reimplementation of the
algorithm, not a port.
Sketch, names are placeholders:
```c
struct rte_rcu_reader { /* one per slot, cache aligned */
RTE_ATOMIC(uint64_t) ctr; /* 0 idle, else token seen at entry */
uint32_t depth; /* nesting, owner only */
};
struct rte_rcu_domain {
RTE_ATOMIC(uint64_t) token; /* bumped by each sync */
bool membarrier; /* false: readers must fence */
struct rte_rcu_reader reader[RTE_MAX_LCORE + 1];
};
static inline void
rcu_enter(struct rte_rcu_domain *d, unsigned int slot)
{
struct rte_rcu_reader *r = &d->reader[slot];
uint64_t t;
if (r->depth++ != 0)
return;
t = rte_atomic_load_explicit(&d->token, rte_memory_order_relaxed);
rte_atomic_store_explicit(&r->ctr, t, rte_memory_order_relaxed);
if (unlikely(!d->membarrier))
rte_atomic_thread_fence(rte_memory_order_seq_cst);
else
rte_compiler_barrier();
}
static inline void
rcu_exit(struct rte_rcu_domain *d, unsigned int slot)
{
struct rte_rcu_reader *r = &d->reader[slot];
if (--r->depth != 0)
return;
rte_compiler_barrier();
rte_atomic_store_explicit(&r->ctr, 0, rte_memory_order_release);
}
```
- On x86-64 this is two `mov` at entry and one at exit. On arm64 it should be
`ldr`, `str`, then `stlr`, with no `dmb`. The arm64 code has not been
compiled.
- The exit store is a release so the fenced fallback needs no second fence.
With `membarrier()` a plain store would do.
- Each slot is written only by its owner, so no shared cache line is written
on the fast path.
- Nesting is needed because an Rx callback that transmits enters the Tx chain
on the same thread.
- Slot index is `rte_lcore_id()`. Index `RTE_MAX_LCORE` is reserved for the
interrupt thread.
## Writer side
The writer unlinks, calls sync, then frees. Sync calls `membarrier()` twice,
before and after waiting for readers, in the same places liburcu does.
```c
static void
rcu_mb_all(struct rte_rcu_domain *d)
{
if (d->membarrier)
membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED, 0, 0);
else
rte_atomic_thread_fence(rte_memory_order_seq_cst);
}
void
rcu_sync(struct rte_rcu_domain *d)
{
uint64_t t, c;
unsigned int i;
/* caller has already unlinked the object */
rcu_mb_all(d); /* readers now see the unlink */
t = rte_atomic_fetch_add_explicit(&d->token, 1,
rte_memory_order_seq_cst) + 1;
for (i = 0; i <= RTE_MAX_LCORE; i++) {
for (;;) {
c = rte_atomic_load_explicit(&d->reader[i].ctr,
rte_memory_order_acquire);
if (c == 0 || c >= t)
break;
rte_pause();
}
}
rcu_mb_all(d); /* readers' accesses are complete */
}
```
### Why two calls
`membarrier()` makes every running thread of the process execute a full
barrier at some point between the call and its return.
1. First call, after the unlink and before the token bump. A reader that
stored its `ctr` before this point is visible to the writer. A reader that
stores it afterwards also loads the list head afterwards, so it sees the
unlink and never holds the removed entry.
2. The wait. A slot reading 0 is idle or entered after the first call. A slot
at `t` or above loaded the token after the bump, so also after the first
call. A slot below `t` may hold the old pointer and is waited for.
3. Second call, after the wait. On weakly ordered CPUs the exit store can
become visible before the reader's last loads have completed. The second
call forces them to finish before the caller frees.
liburcu places its two calls the same way, at the start and end of
`synchronize_rcu()`. The difference here is a 64-bit token in place of
liburcu's parity bit and double wait. That substitution still needs review.
### membarrier
- Linux: `MEMBARRIER_CMD_PRIVATE_EXPEDITED` and its register command exist
since 4.14 ([man
page](https://man7.org/linux/man-pages/man2/membarrier.2.html)). DPDK
requires 5.4. The process registers once at init.
- FreeBSD: since 14.1 ([man
page](https://cgit-dev.freebsd.org/src/plain/lib/libsys/membarrier.2)).
- Windows: `FlushProcessWriteBuffers()` is the equivalent.
- The kernel delivers it as an IPI to each CPU running a thread of the
process. Every busy-polling lcore takes two interrupts per sync.
- It is per process. The primary and each secondary sync their own callback
lists.
- If the syscall is missing or filtered by seccomp, the domain flag is false
and readers do the full fence. That costs what bpf_pkt costs today.
## ethdev
The read section goes in the callback walker, not in each callback.
`rte_eth_rx_burst()` loads the list head inline and only then calls
`rte_eth_call_rx_callbacks()`. A section opened inside the callback starts
after the pointer was fetched. That is why bpf_pkt needs its recheck and a
context that is never freed.
The walker enters the section, reloads the head, walks, and exits. Tracepoints
omitted:
```c
uint16_t
rte_eth_call_rx_callbacks(uint16_t port_id, uint16_t queue_id,
struct rte_mbuf **rx_pkts, uint16_t nb_rx, uint16_t nb_pkts,
void *opaque __rte_unused)
{
const struct rte_eth_rxtx_callback *cb;
unsigned int slot = eth_cb_slot();
if (unlikely(slot == ETH_CB_NO_SLOT))
return nb_rx;
rcu_enter(ð_cb_rcu, slot);
cb = rte_atomic_load_explicit(
&rte_eth_fp_ops[port_id].rxq.clbk[queue_id],
rte_memory_order_relaxed);
while (cb != NULL) {
nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx,
nb_pkts, cb->param);
cb = cb->next;
}
rcu_exit(ð_cb_rcu, slot);
return nb_rx;
}
```
- The walker is out of line and reached only when a callback is installed. A
queue without callbacks pays nothing.
- The inline burst functions and `rte_eth_fp_ops` do not change.
- New experimental `rte_eth_callback_sync(void)`: returns once no thread is
still in a chain it entered before the call.
- It must not be called from inside a callback. A debug check on the nesting
depth catches that.
- `rte_eth_remove_rx_callback()` and `rte_eth_remove_tx_callback()` do not
change. The documented contract holds: remove, sync, then `rte_free()` the
handle.
- The domain is per process, like the callback lists.
Dependency: ethdev gains a dependency on rcu. rcu is always enabled, needs
only eal and ring, and is built before ethdev. bpf and pdump call the ethdev
function and need no direct rcu dependency. A test build with rcu added to
ethdev, bpf and pdump configured and linked.
## Thread model
Every thread that calls the burst functions needs a slot. Lcores and
registered threads use their lcore id, and the interrupt thread gets a
reserved one.
| Thread | Slot | Notes |
| --- | --- | --- |
| EAL lcore, service lcore | `rte_lcore_id()` | |
| Non-EAL thread after `rte_thread_register()` | `rte_lcore_id()` | |
| EAL interrupt thread (alarms, interrupt callbacks) | `RTE_MAX_LCORE` | Found with `rte_thread_is_intr()`, only on the `LCORE_ID_ANY` branch |
| Any other unregistered thread | None | Application bug. The walker skips the callback chain. |
- `MT_LOCKFREE` Tx works because each thread has its own slot, whatever queue
it uses.
- Bonding mode 4 with dedicated queues calls `rte_eth_rx_burst()` and
`rte_eth_tx_burst()` on member ports from `bond_mode_8023ad_periodic_cb()`,
an alarm callback. It is the only in-tree driver found calling the public
burst API off an lcore.
- For a slotless thread, skipping the chain means a Tx filter is not applied
and capture misses its packets. Running the chain unprotected would keep the
removal race.
Debug check: under `RTE_ETHDEV_DEBUG_RX` and `RTE_ETHDEV_DEBUG_TX` the burst
functions verify `rte_lcore_id() != LCORE_ID_ANY || rte_thread_is_intr()`. The
checks already in those blocks log and return 0.
Documentation: the EAL guide currently lists unregistered non-EAL threads as
supported with limitations. It needs a line saying ethdev Rx/Tx requires a
registered thread.
Possible separate bonding bug, not confirmed: `rx_burst_8023ad()` can call
`bond_mode_8023ad_handle_slow_pkt()` on an lcore, which for a marker frame
transmits on the dedicated Tx queue the alarm thread also uses.
## bpf_pkt changes
The use counter, its three helpers, the `cb != NULL` recheck and every barrier
in `bpf_pkt.c` go away. Unload keeps its current behavior: it returns with the
program destroyed.
Callbacks, all eight variants the same way:
```c
/* today */
bpf_eth_cbi_inuse(cbi); /* cbi->use++; rte_smp_mb(); */
rc = (cbi->cb != NULL) ?
pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1) :
nb_pkts;
bpf_eth_cbi_unuse(cbi); /* rte_smp_rmb(); cbi->use++; */
return rc;
/* proposed */
return pkt_filter_vm(cbi->bpf, pkt, nb_pkts, 1);
```
Unload:
```c
/* today */
rte_eth_remove_rx_callback(port, queue, bc->cb);
bc->cb = NULL;
rte_smp_mb();
bpf_eth_cbi_wait(bc); /* spin on bc->use */
rte_bpf_destroy(bc->bpf);
bpf_eth_cbi_cleanup(bc);
/* proposed */
cb = bc->cb;
rte_eth_remove_rx_callback(port, queue, cb);
rte_eth_callback_sync();
rte_free((void *)(uintptr_t)cb);
rte_bpf_destroy(bc->bpf);
bpf_eth_cbi_cleanup(bc);
bc->cb = NULL;
```
- `struct bpf_eth_cbi` loses `use`. The datapath then only reads the cbi;
today it writes that cache line twice per burst.
- The callback handle is freed. Today it is leaked on every unload.
- JIT code is unmapped by `rte_bpf_destroy()` only after sync, so no thread
can still be executing it.
- `MT_LOCKFREE` Tx is fixed with no bpf-specific code.
- Unchanged: the cbi list, `cbh->lock`, callback selection, the public API.
- `cbh->lock` is held across the sync, as it is held across the busy wait
today. The difference is that sync makes a syscall.
- Installing over an existing program still unloads then adds, so the window
with no filter stays as it is today.
- Patch 01/30 of the `rte_smp_mb` removal series converts the barriers in this
same code to C11 fences. This proposal deletes that code.
## pdump changes
`use_count` and its hold, release and wait helpers go away. Disable becomes
remove on every requested queue, one sync, then free.
- After the sync, `rte_free()` each callback handle and destroy the filter.
One enable request shares a single filter across its queues, so it is
destroyed once.
- The MP reply flow does not change. The reply is sent after
`set_pdump_rxtx_cbs()` returns, so after the sync. dumpcap frees the ring
and mempool only once readers are out.
- In the primary the request already runs from an alarm on the interrupt
thread. Sync there is fine: that thread's own slot is idle.
- One grace period covers the whole request. Today there is one wait per
queue.
Stats: `pdump_copy_burst()` has four lock-prefixed updates. `accepted` and
`ringfull` are per burst, `filtered` and `nombuf` per packet. They become
single-writer counters (relaxed load, add, store). That is exact with one
thread per queue. With `MT_LOCKFREE` Tx they can undercount unless made
per-lcore.
Left alone: `rte_ring_enqueue_burst()` on the multi-producer ring is one
`lock cmpxchg` per burst that captured packets.
## cryptodev findings
cryptodev already brackets its enqueue and dequeue callbacks with RCU, but
removal never waits and the read side costs a full barrier per burst. These
are from code reading only; nothing was reproduced.
How it works today:
- One `rte_rcu_qsbr` variable per queue pair and direction, created with
`max_threads = 1`.
- `rte_cryptodev_enqueue_burst()` and `rte_cryptodev_dequeue_burst()` call
`rte_rcu_qsbr_thread_online(qsbr, 0)`, reload the list head, walk the chain,
then call `rte_rcu_qsbr_thread_offline(qsbr, 0)`.
- Remove unlinks, calls
`rte_rcu_qsbr_synchronize(qsbr, RTE_QSBR_THRID_INVALID)`, then frees the
callback.
| # | Finding | Where | Effect |
| --- | --- | --- | --- |
| 1 | Reader is never registered | No `rte_rcu_qsbr_thread_register()` call for these variables in lib/cryptodev, drivers/crypto, app/test or examples | `rte_rcu_qsbr_check()` scans the registered-thread bitmap, finds none and returns at once. Remove frees the callback while a thread can still be inside it. |
| 2 | Full barrier per burst | `rte_rcu_qsbr_thread_online()` is a store plus a seq_cst fence | Same locked instruction per burst as bpf_pkt, on both enqueue and dequeue, whenever a callback is installed |
| 3 | One reader slot per queue pair | Thread id 0 for every caller | Same one-thread-per-queue assumption as the use counters |
| 4 | Global spinlock held across synchronize | `rte_cryptodev_callback_lock` in both remove functions | No effect today because the wait never happens. Once fixed, add and remove on every device stall behind one removal. |
| 5 | Reconfigure frees all callbacks | `rte_cryptodev_configure()` calls `cryptodev_cb_cleanup()` | Handles the application still holds dangle. A later remove reads `cb->fn` for its tracepoint before looking the handle up. |
| 6 | Close does not free | `rte_cryptodev_close()` never calls `cryptodev_cb_cleanup()` | Lists, QSBR variables and callbacks stay allocated until the next configure |
| 7 | Wrong pointer checked | `rte_cryptodev_remove_deq_callback()` tests `dev->enq_cbs == NULL` | Harmless, both arrays are allocated together |
One thing it gets right: the list head is reloaded after going online. That is
the placement the ethdev walker needs.
Fix options, both deferred until bpf and pdump are done:
- Minimal: register thread 0 on each variable in `cryptodev_cb_init()`. That
makes the synchronize real and keeps the barrier.
- Later: move cryptodev to the same reader domain as ethdev. That removes the
barrier and the per-queue-pair variables.
## Cost
The per-burst lock for callback protection goes away, and a stop pays two
syscalls instead. Nothing here was measured; the instruction lists come from
disassembly of the current code and from the sketches above.
| Path | Today | Proposed |
| --- | --- | --- |
| bpf callback, per burst | `lock addl`, two stores to the cbi line | Nothing in bpf_pkt |
| pdump callback, per burst | Two stores for the counter, up to four lock-prefixed stats updates, ring CAS if packets were captured | Plain adds for stats, ring CAS unchanged |
| ethdev walker, per burst, queue with a callback | List walk | Adds: lcore id read, depth inc and dec, one load, two stores to the thread's own line |
| Queue without callbacks | Nothing | Nothing |
| bpf unload, pdump disable | Spin on the counter | Two `membarrier()` calls, so two IPIs per busy lcore, with a spin between |
The spin is bounded by one burst unless a reader is preempted inside a
callback, as with `RTE_WAIT_UNTIL_MASKED` today.
## Patch series and tests
Four patches, each usable on its own once the one before it is in.
1. rcu: reader domain, `rcu_sync()`, membarrier detection and the fenced
fallback. Unit test covering nesting, concurrent sync and the fallback path
forced on.
2. ethdev: the domain, the walker change, `rte_eth_callback_sync()`, the debug
check. Doc updates for the remove-callback text and the EAL guide, plus a
release note.
3. bpf: convert `bpf_pkt.c` as shown above.
4. pdump: convert, free the handle and filter, make the stats single-writer.
Tests:
- net/null advertises `MT_LOCKFREE`. Several lcores transmit on one queue
while a control thread loops bpf load/unload and pdump enable/disable, built
with ASan to catch a use after free.
- Bonding mode 4 with dedicated queues, capture started and stopped on a
member port.
- Existing bpf and pdump autotests.
- A run on arm64.
- The same stress run against current main first, to confirm the bpf_pkt and
pdump problems before changing anything.
## Open questions
For the BPF side:
1. Is it acceptable to hold `cbh->lock`, a spinlock, across
`rte_eth_callback_sync()` now that it makes a syscall? The alternatives are
dropping the lock around the sync or making it a mutex.
2. Keep `bpf_eth_cbi` entries on the list forever as today, or free them on
unload now that it is safe?
3. Installing over an existing program: keep unload then add, or add a swap so
there is no window without a filter?
4. Does anyone run bpf Tx filters from unregistered threads? Under this
proposal such a thread bypasses the filter.
5. Patch 01/30 of the `rte_smp_mb` series touches the same code. Take it as
the interim fix, or drop it if this goes first?
General:
6. Review of the ordering argument, in particular the 64-bit token used in
place of liburcu's parity bit and double wait.
7. Names for the rcu API and for `rte_eth_callback_sync()`.
8. Debug check: assert, or log and return 0 like the checks already there.
9. Stable branches: a minimal pdump fix (recheck plus real fences) adds a lock
per burst. Backport that, the recheck alone, or nothing.
## What was checked
Everything here comes from reading code and one test build. No race was
reproduced and there is no prototype.
- Trees: DPDK main at d129414711aa (2026-09-24), liburcu master at
645fdc9b7206.
- By code reading only: the bpf_pkt `MT_LOCKFREE` problem, both pdump holes,
the leaks, the cryptodev findings, the bonding paths.
- Disassembly: x86-64, gcc 13.3, default build options. This is where the
lock-prefixed instruction lists come from.
- Build: rcu added as a dependency of ethdev, bpf and pdump. Meson configured
and the libraries linked.
- Not done: no measurements, no arm64 build, no run of `membarrier()` under
DPDK. Its behavior is taken from the man pages. IPI delivery is how the
Linux kernel implements it and is not stated there.
## References
- [liburcu
README](https://github.com/urcu/userspace-rcu/blob/master/README.md): the
flavors and their trade-offs
- [Linux
membarrier(2)](https://man7.org/linux/man-pages/man2/membarrier.2.html)
- [FreeBSD
membarrier(2)](https://cgit-dev.freebsd.org/src/plain/lib/libsys/membarrier.2)
- DPDK sources: `lib/bpf/bpf_pkt.c`, `lib/pdump/rte_pdump.c`,
`lib/ethdev/ethdev_private.c`, `lib/ethdev/rte_ethdev.h`,
`lib/rcu/rte_rcu_qsbr.h`, `lib/cryptodev/rte_cryptodev.c`,
`drivers/net/bonding/rte_eth_bond_8023ad.c`