Issues around packet capture when secondary process is doing rx/tx

[Ferruh Yigit]

On 1/8/2024 10:41 AM, Morten Brørup wrote:
>> From: Stephen Hemminger [mailto:stephen at networkplumber.org]
>> Sent: Monday, 8 January 2024 02.59
>>
>> I have been looking at a problem reported by Sandesh
>> where packet capture does not work if rx/tx burst is done in secondary
>> process.
>>
>> The root cause is that existing rx/tx callback model just doesn't work
>> unless the process doing the rx/tx burst calls is the same one that
>> registered the callbacks.
> 
> So, callbacks don't work across processes, because code might differ across processes.
> 
> If process A is running, and RX'ing and TX'ing, and process B wants to install its own callbacks (e.g. packet capture) on RX and RX, we basically want process A to execute code residing in process B, which is impossible.
> 

Callbacks stored in "struct rte_eth_dev", so it is per process, which
means primary and secondaries has their own copies of callbacks, as
Konstantin explained.

So, how pdump works :), it uses MP support and shared ring similar to
you mentioned below. More detail:
- Primary registers a MP handler
- pdump secondary process sends a MP message with a ring and mempool in
the message
- When primary receives the MP message it registers its *own* callbacks
that gets 'ring' as parameter
- Callbacks clone packets to 'ring', that is how pdump secondary process
access to the packets

> An alternative could be to pass the packets through a ring in shared memory. However, this method would add the ring processing latency of process B to the RX/TX latency of process A.
> 
> I think we can conclude that callbacks are one of the things that don't work with secondary processes.
> 
> With this decided, we can then consider how to best add packet capture. The concept of passing "data" (instead of calling functions) across processes obviously applies to this use case.
> 
>>
>> An example sequence would be:
>> 	1. dumpcap (or pdump) as secondary tells pdump in primary to
>> register callback
>> 	2. secondary process calls rx_burst.
>> 	3. rx_burst sees the callback but it has pointer pdump_rx which
>> is not necessarily
>> 	   at same location in primary and secondary process.
>> 	4. indirect function call in secondary to bad location likely
>> causes crash.
>>
>> Some possible workarounds.
>> 	1. Keep callback list per-process: messy, but won't crash.
>> Capture won't work
>>            without other changes. In this primary would register
>> callback, but secondaries
>>            would not use them in rx/tx burst.
>>
>> 	2. Replace use of rx/tx callback in pdump with change to
>> rte_ethdev to have
>>            a capture flag. (i.e. don't use indirection).  Likely ABI
>> problems.
>>            Basically, ignore the rx/tx callback mechanism. This is my
>> preferred
>> 	   solution.
>>
>> 	3. Some fix up mechanism (in EAL mp support?) to have each
>> process fixup
>>            its callback mechanism.
>>
>> 	4. Do something in pdump_init to register the callback in same
>> process context
>> 	   (probably need callbacks to be per-process). Would mean
>> callback is always
>>            on independent of capture being enabled.
>>
>>         5. Get rid of indirect function call pointer, and replace it by
>> index into
>>            a static table of callback functions. Every process would
>> have same code
>>            (in this case pdump_rx) but at different address.  Requires
>> all callbacks
>>            to be statically defined at build time.
>>
>> The existing rx/tx callback is not safe id rx/tx burst is called from
>> different process
>> than where callback is registered.
>>
>