[dpdk-dev] [RFC] eventdev: introduce event dispatcher

[Mattias Rönnblom]

On 2021-03-15 16:00, Van Haaren, Harry wrote:
>> -----Original Message-----
>> From: dev <dev-bounces at dpdk.org> On Behalf Of Mattias Rönnblom
>> Sent: Monday, March 15, 2021 2:45 PM
>> To: Jerin Jacob <jerinjacobk at gmail.com>
>> Cc: Jerin Jacob <jerinj at marvell.com>; dpdk-dev <dev at dpdk.org>; Richardson,
>> Bruce <bruce.richardson at intel.com>
>> Subject: Re: [dpdk-dev] [RFC] eventdev: introduce event dispatcher
>>
>> On 2021-03-07 14:04, Jerin Jacob wrote:
>>> On Fri, Feb 26, 2021 at 1:31 PM Mattias Rönnblom
>>> <mattias.ronnblom at ericsson.com> wrote:
>>>> On 2021-02-25 13:32, Jerin Jacob wrote:
>>>>> On Fri, Feb 19, 2021 at 12:00 AM Mattias Rönnblom
>>>>> <mattias.ronnblom at ericsson.com> wrote:
>>>>>> The purpose of the event dispatcher is primarily to decouple different
>>>>>> parts of an application (e.g., processing pipeline stages), which
>>>>>> share the same underlying event device.
>>>>>>
>>>>>> The event dispatcher replaces the conditional logic (often, a switch
>>>>>> statement) that typically follows an event device dequeue operation,
>>>>>> where events are dispatched to different parts of the application
>>>>>> based on the destination queue id.
>>>>> # If the device has all type queue[1] this RFC would restrict to
>>>>> use queue ONLY as stage. A stage can be a Queue Type also.
>>>>> How we can abstract this in this model?
>>>> "All queue type" is about scheduling policy. I would think that would be
>>>> independent of the "logical endpoint" of the event (i.e., the queue id).
>>>> I feel like I'm missing something here.
>>> Each queue type also can be represented as a stage.
>>> For example, If the system has only one queue, the Typical IPsec
>>> outbound stages can be
>>> Q0-Ordered(For SA lookup) -> Q0(Atomic)(For Sequence number update) ->
>>> Q0(Orderd)(Crypto operation)->Q0(Atomic)(Send on wire)
>>
>> OK, this makes sense.
>>
>>
>> Would such an application want to add a callback
>> per-queue-per-sched-type, or just per-sched-type? In your example, if
>> you would have a queue Q1 as well, would want to have the option to have
>> different callbacks for atomic-type events on Q0 and Q1?
>>
>>
>> Would you want to dispatch based on anything else in the event? You
>> could basically do it on any field (flow id, priority, etc.), but is
>> there some other field that's commonly used to denote a processing stage?
> I expect that struct rte_event::event_type and sub_event_type would regularly
> be used to split out different type of "things" that would be handled separately.
>
> Overall, I think we could imagine the Queue number, Queue Scheduling type (Re-Ordered, Atomic),
> Event type, sub event type, Flow-ID.. all contributing somehow to what function to execute in some situation.


Sure, and add to this list the contents of the mbuf (or other user payload).


What you should keep in mind (and maybe you did), is that the primary 
aim is to allow decoupling of different parts of an application (or even 
multiple applications), sharing an event device.


> As a somewhat extreme example to prove a point:
> An RX core might use rte_flow rules to split traffic into some arbitrary grouping, and
> then the rte_event::flow_id could be used to select the function-pointer to jump to handle it?
>
> I like the *concept* of having a table of func-ptrs, and removing of a switch() in that way,
> but I'm not sure that DPDK Eventdev APIs are the right place for it. I think Jerin already
> suggested the "helper function" concept, which seems a good idea to allow optional usage.
>
> To be clear, I'm not against upstreaming of such an event-dispatcher, but I'm not sure
> its possible to build it to be generic enough for all use-cases. Maybe focusing on an actual
> use-case and driving the design from that is a good approach?
>
The intention is that the event dispatcher is optional. For performance 
and/or flexibility, many applications would still use the normal event 
device enqueue and dequeue operations.


The event dispatcher not supposed to cover all possible use cases in the 
sense that it will able to remove all conditional logic used to select 
the function call which marks the beginning of the processing for an 
event. It should be an aim to cover most cases, where one set of events 
goes to one software module, and another goes elsewhere. (The original 
RFC design stems from an actual use case - but it's just one.)


An alternative design would be to turn the whole thing around, in the 
sense that instead of the application specifying which queue id/sched 
type/sub event type/etc goes to which callback, you split the callback 
in two: one callback to answer the question "is this your event" and 
another callback to actually dispatch the event (with the same signature 
as the callback in the RFC).


If made completely generic (and thus remove any references made to queue 
id in the API), it would require a handful (or maybe more) callback 
calls per event in the dispatcher. Sounds less than ideal from a 
performance point of view.


If you kept the queue id as the basic arbiter of the event stream, the 
overhead should be reduced to something more manageable in most 
applications.


A side-effect of this scheme is it provides an opportunity for the 
dispatcher to order the just-dequeued events in batches to the different 
consumer callbacks, before dispatching them - basically for free. For 
many applications, this should have a large upside in the form of 
improved cache locality and fewer branch mispredicts. To reap those 
benefits, batching on queue id should also be performed.


If there's interest, I could try to do a RFC of this alternative 
approach as well, and have a look at the performance implications.

> Regards, -Harry
>
> <snip patch contents below>