[PATCH v3 2/2] eventdev: add support for enqueue reorder

[Mattias Rönnblom]

On 2024-06-22 00:24, Abdullah Sevincer wrote:
> This commit adds support flag to enable enqueue reorder
> feature.
> 
> When this flag is enabled in the port configuration PMD
> restores dequeue order on enqueue if applications happen to
> change it.
> 
> Signed-off-by: Abdullah Sevincer <abdullah.sevincer at intel.com>
> ---
>   lib/eventdev/rte_eventdev.h | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/lib/eventdev/rte_eventdev.h b/lib/eventdev/rte_eventdev.h
> index 08e5f9320b..f4220dd5dc 100644
> --- a/lib/eventdev/rte_eventdev.h
> +++ b/lib/eventdev/rte_eventdev.h
> @@ -1073,6 +1073,14 @@ rte_event_queue_attr_set(uint8_t dev_id, uint8_t queue_id, uint32_t attr_id,
>    *  @see rte_event_port_setup()
>    */
>   
> +#define RTE_EVENT_PORT_CFG_RESTORE_DEQ_ORDER   (1ULL << 5)
> +/**< Flag to enable feature enqueue reordering to dequeue.
> + * The feature restores dequeue order on enqueue if applications
> + * happen to change the order.
> + *

Add a device-level
RTE_EVENT_DEV_CAP_INDEPENDENT_ENQ
as well.

The documentation of that flag should probably house the detailed 
description of this feature.

Here's how I would describe this feature:

#define RTE_EVENT_PORT_CFG_INDEPENDENT_ENQ   (1ULL << 5)

/**< Flag to enable independent enqueue. Must be unset if the device
  * is not RTE_EVENT_DEV_CAP_INDEPENDENT_ENQ capable. This feature
  * allows an application to enqueue RTE_EVENT_OP_FORWARD or
  * RTE_EVENT_OP_RELEASE in an order different than the order the
  * events were dequeued from the event device, while maintaining
  * RTE_SCHED_TYPE_ATOMIC or RTE_SCHED_TYPE_ORDERED semantics.
  *
  * If the application wish to change the order of two events *within
  * a flow*, it must both change the enqueue order and exchange the
  * impl_opaque field, to be portable across all event devices.
  */

That second paragraph allows DSW to support this feature without 
modification, since this is the only difference between DSW-style 
independent enqueue, and DLB enqueue reordering. DLB will restore a 
total order, while DSW doesn't (since it would be both pointless and 
costly, given its design).

The upside with DSW-style implementation is that it's very simple and 
efficient, and does not impose any head-of-line blocking (which follows 
from restoring a total order between dequeue and enqueue). The downside 
is it does not allow for a scenario where a particular flow is split 
across different modules, the application performs reordering (e.g., 
with the dispatcher library) *and* wants to maintain ordering between 
events pertaining to those "sub flows". I've never come across such a 
scenario, but it may well exist.

If we fail to make DLB2 and DSW compatible, we'll probably need another 
flag for DSW, because needlessly imposing a total order DSW does not 
make a lot of sense.

You may want to add an example as well. And a note on the importance of 
maintaining impl_opaque between dequeue and enqueue.

> + *  @see rte_event_port_setup()
> + */
> +
>   /** Event port configuration structure */
>   struct rte_event_port_conf {
>   	int32_t new_event_threshold;