[PATCH v1] event/dlb2: add dequeue interrupt mode support

[Mattias Rönnblom]

On 2025-07-05 06:11, Pathak, Pravin wrote:
> 
> 
>> -----Original Message-----
>> From: Mattias Rönnblom <hofors at lysator.liu.se>
>> Sent: Thursday, July 3, 2025 5:25 AM
>> To: Pathak, Pravin <pravin.pathak at intel.com>; Jerin Jacob
>> <jerinjacobk at gmail.com>
>> Cc: dev at dpdk.org; jerinj at marvell.com; Chen, Mike Ximing
>> <mike.ximing.chen at intel.com>; Richardson, Bruce
>> <bruce.richardson at intel.com>; thomas at monjalon.net; Marchand, David
>> <david.marchand at redhat.com>; nipun.gupta at amd.com;
>> chenbox at nvidia.com; Sarkar, Tirthendu <tirthendu.sarkar at intel.com>; Pavan
>> Nikhilesh <pbhagavatula at marvell.com>; Shijith Thotton
>> <sthotton at marvell.com>; Hemant Agrawal <hemant.agrawal at nxp.com>;
>> Sachin Saxena <sachin.saxena at oss.nxp.com>; harry.chang at intel.com;
>> Mattias Rönnblom <mattias.ronnblom at ericsson.com>
>> Subject: Re: [PATCH v1] event/dlb2: add dequeue interrupt mode support
>>
>> On 2025-07-01 23:08, Pathak, Pravin wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Mattias Rönnblom <hofors at lysator.liu.se>
>>>> Sent: Tuesday, July 1, 2025 4:16 AM
>>>> To: Pathak, Pravin <pravin.pathak at intel.com>; Jerin Jacob
>>>> <jerinjacobk at gmail.com>
>>>> Cc: dev at dpdk.org; jerinj at marvell.com; Chen, Mike Ximing
>>>> <mike.ximing.chen at intel.com>; Richardson, Bruce
>>>> <bruce.richardson at intel.com>; thomas at monjalon.net; Marchand, David
>>>> <david.marchand at redhat.com>; nipun.gupta at amd.com;
>> chenbox at nvidia.com;
>>>> Sarkar, Tirthendu <tirthendu.sarkar at intel.com>; Pavan Nikhilesh
>>>> <pbhagavatula at marvell.com>; Shijith Thotton <sthotton at marvell.com>;
>>>> Hemant Agrawal <hemant.agrawal at nxp.com>; Sachin Saxena
>>>> <sachin.saxena at oss.nxp.com>; harry.chang at intel.com; Mattias Rönnblom
>>>> <mattias.ronnblom at ericsson.com>
>>>> Subject: Re: [PATCH v1] event/dlb2: add dequeue interrupt mode
>>>> support
>>>>
>>>> On 2025-06-30 19:34, Pathak, Pravin wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Mattias Rönnblom <hofors at lysator.liu.se>
>>>>>> Sent: Monday, June 30, 2025 12:51 PM
>>>>>> To: Pathak, Pravin <pravin.pathak at intel.com>; Jerin Jacob
>>>>>> <jerinjacobk at gmail.com>
>>>>>> Cc: dev at dpdk.org; jerinj at marvell.com; Chen, Mike Ximing
>>>>>> <mike.ximing.chen at intel.com>; Richardson, Bruce
>>>>>> <bruce.richardson at intel.com>; thomas at monjalon.net; Marchand,
>> David
>>>>>> <david.marchand at redhat.com>; nipun.gupta at amd.com;
>>>> chenbox at nvidia.com;
>>>>>> Sarkar, Tirthendu <tirthendu.sarkar at intel.com>; Pavan Nikhilesh
>>>>>> <pbhagavatula at marvell.com>; Shijith Thotton <sthotton at marvell.com>;
>>>>>> Hemant Agrawal <hemant.agrawal at nxp.com>; Sachin Saxena
>>>>>> <sachin.saxena at oss.nxp.com>; harry.chang at intel.com; Mattias
>>>>>> Rönnblom <mattias.ronnblom at ericsson.com>
>>>>>> Subject: Re: [PATCH v1] event/dlb2: add dequeue interrupt mode
>>>>>> support
>>>>>>
>>>>>> On 2025-06-30 18:18, Pathak, Pravin wrote:
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Jerin Jacob <jerinjacobk at gmail.com>
>>>>>>>> Sent: Monday, June 30, 2025 7:44 AM
>>>>>>>> To: Mattias Rönnblom <hofors at lysator.liu.se>
>>>>>>>> Cc: Pathak, Pravin <pravin.pathak at intel.com>; dev at dpdk.org;
>>>>>>>> jerinj at marvell.com; Chen, Mike Ximing
>>>>>>>> <mike.ximing.chen at intel.com>; Richardson, Bruce
>>>>>>>> <bruce.richardson at intel.com>; thomas at monjalon.net; Marchand,
>>>>>>>> David
>>>> <david.marchand at redhat.com>;
>>>>>> nipun.gupta at amd.com;
>>>>>>>> chenbox at nvidia.com; Sarkar, Tirthendu
>>>>>>>> <tirthendu.sarkar at intel.com>; Pavan Nikhilesh
>>>>>>>> <pbhagavatula at marvell.com>; Shijith Thotton
>>>>>>>> <sthotton at marvell.com>; Hemant Agrawal
>>>> <hemant.agrawal at nxp.com>;
>>>>>>>> Sachin Saxena <sachin.saxena at oss.nxp.com>; harry.chang at intel.com;
>>>>>>>> Mattias Rönnblom <mattias.ronnblom at ericsson.com>
>>>>>>>> Subject: Re: [PATCH v1] event/dlb2: add dequeue interrupt mode
>>>>>>>> support
>>>>>>>>
>>>>>>>> On Mon, Jun 30, 2025 at 4:47 PM Mattias Rönnblom
>>>>>>>> <hofors at lysator.liu.se>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 2025-06-30 11:19, Jerin Jacob wrote:
>>>>>>>>>> On Sat, Jun 28, 2025 at 11:17 AM Pravin Pathak
>>>>>>>> <pravin.pathak at intel.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> DLB2 port interrupt is implemented using DPDK interrupt
>> framework.
>>>>>>>>>>> This allows eventdev dequeue API to sleep when the port queue
>>>>>>>>>>> is empty and gets wakeup when event arrives at the port. Port
>>>>>>>>>>> dequeue mode is configured using devargs argument
>>>> port_dequeue_wait.
>>>>>>>>>>> Supported modes are polling and interrupt. Default mode is polling.
>>>>>>>>>>> This commit also adds code to handle device error interrupts
>>>>>>>>>>> and print alarm details.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Pravin Pathak <pravin.pathak at intel.com>
>>>>>>>>>>> Signed-off-by: Tirthendu Sarkar <tirthendu.sarkar at intel.com>
>>>>>>>>>>> ---
>>>>>>>>>>>       doc/guides/eventdevs/dlb2.rst              |  20 +
>>>>>>>>>>>       drivers/event/dlb2/dlb2.c                  | 236 +++++-
>>>>>>>>>>>       drivers/event/dlb2/dlb2_iface.c            |   7 +
>>>>>>>>>>>       drivers/event/dlb2/dlb2_iface.h            |   8 +
>>>>>>>>>>>       drivers/event/dlb2/dlb2_priv.h             |  18 +
>>>>>>>>>>>       drivers/event/dlb2/dlb2_user.h             | 112 +++
>>>>>>>>>>>       drivers/event/dlb2/pf/base/dlb2_hw_types.h |  70 ++
>>>>>>>>>>>       drivers/event/dlb2/pf/base/dlb2_osdep.h    |  46 ++
>>>>>>>>>>>       drivers/event/dlb2/pf/base/dlb2_regs.h     | 149 +++-
>>>>>>>>>>>       drivers/event/dlb2/pf/base/dlb2_resource.c | 825
>>>>>>>> +++++++++++++++++++++
>>>>>>>>>>>       drivers/event/dlb2/pf/base/dlb2_resource.h |   6 +
>>>>>>>>>>>       drivers/event/dlb2/pf/dlb2_pf.c            | 223 ++++++
>>>>>>>>>>>       12 files changed, 1711 insertions(+), 9 deletions(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/doc/guides/eventdevs/dlb2.rst
>>>>>>>>>>> b/doc/guides/eventdevs/dlb2.rst index 8ec7168f20..a4ba857351
>>>>>>>> 100644
>>>>>>>>>>> --- a/doc/guides/eventdevs/dlb2.rst
>>>>>>>>>>> +++ b/doc/guides/eventdevs/dlb2.rst
>>>>>>>>>>> @@ -477,6 +477,26 @@ Example command to use as meson
>> option
>>>>>> for
>>>>>>>> credit handling:
>>>>>>>>>>>
>>>>>>>>>>>              meson configure
>>>>>>>>>>> -Dc_args='-DDLB_SW_CREDITS_CHECKS=0 -
>>>>>>>> DDLB_HW_CREDITS_CHECKS=1'
>>>>>>>>>>>
>>>>>>>>>>> +Interrupt Mode Support
>>>>>>>>>>> +~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>> +DLB dequeue supports interrupt mode for the API
>>>>>>>> rte_event_dequeue_burst().
>>>>>>>>>>> +The default port dequeue mode is polling. Dequeue wait mode
>>>>>>>>>>> +can be configured on per eventdev port basis using devargs
>>>>>>>>>>> +argument 'port_dequeue_wait'. In interrupt mode, if the port
>>>>>>>>>>> +queue is empty, the application thread will block on the
>>>>>>>>>>> +interrupt until a new event arrives. It enters blocking mode
>>>>>>>>>>> +only after any specified timeout. During the timeout, it will
>>>>>>>>>>> +poll the port queue for
>>>>>>>> events as usual. Interrupt mode uses the DPDK interrupt support
>>>>>> framework.
>>>>>>>>>>> +
>>>>>>>>>>> +    .. code-block:: console
>>>>>>>>>>> +
>>>>>>>>>>> +       --allow ea:00.0,port_dequeue_wait=all:interrupt
>>>>>>>>>>
>>>>>>>>>> Adding other eventdev PMD mainatainers.
>>>>>>>>>>
>>>>>>>>>> Looks like it can be a generic feature. i.e set this option is
>>>>>>>>>> dev_configure() If there is no objection, Please send a new
>>>>>>>>>> patch around
>>>>>>>> that.
>>>>>>>>>
>>>>>>>>> I've considered implementing this in DSW, although in a
>>>>>>>>> different manner (with eventfds and poll()).
>>>>>>>>>
>>>>>>>>> The dequeue timeout will still be honored in "interrupt mode",
>> correct?
>>>>>>>>> It wasn't obvious from the description.
>>>>>>>>
>>>>>>>> How is it in Intel PMD?
>>>>>>>
>>>>>>> It would be best if we configure it per port using
>>>>>>> RTE_EVENT_PORT_CFG_*
>>>>>> flags. Will it be, OK?
>>>>>>> The dequeue timeout will be honored, and the decision to block or
>>>>>>> return
>>>>>> will be made after the timeout.
>>>>>>
>>>>>> That doesn't sound like the timeout is honored.
>>>>>>
>>>>>> The reason an application wants the dequeue call to complete within
>>>>>> a certain time, even though there wasn't any events, is likely
>>>>>> because it want to go do something else with that thread, after the
>> timeout.
>>>>>>
>>>>>> Thus, you can't decide to block the thread *after* the timeout. If
>>>>>> you block, you have to do some time *before* the timeout, and wake
>>>>>> up in time to meet the deadline.
>>>>>>
>>>>>> For example, if the event device is given a 1 ms dequeue timeout by
>>>>>> the application, it could go busy-poll for 10 us, then busy-poll
>>>>>> with a short tpause for 100 us, and then put the thread to sleep
>>>>>> blocking on some fd for the remaining 890 us.
>>>>>>
>>>>>> The hardware-specific nature of the timing and exact mechanism to
>>>>>> use speaks against having this kind of configuration in the Eventdev API.
>>>>>
>>>>> The mode is not for what to do during a timeout, but after a timeout.
>>>>
>>>> My comments were not so much concerning any DLB-specific extension,
>>>> but rather how this kind of function should work, if it was a part of
>>>> the standard API.
>>>>
>>>>> We can enter sleep mode immediately by setting the timeout to 0 if
>>>>> we need
>>>> to.
>>>>> This mode is not changing the current timeout behavior.  After the
>>>>> timeout, it allows HW devices supporting interrupts to block on
>> an interrupt.
>>>>
>>>> The current API specifies that control is returned to the
>>>> application, after the timeout has expired. If you change that with a
>>>> PMD parameter, the DLB behavior will be in violation of the API
>>>> contract. Applications using
>>>> rte_keepalive_alive() between dequeues is one example of those that
>>>> will break. In fact, all applications that use more than one RTE service will
>> break.
>>>>
>>>>> It will wake up only after a new event arrives at the port. If the
>>>>> application needs control back then it can use the current default
>>>>> non-blocking mode. A better mode to save power during timeout will
>>>>> be to use umwait-based sleep.  Consider this as extension to current
>>>>> timeout
>>>> behavior if device supports interrupts.
>>>>>
>>>>
>>>> What about applications that both want power efficiency *and* require
>>>> a timeout?
>>>>
>>>> It makes no sense to me the change the semantics of the dequeue
>>>> timeout parameter from "the time until I want control back" to "the
>>>> time I want the event device to use polling, after which it should put the
>> thread to sleep".
>>>> Those two are pretty much orthogonal.
>>>>
>>>> The current API doesn't specify what happens during the timeout. If
>>>> you by "non-blocking" mean "busy-polling", that is not an API
>>>> requirement. I don't see why the event device couldn't put the lcore
>>>> thread to sleep during a long timeout (and I also see why you may not
>>>> want that to be the default behavior).
>>>
>>> I got your point. Application will prefer to return and not block
>>> after the timeout as per current API description. Current API
>>> signature is = 0 no-wait, returns immediately if there is no event.
>>>> 0 wait for the event or timeout
>>> Will it be OK to extend it to treat max timeout 0xffffffff as block
>>> for event with PMD specific Mechanism to wait for the event ? Port specific
>> configuration will not be required in this case.
>>> Devices supporting interrupts then can use interrupt blocking mode
>> internally.
>>>
>>
>> Correct me if I'm wrong here, but I think an event device is free to block using
>> a PMD-specific mechanism with any non-zero timeout. The longer the
>> timeout, the more sense it make to put the thread to sleep (blocking on an fd).
>>
>> I also think that the details of how to spend the timeout time probably needs
>> to be configured by the app (at startup). Different modes, thresholds, etc.
>> Maybe such are best off as PMD parameters at this point.
>>
>> Is there some limitation in the interrupt APIs, or what is preventing the DLB
>> driver to both honor a finite timeout and block/put the lcore thread to sleep?
> 
> Current API definition does not specify or put restriction on timeout mechanism PMD decides
> to  implement.  But it will be nice to have application some control on it. DLB PMD supports
> polling, umwait and interrupt blocking as three modes. Current configuration is devargs based
> and specific to DLB PMD. If application has choice at each dequeue call, it can decide what
> mechanism to use. Polling for lower latencies, umwait for power saving and interrupt blocking
> to make a de-scheduling call and release CPU for other thread. We had implementation supporting these
> using most significant bits of timeout value on per dequeue call basis. But as it breaks API, now it is done
> with devargs.  If control over dequeue wait is useful to applications, then it can be standardized instead of
> being PMD specific. Else, we will keep it the way it is now in the patch.
> 

I agree it makes sense to allow the application (or the user deploying 
the app) control over power management / latency parameters.

It's not obvious those need (or should) be specified on a per-dequeue 
call basis. Wouldn't it suffice to make it a per-device or per-port (~= 
per-lcore) configuration?

Maybe this kind of configuration is better off starting as PMD devargs, 
since it will vary both with hardware/ISA and event device 
implementation. When we have a couple of different implementation, we 
can try to generalize a common API from them.

> I took look at ethdev PMD and there additional APIs are used for blocking.
> rte_eth_dev_rx_intr_enable
> epoll_wait()
> rte_eth_dev_rx_intr_disable
> And other APIs to configure interrupts.  But Eventdev mechanism of doing all inside dequeue API looks
> Simple.
> 
>>
>>>>
>>>>>>
>>>>>>> If not interrupt, it can be called blocking vs polling mode.  If
>>>>>>> the port config is
>>>>>> fine, I will create a new patch with it.
>>>>>>> Also, we should have this as a capability for eventdevs.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> What's being configured should just be a threshold time at which
>>>>>>>>> the event device would go from busy-polling to blocking the thread.
>>>>>>>>>
>>>>>>>>> Maybe it should be called something with "blocking" or
>>>>>>>>> "sleeping", instead of "interrupt", since interrupts are never directly
>> involved.
>>>>>>>>
>>>>>>>> Agree. or make it a power save mode or so.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Anyway, seems like a good candidate for a generic feature to me.
>>>>>>>>>
>>>>>
>>>
>