[dpdk-dev] Recent changes related to interrupt thread

Stephen Hemminger stephen at networkplumber.org
Mon Nov 16 18:40:22 CET 2015


I was thinking of something like:

rte_intr_affinity(portid, queueid, lcoreid)

And per-lcore interrupt threads.

On Mon, Nov 16, 2015 at 9:19 AM, Ananyev, Konstantin <
konstantin.ananyev at intel.com> wrote:

>
>
> > -----Original Message-----
> > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Stephen Hemminger
> > Sent: Monday, November 16, 2015 5:07 PM
> > To: Thomas Monjalon
> > Cc: dev at dpdk.org; Nirranjan Kirubaharan; Felix Marti; Kumar Sanghvi
> > Subject: Re: [dpdk-dev] Recent changes related to interrupt thread
> >
> > On Mon, 16 Nov 2015 14:48:42 +0100
> > Thomas Monjalon <thomas.monjalon at 6wind.com> wrote:
> >
> > > Hi,
> > >
> > > 2015-11-16 18:02, Rahul Lakkireddy:
> > > > Hi,
> > > >
> > > > I notice that the following changeset:
> > > >
> > > > Fixes: fd6949c55c9a ("eal: fix io permission for virtio interrupt
> > > > handler")
> > > >
> > > > has moved the initialization of the interrupt thread to after the
> master
> > > > lcore has been initialized.  However, this causes the interrupt
> thread
> > > > to _inherit_ the affinity of the master lcore. Hence, this seems to
> > > > make all interrupts to be handled by _only_ the master lcore. Because
> > > > of this change, it seems that now alarm interrupts would also be
> handled
> > > > by master lcore only, IIUC.
> > > >
> > > > We are seeing a performance regression for cxgbe PMD after this
> commit
> > > > since, cxgbe PMD relies on alarm to periodically transmit pending
> > > > coalesced packets.
> > > >
> > > > Also, this perf degradation is only seen if there's a queue allocated
> > > > on the master lcore, such as in l3fwd app.  If the master lcore has
> > > > been skipped, then no degradation in perf is seen since only the
> alarm
> > > > will run on the master lcore.
> > > >
> > > > So, is the change done to make all interrupts, including alarm
> > > > interrupts, be handled by _only_ the master lcore intended?
> > >
> > > No it was not intended. The idea was to inherit settings (iopl) from
> > > the device initialization into the interrupt thread.
> > > Though a DPDK driver is not really supposed to rely on interrupt
> performance.
> > > So having interrupts managed on any core was more or less a side
> effect.
> > >
> > > > BTW, I have tried setting the affinity to all cpus instead in
> > > > eal_intr_init() and this seems to restore the perf back. Perhaps it's
> > > > better to move the master lcore initialization to after the interrupt
> > > > thread has been initialized as well? Thoughts?
> > >
> > > Yes, i think it's possible.
> > > We can also imagine a command line option to set the interrupt affinity
> > > with a default which mimics the old behaviour.
> > >
> > > In order to make this conversation clearer, and for later references,
> > > below is the DPDK init call tree:
> > >
> >
> > With the new interrupt mode, the interrupt thread needs some rework
> anyway.
> > Ideally, there would be multiple interrupt threads, one per core;
> > then use SMP affinity to align the MSI-x interrupt for the device queue
> > to run on the core that is processing that queue.
> >
> > This would require new API's to do SMP affinity, wrapper around /proc/irq
> > and an API to tell DPDK which lcore is being to process a RX (and TX)
> > queue.
>
> There is no one to one mapping between lcore and device queue.
> Any lcore can do RX/TX on the device queue.
> Of course it is preferable to do it from the core on the same socket, but
> not required.
> You can even have multiple threads  RX/TX from/to the same queue -
> as long as you provide some sync mechanism between them.
> Konstantin
>
> >
> >
>
>


More information about the dev mailing list