[dpdk-dev] [PATCH v13 09/10] eal: add EAL argument for setting thread priority

[Narcisa Ana Maria Vasile]

On Thu, Aug 19, 2021 at 10:06:06AM +0100, Bruce Richardson wrote:
> On Wed, Aug 18, 2021 at 02:28:33PM -0700, Stephen Hemminger wrote:
> > On Tue,  3 Aug 2021 12:01:30 -0700 Narcisa Ana Maria Vasile
> > <navasile at linux.microsoft.com> wrote:
> > 
> > > +static int +eal_parse_thread_priority(const char *arg) +{ +
> > > struct internal_config *internal_conf = +
> > > eal_get_internal_configuration(); +	enum rte_thread_priority priority;
> > > + +	if (!strncmp("normal", arg, sizeof("normal"))) +
> > > priority = RTE_THREAD_PRIORITY_NORMAL; +	else if
> > > (!strncmp("realtime", arg, sizeof("realtime"))) +		priority =
> > > RTE_THREAD_PRIORITY_REALTIME_CRITICAL; +	else +		return -1;
> > > + +	internal_conf->thread_priority = priority; +	return 0; +} +
> > 
> > In my experience using real time priority is dangerous and risks
> > starvation and deadlock. The problem is that DPDK applications are
> > typically 100% CPU poll mode with no system calls; but the kernel has a
> > number of worker threads that can be required on those CPUs.
> > 
> > The typical failure is a disk completion interrupt happens on a CPU that
> > is being used by DPDK lcore thread. With RT priority, the kernel thread
> > to process that I/O completion never runs because the RT user thread has
> > higher priority than the kernel I/O completion thread.
> > 
> > It maybe possible to workaround this with lots of hand crafting through
> > sysfs to reassign threads and irq's. Also, later kernels with full RT
> > might also help.
> > 
> > Before putting this in as part of DPDK in EAL, a full set of testing and
> > documentation of how to configure these kind of applications and systems
> > is needed.
> >
> I would tend to agree caution here, based on my experience of having locked
> up a number of systems in the past when testing running DPDK apps with RT
> priority!

Thank you for the comments! I've added this option since it was requested by
multiple users. I understand RT priority causes issues on Linux platforms.
On Windows we want to be able to use REALTIME priority in certain scenarios.

Would it be acceptable to replace this option with a "HIGH_PRIORITY" one
and keep it realtime on Windows and choose a higher (but non-realtime) option on Linux?
However, there are 2 issues here:
 * We will have different behaviors between the 2 platforms.
 * Not sure if I can set a normal but higher priority on Linux. SCHED_OTHER only allows
   one value and Linux "nice" values don't help. If anyone knows of a way to accomplish
   this on Linux, please do advise.
Alternatively, we can have this option for Windows only.

In the meantime, I've removed this patch from this patchset in v14 as the cmdline option is not
being enabled yet, as DmitryK noted.