[dpdk-dev] [PATCH 0/2] new headroom stats library and example application

[Wodkowski, PawelX]

> -----Original Message-----
> From: Neil Horman [mailto:nhorman at tuxdriver.com]
> Sent: Thursday, January 29, 2015 8:13 PM
> To: Wodkowski, PawelX
> Cc: dev at dpdk.org
> Subject: Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and example
> application
> 
> On Thu, Jan 29, 2015 at 05:10:36PM +0000, Wodkowski, PawelX wrote:
> > > -----Original Message-----
> > > From: Neil Horman [mailto:nhorman at tuxdriver.com]
> > > Sent: Thursday, January 29, 2015 2:25 PM
> > > To: Wodkowski, PawelX
> > > Cc: dev at dpdk.org
> > > Subject: Re: [dpdk-dev] [PATCH 0/2] new headroom stats library and
> example
> > > application
> > >
> > > On Thu, Jan 29, 2015 at 12:50:04PM +0100, Pawel Wodkowski wrote:
> > > > Hi community,
> > > > I would like to introduce library for measuring load of some arbitrary jobs.
> It
> > > > can be used to profile every kind of job sets on any arbitrary execution unit.
> > > > In provided l2fwd-headroom example I demonstrate how to use this library
> to
> > > > profile packet forwarding (job set is froward, flush and stats) on LCores
> > > > (execution unit). This example does no limit possible schemes on which this
> > > > library can be used.
> > > >
> > > > Pawel Wodkowski (2):
> > > >   librte_headroom: New library for checking core/system/app load
> > > >   examples: introduce new l2fwd-headroom example
> > > >
> > > >  config/common_bsdapp               |    6 +
> > > >  config/common_linuxapp             |    6 +
> > > >  examples/Makefile                  |    1 +
> > > >  examples/l2fwd-headroom/Makefile   |   51 +++
> > > >  examples/l2fwd-headroom/main.c     |  875
> > > ++++++++++++++++++++++++++++++++++++
> > > >  lib/Makefile                       |    1 +
> > > >  lib/librte_headroom/Makefile       |   50 +++
> > > >  lib/librte_headroom/rte_headroom.c |  368 +++++++++++++++
> > > >  lib/librte_headroom/rte_headroom.h |  481 ++++++++++++++++++++
> > > >  mk/rte.app.mk                      |    4 +
> > > >  10 files changed, 1843 insertions(+)
> > > >  create mode 100644 examples/l2fwd-headroom/Makefile
> > > >  create mode 100644 examples/l2fwd-headroom/main.c
> > > >  create mode 100644 lib/librte_headroom/Makefile
> > > >  create mode 100644 lib/librte_headroom/rte_headroom.c
> > > >  create mode 100644 lib/librte_headroom/rte_headroom.h
> > > >
> > > > --
> > > > 1.7.9.5
> > > >
> > > >
> > >
> > > Whats the advantage of this library over the other tools to preform the same
> > > function.
> >
> > Hi Neil,
> >
> > Good point, what is advantage over perf. Answer is: this library does not
> > supposed to be a perf competition and is not for profiling app in the way perf
> does.
> > It is an small and fast extension. It's main task is to manage job list to invoke
> > them exactly when needed and provide some basic stats about application idle
> > time (whatever programmer will consider the idle) and busy time.
> >
> > For example:
> > application might decide to add remove some jobs to/from LCore(s)
> dynamically
> > basing on current idle time (ex: move job from one core to another).
> >
> > Also application might have some information's about traffic type it handles
> > and provide own algorithm to calculate invocation time (it can also
> dynamically
> > switch between those algorithms only replacing handlers).
> >
> > >  Perf can provide all the information in this library, and do so
> > > without having to directly modify the source for the execution unit under
> test
> >
> > Yes, perf can provide those information's  but it can't handle the case when
> > you are poling for packets too fast or too slow and waist time getting only
> couple
> > of them. Library will adjust time when it execute job basing on value this job
> > returned previously. Code modifications are not so deep, as you can see
> comparing
> > l2wf vs l2fwd-headroom app.
> >
> > For example in  application I introduced, when forward job return less than
> > MAX_PKT_BURST execution period will be increased. If it return more it will
> decrease
> > execution period. Stats provided for that can be used to determine if
> application is
> > behaving correctly and if there is a time for handling another port (what did for
> tests).
> >
> You're still re-inventing the wheel here, and I don't see any advantage to doing
> so.  If the goal of the library is to profile the run time of a task, then you
> have perf and systemtap for such purposes.  If the goal is to create a job
> scheduler that allows you to track multiple parallel tasks, and adjust their
> execution, there are several pre-existing libraries that any application
> programmer can already leverage to do just that (Berkely UPC or libtask to name
> just two examples).  Truthfully, on a dedicated cpu, you could just as easily
> create multiple child processes runnnig at SCHED_RR and set their priorities
> accordingly.
> 
> I don't see why we need another library to do what several other tools/libraries
> can do quite well.
> 

I am under impression that I am unable to express myself clearly enough.

I did not meant to make competition to perf nor reinvent the tasking library.

1. Idle and runtime statistics are provided "by the way" and you can use them
or use perf on linux or libpmc on freebsd or whatever tool you like to get more
sophisticated ones.
2. You can also use tasking library what you like with a headroom object on top
of every task you created with any scheduling you want. This way you can assign
priorities to job sets and adjust sleep time as you like. You can also decide what
is your idle/load time by placing task-dependet-yeld/sleep in idle callback or separate
job callback or loop end callback.

Both of those two points above are no covering one gap - we are running in poll
mode and here is a lack of mutex, semaphores or other synchronization mechanisms
provided by OS/tasking layer. This library try to provide facility for estimating optimal
job execution time because every mis-poll is a waste of time and can lead to
impression that core/task is fully loaded but what it does is only polling and getting 1
or 2 packets instead of 32 (it degrades total NIC throughput).

You can use perf (or whatever tool you like) to profile your code for performance. If
the code good enough, you use headroom library that will estimate how often or
when your perfect-profiled-and-optimized code need to be executed. When execution
period estimation settles you have your headroom for other jobs on this core/task.
You can also provide your own algorithm for estimating execution period. You are also
able to drop using this headroom library at runtime and switch to tasking one when
 you decide that you have enough data make decision how many jobs you can execute
on lcore/task basing on current throughput.

I deliberately did not used 'task' but 'job' to not make an impression that I am 
reinventing another task library. This code is to be simple and fast. Of course you can
do all those things in every application you make but this library is provided to not
reinvent this logic all the time.

Pawel