[dpdk-dev] [PATCH v4 2/2] Add l2reflect measurement application

[Henning Schild]

Am Wed, 02 Sep 2020 10:30:22 +0200
schrieb Thomas Monjalon <thomas at monjalon.net>:

> 02/09/2020 09:56, Henning Schild:
> > Am Tue, 01 Sep 2020 17:07:38 +0200
> > schrieb Thomas Monjalon <thomas at monjalon.net>:
> >   
> > > 28/08/2020 16:22, Henning Schild:  
> > > > Thomas,
> > > > 
> > > > what is the state on this one? Any more steps to take or people
> > > > to involve?    
> > > 
> > > I can try adding some key people in Cc list,
> > > while doing a quick review.
> > >
> > >  
> > > > I first showed that in action back in 2016 on FOSDEM.
> > > > https://archive.fosdem.org/2016/schedule/event/virt_iaas_real_time_cloud/
> > > > After my talk two devs from dpdk approached me because they
> > > > liked the idea of a "network cyclictest". It took us some time
> > > > to finally go upstream with it, unfortunately i do not recall
> > > > names.    
> > > 
> > > I think I was one of them.
> > > 
> > > There will be some talks about latency in the next virtual DPDK
> > > event:
> > > https://events.linuxfoundation.org/dpdk-userspace-summit/program/schedule/
> > > (speakers are Cc'ed)  
> > 
> > Thanks, that sure looks like we are not the only ones who are
> > interested in latency and probably a way to measure it. From other
> > doing it as well it would be nice to hear how they currently do
> > that. 
> > > > This application can potentially be integrated into the
> > > > test-suite for QA, making sure no changes heavily increase the
> > > > package transmission worst case timing.    
> > > 
> > > Good
> > > 
> > >   
> > > > Felix Moessbauer <felix.moessbauer at siemens.com> wrote:
> > > >     
> > > > > The l2reflect application implements a ping-pong benchmark to
> > > > > measure the latency between two instances. For communication,
> > > > > we use raw ethernet and send one packet at a time. The timing
> > > > > data is collected locally and min/max/avg values are
> > > > > displayed in a TUI. Finally, a histogram of the latencies is
> > > > > printed which can be further processed with the
> > > > > jitterdebugger visualization scripts.    
> > > 
> > > Please can you show an example of script usage?  
> 
> [...]
> > > > > + * Copyright(c) 2020 Siemens AG
> > > > > + *
> > > > > + * authors:
> > > > > + *   Felix Moessbauer <felix.moessbauer at siemens.com>
> > > > > + *   Henning Schild <henning.schild at siemens.com>    
> > > 
> > > It is uncommon to mention authors in the file.
> > > In general, git metadata covers this info.
> > > Any special requirement?  
> > 
> > That application has some history. Written by me and eventually
> > improved for upstream by Felix. The idea here was to give credit to
> > two people with just one author field in git. If that is really in
> > the way, we can drop it and make Felix the sole author.  
> 
> You are both marked as authors thanks to the sign-off:
> 	Signed-off-by: Felix Moessbauer <felix.moessbauer at siemens.com>
> 	Signed-off-by: Henning Schild <henning.schild at siemens.com>
> 
> [...]
> > > > > +__rte_format_printf(1, 0)
> > > > > +static void
> > > > > +trace_write(const char *fmt, ...)
> > > > > +{
> > > > > +	va_list ap;
> > > > > +	char buf[256];
> > > > > +	int n, err;
> > > > > +
> > > > > +	if (trace_fd == 0)
> > > > > +		trace_fd =
> > > > > open("/sys/kernel/debug/tracing/trace_marker",
> > > > > +				O_WRONLY);    
> > > 
> > > Why not using rte_trace framework?  
> > 
> > I am not sure where rte_trace ends up eventually. But the idea here
> > is to have kernel traces when looking at latency peaks. You might
> > just be looking at an incorrect host tuning when you see a spike
> > i.e. higher prio threads on your possibly isolated core.
> > It is not tracing dpdk, but more how it fits in the overall
> > picture.  
> 
> How Linux trace helps more than DPDK one? Because of scheduler
> context? Anyway this is Linux-specific.

The kernel trace gives you a system view. So you might see competing
threads with higher prio on your cpu, all kernel activity on your
behalf or on someone elses ... other device interrupts coming in on
your core. All information related to the system, not really the
application itself. You would be using this to make sure your setup is
correct. Once you are sure the spike must be coming from inside your
application you would go and trace that.
So i think there might be value for rte_trace once tracing the
application, here we care about the overall system. But i also do not
know rte_trace, its scope and features ... just guessing it is app+lib
only.

Henning

> [...]
> > > If I understand well, this requires a special cabling?
> > > Would it be possible to measure latency of hardware NIC internal
> > > loopback or software PMD loopback?  
> > 
> > Not sure. We always want to see the full picture ... i.e would also
> > want to see if a NIC is slow to put a packet on the "wire".
> > For testing we do use multi-port NICs with cable-loops. The other
> > setup is Qemus with the application inside and a dpdk-based vhost
> > switch. But that virtual setup is also not trivial. In fact we
> > usually do make that switch send the packets over the cable-loop to
> > again cover the whole way the packets go "in reality".  
> 
> That's why I wonder whether it should be located in DTS.
> 
> [...]
> > > > > +cjson = dependency('libcjson', required: false)    
> > > 
> > > Some other parts require jansson:
> > > 	jansson = dependency('jansson', required: false)
> > > How libcjson is different? Would it be possible to unify
> > > dependency? 
> > > > > +if not is_linux
> > > > > +        build = false    
> > > 
> > > Why limiting to Linux?  
> > 
> > Probably because of the RT stuff (sched param setting, mlock) and
> > the tracing. I would propose it like that and hope that others
> > might ifdef when enabling other OSs.  
> 
> Yes that's reasonnable to support only Linux in the first version.
> 
> > Maintaining low latency requires quite a bit of host tuning. This
> > can get especially tricky when consuming the whole CPU with
> > polling, you can starve your OS. The application alone does not do
> > that for you, not even on Linux. Like cyclictest or jitterdebugger
> > i would see it as a tool that requires knowledge to measure
> > correctly. And it should probably be added as such, with an
> > integration into automated tests and support for other OSs possibly
> > coming in the future.  
> 
> Such integration can be done in DTS.
> 
> > I hope that others doing dpdk for low latency on preempt-rt will
> > find it useful as is. They might have been looking for something
> > like it, possibly have something like it in their test-labs and did
> > not publish it yet.  
> 
> Thanks for publishing.
> 
>