[dpdk-users] TimeStamping Packets Generated and Received via Pktgen Application

Paul Emmerich emmericp at net.in.tum.de
Mon Dec 19 19:16:32 CET 2016


Hi,

you can synchronize the NICs clock to the system clock using rte_eth_timesync_adjust_time and/or rte_eth_timesync_write_time
There is currently no wrapper function for these two calls in MoonGen as we only use it for latency measurements and typically only sync the NICs with each other.

But that should be easy to add, the more challenging part is keeping the clocks in sync/re-syncing due to drift.
Clock drift between different NICs or the system clock is typically ~10-35 ppm in my experience 

(There is probably some functionality in DPDK that does this, but I haven't used that part of DPDK yet)


Paul

> Am 19.12.2016 um 18:51 schrieb Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
> 
> Hi Paul, 
> 
> I was able to read the timestamp value of the packets which are transmitted using MoonGen. 
> 
> I tried looking how I can synchronize the timestamp(read from the NIC) of the transmitted packet  with the packets received on the controller side ? 
> 
> I read about packet time referencing, epoch time however it doesn't seem to do what I want.  
> 
>  Is there a way at all which will synchronize the timestamp of those packets ??
> 
> On Wed, Dec 14, 2016 at 7:00 PM, Ajinkya D Kadam <ajinkya.kadam at nyu.edu> wrote:
> Thanks a lot for your time and inputs. 
> 
> I have added my comments inline.
> 
> On Wed, Dec 14, 2016 at 4:05 PM, Paul Emmerich <emmericp at net.in.tum.de> wrote:
> Hi,
> 
>> Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
>> I want packets to be generated using different IP address and each packet should be timestamped at two places. (Please refer the topology attached) Once when it is sent from the server (to switch port 1) i.e t1 and next time when it reaches the controller i.e t2. I am trying to measure the time taken by switch to generate packet in message. Also the reason why I want to work with different IP address is the switch doesn't currently understand mac addresses. So I can program the flows in the switch only using IP address information.
>> 
>> I believe that I can generate packets from different IP addresses (the above script doesn't have that functionality yet)
> 
> example for varying IP addresses: https://github.com/emmericp/MoonGen/blob/master/examples/l3-load-latency.lua#L92-L96
> 
> This is what I was looking for. Thanks so much. 
>> but what my concern is, right now the timestamped logic I have written in line 48-50  returns a "nil" as timestamped value when I try to print it. The NIC that is being used here is Intel X710. Can you please suggest what might be possibly wrong here ?
> 
> You have to set the timestamp flag on the buffer that you want to timestamp via  buf:enableTimestamps(). See my previous email for example code.
> Also, you only need to call queue:enableTimestamps() once before the main loop.
> 
> 
> Ahh..Got the mistake. Thanks for the correction.  
>> Next, another solution is to use ptp packets which are timestamped and then encapsulate them with packets differing in source IP. However the PTP packets timestamped at the server side
>> have nanosecond resolution like "26155" (which is nice, also check wireshark output attached) however the packets i am capturing at controller using libpcap are timestamped according to the unix timestamp which counts the number of seconds passed from january 1st 1970. Is there a way in which I can receive these packets with the same resolution as the one PTP packets are timestamped ? If not is there any other way which I should look for? (The controller NIC is X520 Intel NIC)
> 
> I'd advise against measuring latencies by taking timestamps on different devices, that's very complicated to get right (and typically involves a GPS receiver for time synchronization).
> (However, some fancy stuff with PTP might work fine, depending on your requirements on accuracy and precision).
> Also, using libpcap will lower your accuracy and precision by several microseconds if not backed by hardware timestamping (libpcap actually supports this for NICs that expose this in the driver, the challenge will just be to sync the clocks of the NICs, but I'm not an expert on this).
> For the granularity in the pcap format: You can try using the pcapng format which supports storing timestamps with nanosecond granularity.
> 
> If I were running such a test, I would use a different test setup:
> * only one server attached to the switch
>  
> Agreed. I am sorry the topology that I have shown actually has only one server. So the controller, Moongen all are running on the same machine. I drew it that way to make things more clearer. Sorry about that. The purpose of keeping the same machine is to keep the synchronization intact as you have said. As the machine is the same can I now just use libpcap feature to read timestamps ? I read the X710 datasheet. It seems that "PRTTSYN_TXTIME" is used to store the transmit timestamp and "PRTTSYN_RXTIME" for received timestamps.  Do I need to read these registers using libpcap ? 
>  
> * receive the controller traffic via an interface bound to DPDK, do whatever you need to do there, then bridge it to the host (e.g., KNI or a separate cable)
> 
> I believe I can bind the DPDK port to OVS bridge and then run the controller there. I will try doing this tonight. 
>  
>  
>  
> This is more complicated, but avoids a lot of pitfalls and provides a nice centralized control over the whole setup from your test application.
> You can use a NIC that is capable of timestamping arbitrary RX packets for the "bridge" port (e.g., 82580).
> 
> 
> I have a 82571 NIC. I will look into it if this can support RX packet timestamping. 
>  
> Paul
> 
>> 
>> 
>> On Tue, Dec 6, 2016 at 10:55 AM, Paul Emmerich <emmericp at net.in.tum.de> wrote:
>> Hi,
>> 
>> most NICs don't support timestamping TCP packets.
>> It works for TX on some NICs, but RX is more difficult: of the Intel NICs, only some of the igb (1 Gbit/s) family and the X550 support this.
>> 
>> For RX:
>> I've implemented it for the 82580 igb NIC, but I'm not sure if it still works since the driver refactoring in MoonGen.
>> The X550 10 Gbit/s NIC would need some driver magic, but even the NIC's datasheet is inconsistent about the registers here.
>> 
>> For TX (which seems to be your use case):
>> It might work depending on your HW, you can test it:
>>  
>> 1. call buf:enableTimestamps() on the buf you are interested in
>> 2. send the packet
>> 3. get the timestamp with queue:getTimestamp(maxWaitMicros)
>> 
>> Note that the timestamp is kept in a register on the NIC. It stores only one TX timestamp at a time, irregardless of the number of queues etc.
>> You have to read this register via queue:getTimestamp() before another packet can be timestamped.
>> 
>> Our default measureLatency() function might be helpful:
>> https://github.com/libmoon/libmoon/blob/master/lua/timestamping.lua#L64
>> 
>> 
>> Paul
>> 
>>> Am 06.12.2016 um 16:40 schrieb Ajinkya D Kadam <ajinkya.kadam at nyu.edu>:
>>> 
>>> Hi Paul, 
>>> 
>>> If I am not wrong this [1] script enables only timestamps for PTP or UDP packets. Is this similar functionality available for TCP packets ? 
>>> 
>>> I am generating multiple TCP flows and I just want to time-stamp first packet of each flow. Is this possible using the NICs hardware time-stamping capability ?
>>> 
>>> 
>>> [1] : https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>> 
>>> On Tue, Oct 25, 2016 at 6:57 AM, Paul Emmerich <emmericp at net.in.tum.de> wrote:
>>> Hi,
>>> 
>>> the examples in "timestamping-tests.lua" are only meant as a demonstration of the different timestamping capabilities (and/or as a starting point for a custom script).
>>> 
>>> In your case, you could use a device counter to print the whole throughput of the device. You can use the default stats task to do that by adding the following call in the master task:
>>> 
>>>         stats.startStatsTask({rxDev, txDev})
>>> 
>>> I'll also add the call to the example script in the repository later today as having this is probably a good idea :)
>>> 
>>> Paul
>>> 
>>> > Am 22.10.2016 um 12:19 schrieb Huynhtu Dang <huynh.tu.dang at usi.ch>:
>>> >
>>> > Hello Emmerich,
>>> >
>>> > MoonGen is really helpful in measuring performance of network devices.
>>> > I wonder if we could get some information about packet loss
>>> > while running timestamps-software.lua?
>>> >
>>> > Thanks,
>>> > Tu
>>> >
>>> > On Oct 17, 2016, at 12:41 PM, Paul Emmerich <emmericp at net.in.tum.de<mailto:emmericp at net.in.tum.de>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Ajinkya D Kadam:
>>> > I was reading through your paper and I think this tool will be much more
>>> > helpful to me. Btw I am using quad X710 and dual X520 NICs.
>>> > Is this [1] the right code to look at if i want to see how you have
>>> > achieved hardware based time stamping ?
>>> >
>>> > Yes, run this example script with two directly connected ports for a simple demo and test of your hardware's capabilities. It will work with both of your NICs.
>>> >
>>> > In addition, I want to confirm my understanding of why MoonGen is better
>>> > than PktGen in time stamping context.
>>> > PktGen reads the value of rdtsc which it then appends to packet, this
>>> > adds more delay and hence the precision is bad.
>>> >
>>> > Software timestamping by writing the TSC to the packet is also supported (but the API is less nice, see issue #153):
>>> >
>>> > See examples/timestamping-tests/timestamps-software.lua for an example.
>>> >
>>> > The main problem is that there is unpredictable jitter from the NIC and PCIe transfer and other random errors. Especially if you are running this at higher packet rates.
>>> > This leads to the 200-300ns random error that I've previously mentioned.
>>> >
>>> >
>>> > In case of MoonGen how does this work ? I am not sure. Could you please
>>> > elaborate ?
>>> >
>>> > MoonGen enables the hardware timestamping feature of the NIC and uses it. The NIC will store the timestamp in a register which needs to be read before another packet can be timestamped, this limits the throughput of timestamped packets. However, I've found that you rarely need to timestamp *all* packets in a packet generator. You'll have to use software timestamping if you really need that.
>>> >
>>> >
>>> > Paul
>>> >
>>> >
>>> > Thanks,
>>> > Ajinkya
>>> >
>>> >
>>> > [1] https://github.com/libmoon/libmoon/blob/b5f6c2cac42c02db64073b57dd8ca82692d3858c/examples/hardware-timestamping.lua
>>> >
>>> > ᐧ
>>> >
>>> > On Sun, Oct 16, 2016 at 6:55 PM, Paul Emmerich <emmericp at net.in.tum.de<mailto:emmericp at net.in.tum.de>
>>> > <mailto:emmericp at net.in.tum.de>> wrote:
>>> >
>>> >   Hi,
>>> >
>>> >
>>> >   Ajinkya D Kadam:
>>> >
>>> >       If yes I would like to modify the pktgen code so that each
>>> >       transmitting and
>>> >       received packet is timestamped.  Right now I am exploring the
>>> >       example
>>> >       applications  like rxtx_callbacks which timestamps packets in
>>> >       DPDK, Is this
>>> >       the right direction to go ?
>>> >
>>> >
>>> >   Check out my packet generator MoonGen
>>> >   https://github.com/emmericp/MoonGen
>>> >   <https://github.com/emmericp/MoonGen>
>>> >
>>> >   It uses the hardware timestamping features (PTP) to do latency
>>> >   measurements in the nanosecond-range.
>>> >
>>> >   However, if you will run into hardware limitations if you want to
>>> >   timestamp *all* packets. This is sometimes supported on RX (e.g.,
>>> >   i310, X550) but I don't know a NIC that supports this on TX.
>>> >
>>> >   As for the precision that is achievable: ~10ns (depending on the
>>> >   NIC) with hardware support. Software timestamping will typically
>>> >   result in a standard deviation of 200-300ns under load and there
>>> >   will be huge outliers.
>>> >
>>> >
>>> >    Paul
>>> 
>>> 
>> 
>> Chair of Network Architectures and Services
>> Department of Informatics
>> TU München
>> Boltzmannstr. 3
>> 85748 Garching bei München, Germany 
>> 
>> 
>> 
>> 
>> <topology.jpg><PTPTimestamp.png><libpcapTimestamping.png>
> 
> Chair of Network Architectures and Services
> Department of Informatics
> TU München
> Boltzmannstr. 3
> 85748 Garching bei München, Germany 
> 
> 
> 
> 
> 



More information about the users mailing list