[PATCH 2/2] service: fix potential stats race-condition on MT services

[Mattias Rönnblom]

On 2022-07-08 18:21, Bruce Richardson wrote:
> On Fri, Jul 08, 2022 at 03:31:01PM +0000, Van Haaren, Harry wrote:
>>> -----Original Message-----
>>> From: Honnappa Nagarahalli <Honnappa.Nagarahalli at arm.com>
>>> Sent: Friday, July 8, 2022 4:16 PM
>>> To: Van Haaren, Harry <harry.van.haaren at intel.com>; dev at dpdk.org
>>> Cc: mattias.ronnblom <mattias.ronnblom at ericsson.com>; Morten Brørup
>>> <mb at smartsharesystems.com>; nd <nd at arm.com>; nd <nd at arm.com>
>>> Subject: RE: [PATCH 2/2] service: fix potential stats race-condition on MT services
>>
>> <snip previous discussions>
>>
>>>> diff --git a/lib/eal/common/rte_service.c b/lib/eal/common/rte_service.c
>>>> index ef31b1f63c..f045e74ef3 100644
>>>> --- a/lib/eal/common/rte_service.c
>>>> +++ b/lib/eal/common/rte_service.c
>>>> @@ -363,9 +363,15 @@ service_runner_do_callback(struct
>>>> rte_service_spec_impl *s,
>>>>   		uint64_t start = rte_rdtsc();
>>>>   		s->spec.callback(userdata);
>>>>   		uint64_t end = rte_rdtsc();
>>>> -		s->cycles_spent += end - start;
>>>> +		uint64_t cycles = end - start;
>>>>   		cs->calls_per_service[service_idx]++;
>>>> -		s->calls++;
>>>> +		if (service_mt_safe(s)) {
>>>> +			__atomic_fetch_add(&s->cycles_spent, cycles,
>>>> __ATOMIC_RELAXED);
>>>> +			__atomic_fetch_add(&s->calls, 1,
>>>> __ATOMIC_RELAXED);
>>>> +		} else {
>>>> +			s->cycles_spent += cycles;
>>>> +			s->calls++;
>>> This is still a problem from a reader perspective. It is possible that the writes could be
>>> split while a reader is reading the stats. These need to be atomic adds.
>>
>> Thanks for pointing out; I do "think" in x86 in terms of load/store tearing; and on x86
>> naturally aligned load/stores will not tear. Apologies for missing the ARM angle here.
>>
>> I'm not sure how to best encode the difference between tearing & "locked instructions"
>> to make things multi-writer safe. But they're not the same thing, and I'd prefer not pay
>> the penalty for LOCK instructions (multi-writer) only to satisfy the non-tearing requirements.
>>
>> Is there an rte_atomic-* type that is guaranteed as non-tearing?
>>
>> In that case, changing the type of the calls/cycles_spent variables to such a type to ensure "non-tearing"
>> single-reader, single-writer behaviour is enough, instead of forcing __atomic_fetch_add() everywhere?
> 
> Regular read, increment and then atomic store should work without locks
> where alignment is correct on most architectures, and doing the store
> atomically should prevent any tearing.

This is a good pattern, and provides a noticeable performance benefit 
compared to using an atomic add (which is an overkill in single-writer 
scenarios), in my experience. The DPDK seqcount implementation 
increments the count in this manner.