[RFC v2] non-temporal memcpy

[Mattias Rönnblom]

On 2022-07-29 22:26, Morten Brørup wrote:
> +TO: @Honnappa, we need input from ARM
> 
>> From: Konstantin Ananyev [mailto:konstantin.ananyev at huawei.com]
>> Sent: Friday, 29 July 2022 21.49
>>>
>>>> From: Konstantin Ananyev [mailto:konstantin.ananyev at huawei.com]
>>>> Sent: Friday, 29 July 2022 14.14
>>>>
>>>>
>>>> Sorry, missed that part.
>>>>
>>>>>
>>>>>> Another question - who will do 'sfence' after the copying?
>>>>>> Would it be inside memcpy_nt (seems quite costly), or would
>>>>>> it be another API function for that: memcpy_nt_flush() or so?
>>>>>
>>>>> Outside. Only the developer knows when it is required, so it
>> wouldn't
>>>> make any sense to add the cost inside memcpy_nt().
>>>>>
>>>>> I don't think we should add a flush function; it would just be
>>>> another name for an already existing function. Referring to the
>>>> required
>>>>> operation in the memcpy_nt() function documentation should
>> suffice.
>>>>>
>>>>
>>>> Ok, but again wouldn't it be arch specific?
>>>> AFAIK for x86 it needs to boil down to sfence, for other
>> architectures
>>>> - I don't know.
>>>> If you think there already is some generic one (rte_wmb?) that
>> would
>>>> always produce
>>>> correct instructions - sure let's use it.
>>>>
>>>
>>> DPDK has generic functions to wrap architecture specific stuff like
>> memory barriers.
>>>
>>> Because they are non-temporal stores, I suspect that rte_mb() is
>> required before reading the data from the location it was copied to.
>>> Ensuring that STORE operations are ordered (rte_wmb) might not
>> suffice. However, I'm not a CPU expert, so I will seek advice from
>>> more qualified people in the community on this.
>>
>> I think for IA sfence is enough, see citation below,
>> for other architectures - no idea.
>> What I am trying to say - it needs to be the *same* function on all
>> archs we support.
> 
> Now I get it: rte_wmb() might be appropriate on x86, but if any other architecture requires something else, we should add a new common function for flushing, e.g. rte_memcpy_nt_flush().
> 

rte_wmb() not being enough also my understanding. NT stores are weakly 
ordered on x86, and requires a sfence to be ordered with non-NT stores. 
Unfortunately, this per-memcpy sfence instruction make even 1500-byte 
sized copy operations much slower - at least in micro benchmarks.

>>
>> IA SW optimization manual:
>> 9.4.2 Streaming Store Usage Models
>> The two primary usage domains for streaming store are coherent requests
>> and non-coherent requests.
>> 9.4.2.1 Coherent Requests
>> Coherent requests are normal loads and stores to system memory, which
>> may also hit cache lines
>> present in another processor in a multiprocessor environment. With
>> coherent requests, a streaming store
>> can be used in the same way as a regular store that has been mapped
>> with a WC memory type (PAT or
>> MTRR). An SFENCE instruction must be used within a producer-consumer
>> usage model in order to ensure
>> coherency and visibility of data between processors.
>> Within a single-processor system, the CPU can also re-read the same
>> memory location and be assured of
>> coherence (that is, a single, consistent view of this memory location).
>> The same is true for a multiprocessor
>> (MP) system, assuming an accepted MP software producer-consumer
>> synchronization policy is
>> employed.
>>
> 
> With this reference, I am convinced that you are right about the SFENCE. This puts a checkmark on this item on my TODO list for the patch. Thank you, Konstantin!
> 
> Any ARM CPU experts on the mailing list seeing this, not on vacation? @Honnappa, I'm looking at you. :-)
> 
> Summing up, the question is:
> 
> After a bunch of *non-temporal* stores (STNP instruction) on ARM architecture, does calling rte_wmb() suffice to ensure the data is visible across the system?
>