[RFC v2] non-temporal memcpy

[Mattias Rönnblom]

On 2022-08-09 11:34, Morten Brørup wrote:
>> From: Mattias Rönnblom [mailto:hofors at lysator.liu.se]
>> Sent: Sunday, 7 August 2022 22.20
>>
>> On 2022-07-29 22:26, Morten Brørup wrote:
>>> +TO: @Honnappa, we need input from ARM
>>>
>>>> From: Konstantin Ananyev [mailto:konstantin.ananyev at huawei.com]
>>>> Sent: Friday, 29 July 2022 21.49
>>>>>
>>>>>> From: Konstantin Ananyev [mailto:konstantin.ananyev at huawei.com]
>>>>>> Sent: Friday, 29 July 2022 14.14
>>>>>>
>>>>>>
>>>>>> Sorry, missed that part.
>>>>>>
>>>>>>>
>>>>>>>> Another question - who will do 'sfence' after the copying?
>>>>>>>> Would it be inside memcpy_nt (seems quite costly), or would
>>>>>>>> it be another API function for that: memcpy_nt_flush() or so?
>>>>>>>
>>>>>>> Outside. Only the developer knows when it is required, so it
>>>> wouldn't
>>>>>> make any sense to add the cost inside memcpy_nt().
>>>>>>>
>>>>>>> I don't think we should add a flush function; it would just be
>>>>>> another name for an already existing function. Referring to the
>>>>>> required
>>>>>>> operation in the memcpy_nt() function documentation should
>>>> suffice.
>>>>>>>
>>>>>>
>>>>>> Ok, but again wouldn't it be arch specific?
>>>>>> AFAIK for x86 it needs to boil down to sfence, for other
>>>> architectures
>>>>>> - I don't know.
>>>>>> If you think there already is some generic one (rte_wmb?) that
>>>> would
>>>>>> always produce
>>>>>> correct instructions - sure let's use it.
>>>>>>
>>>>>
>>>>> DPDK has generic functions to wrap architecture specific stuff like
>>>> memory barriers.
>>>>>
>>>>> Because they are non-temporal stores, I suspect that rte_mb() is
>>>> required before reading the data from the location it was copied to.
>>>>> Ensuring that STORE operations are ordered (rte_wmb) might not
>>>> suffice. However, I'm not a CPU expert, so I will seek advice from
>>>>> more qualified people in the community on this.
>>>>
>>>> I think for IA sfence is enough, see citation below,
>>>> for other architectures - no idea.
>>>> What I am trying to say - it needs to be the *same* function on all
>>>> archs we support.
>>>
>>> Now I get it: rte_wmb() might be appropriate on x86, but if any other
>> architecture requires something else, we should add a new common
>> function for flushing, e.g. rte_memcpy_nt_flush().
>>>
>>
>> rte_wmb() not being enough also my understanding. NT stores are weakly
>> ordered on x86, and requires a sfence to be ordered with non-NT stores.
>>
>> Unfortunately, this per-memcpy sfence instruction make even 1500-byte
>> sized copy operations much slower - at least in micro benchmarks.
> 
> I agree that calling rte_mb() from each NT memcpy would be counterproductive on small/medium copy operations.
> 

There's no need for a full barrier after the copy operation. sfence 
(e.g., rte_wmb()) is enough. I guess you will want a rte_rmb() before 
you start copying as well, if you are using NT loads.

> Which is why rte_mb() is not called by the NT memcpy function itself, but by the application. This requirement will be part of the function's documentation.
> 

rte_memcpy_surprise()

:)

> The application must call rte_mb() before it accesses the copy.
> 
> Alternatively, the application can call rte_mb() after a burst of NT memcopies.
> 
>>
>>>>
>>>> IA SW optimization manual:
>>>> 9.4.2 Streaming Store Usage Models
>>>> The two primary usage domains for streaming store are coherent
>> requests
>>>> and non-coherent requests.
>>>> 9.4.2.1 Coherent Requests
>>>> Coherent requests are normal loads and stores to system memory,
>> which
>>>> may also hit cache lines
>>>> present in another processor in a multiprocessor environment. With
>>>> coherent requests, a streaming store
>>>> can be used in the same way as a regular store that has been mapped
>>>> with a WC memory type (PAT or
>>>> MTRR). An SFENCE instruction must be used within a producer-consumer
>>>> usage model in order to ensure
>>>> coherency and visibility of data between processors.
>>>> Within a single-processor system, the CPU can also re-read the same
>>>> memory location and be assured of
>>>> coherence (that is, a single, consistent view of this memory
>> location).
>>>> The same is true for a multiprocessor
>>>> (MP) system, assuming an accepted MP software producer-consumer
>>>> synchronization policy is
>>>> employed.
>>>>
>>>
>>> With this reference, I am convinced that you are right about the
>> SFENCE. This puts a checkmark on this item on my TODO list for the
>> patch. Thank you, Konstantin!
>>>
>>> Any ARM CPU experts on the mailing list seeing this, not on vacation?
>> @Honnappa, I'm looking at you. :-)
>>>
>>> Summing up, the question is:
>>>
>>> After a bunch of *non-temporal* stores (STNP instruction) on ARM
>> architecture, does calling rte_wmb() suffice to ensure the data is
>> visible across the system?
>>>
>