[RFC v2] non-temporal memcpy

[Konstantin Ananyev]

> 
> > From: Konstantin Ananyev [mailto:konstantin.ananyev at huawei.com]
> > Sent: Friday, 29 July 2022 14.14
> >
> >
> > Sorry, missed that part.
> >
> > >
> > > > Another question - who will do 'sfence' after the copying?
> > > > Would it be inside memcpy_nt (seems quite costly), or would
> > > > it be another API function for that: memcpy_nt_flush() or so?
> > >
> > > Outside. Only the developer knows when it is required, so it wouldn't
> > make any sense to add the cost inside memcpy_nt().
> > >
> > > I don't think we should add a flush function; it would just be
> > another name for an already existing function. Referring to the
> > required
> > > operation in the memcpy_nt() function documentation should suffice.
> > >
> >
> > Ok, but again wouldn't it be arch specific?
> > AFAIK for x86 it needs to boil down to sfence, for other architectures
> > - I don't know.
> > If you think there already is some generic one (rte_wmb?) that would
> > always produce
> > correct instructions - sure let's use it.
> >
> 
> DPDK has generic functions to wrap architecture specific stuff like memory barriers.
> 
> Because they are non-temporal stores, I suspect that rte_mb() is required before reading the data from the location it was copied to.
> Ensuring that STORE operations are ordered (rte_wmb) might not suffice. However, I'm not a CPU expert, so I will seek advice from
> more qualified people in the community on this.

I think for IA sfence is enough, see citation below,
for other architectures - no idea.
What I am trying to say - it needs to be the *same* function on all archs we support. 

IA SW optimization manual:
9.4.2 Streaming Store Usage Models
The two primary usage domains for streaming store are coherent requests and non-coherent requests.
9.4.2.1 Coherent Requests
Coherent requests are normal loads and stores to system memory, which may also hit cache lines
present in another processor in a multiprocessor environment. With coherent requests, a streaming store
can be used in the same way as a regular store that has been mapped with a WC memory type (PAT or
MTRR). An SFENCE instruction must be used within a producer-consumer usage model in order to ensure
coherency and visibility of data between processors.
Within a single-processor system, the CPU can also re-read the same memory location and be assured of
coherence (that is, a single, consistent view of this memory location). The same is true for a multiprocessor
(MP) system, assuming an accepted MP software producer-consumer synchronization policy is
employed.