[dpdk-dev] [PATCH v2 1/3] eal/arm64: relax the io barrier for aarch64

[Jerin Jacob]

On Fri, Dec 20, 2019 at 9:49 AM Gavin Hu <Gavin.Hu at arm.com> wrote:
>
> Hi Jerin,
>
> Thanks for review, inline comments,
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk at gmail.com>
> > Sent: Friday, December 20, 2019 11:38 AM
> > To: Gavin Hu <Gavin.Hu at arm.com>
> > Cc: dpdk-dev <dev at dpdk.org>; nd <nd at arm.com>; David Marchand
> > <david.marchand at redhat.com>; thomas at monjalon.net;
> > rasland at mellanox.com; maxime.coquelin at redhat.com;
> > tiwei.bie at intel.com; hemant.agrawal at nxp.com; jerinj at marvell.com;
> > Pavan Nikhilesh <pbhagavatula at marvell.com>; Honnappa Nagarahalli
> > <Honnappa.Nagarahalli at arm.com>; Ruifeng Wang
> > <Ruifeng.Wang at arm.com>; Phil Yang <Phil.Yang at arm.com>; Joyce Kong
> > <Joyce.Kong at arm.com>; Steve Capper <Steve.Capper at arm.com>
> > Subject: Re: [dpdk-dev] [PATCH v2 1/3] eal/arm64: relax the io barrier for
> > aarch64
> >
> > On Fri, Dec 20, 2019 at 9:03 AM Jerin Jacob <jerinjacobk at gmail.com>
> > wrote:
> > >
> > > On Fri, Dec 20, 2019 at 8:40 AM Gavin Hu <gavin.hu at arm.com> wrote:
> > > >
> > > > Armv8's peripheral coherence order is a total order on all reads and
> > writes
> > > > to that peripheral.[1]
> > > >
> > > > The peripheral coherence order for a memory-mapped peripheral
> > signifies the
> > > > order in which accesses arrive at the endpoint.  For a read or a write
> > RW1
> > > > and a read or a write RW2 to the same peripheral, then RW1 will appear
> > in
> > > > the peripheral coherence order for the peripheral before RW2 if either
> > of
> > > > the following cases apply:
> > > >  1. RW1 and RW2 are accesses using Non-cacheable or Device attributes
> > and
> > > >     RW1 is Ordered-before RW2.
> > > >  2. RW1 and RW2 are accesses using Device-nGnRE or Device-nGnRnE
> > attributes
> > > >     and RW1 appears in program order before RW2.
> > >
> > >
> > > This is true if RW1 and RW2 addresses are device memory. i.e the
> > > registers in the  PCI bar address.
> > > If RW1 is DDR address which is been used by the controller(say NIC
> > > ring descriptor) then there will be an issue.
> > > For example Intel i40e driver, the admin queue update in Host DDR
> > > memory and it updates the doorbell.
> > > In such a case, this patch will create an issue. Correct? Have you
> > > checked this patch with ARM64 + XL710 controllers?
>
> This patch relaxes the rte_io_*mb barriers for pure PCI device memory accesses.

Yes. This would break cases for mixed access fro i40e drivers.

>
> For mixed accesses of DDR and PCI device memory, rte_smp_*mb(DMB ISH) is not sufficient.
> But rte_cio_*mb(DMB OSH) is sufficient and can be used.

Yes. Let me share a bit of history.

1) There are a lot of drivers(initially developed in x86) that have
mixed access and don't have any barriers as x86 does not need it.
2) rte_io introduced to fix that
3) Item (2) introduced the performance issues in the fast path as an
optimization rte_cio_* introduced.

So in the current of the scheme of things, we have APIs to FIX
portability issue(rte_io) and performance issue(rte_cio).
IMO, we may not need any change in infra code now. If you think, the
documentation is missing then we can enhance it.
If we make infra change then again drivers needs to be updated and tested.