[PATCH v4 1/4] bus/cdx: introduce cdx bus

[Xia, Chenbo]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit at amd.com>
> Sent: Tuesday, May 9, 2023 7:26 PM
> To: Gupta, Nipun <Nipun.Gupta at amd.com>; Xia, Chenbo <chenbo.xia at intel.com>;
> dev at dpdk.org; thomas at monjalon.net; david.marchand at redhat.com
> Cc: Anand, Harpreet <harpreet.anand at amd.com>; Agarwal, Nikhil
> <nikhil.agarwal at amd.com>
> Subject: Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> 
> On 5/9/2023 12:09 PM, Gupta, Nipun wrote:
> > [AMD Official Use Only - General]
> >
> >
> >
> >> -----Original Message-----
> >> From: Xia, Chenbo <chenbo.xia at intel.com>
> >> Sent: Tuesday, May 9, 2023 12:25 PM
> >> To: Gupta, Nipun <Nipun.Gupta at amd.com>; dev at dpdk.org;
> >> thomas at monjalon.net; david.marchand at redhat.com
> >> Cc: Yigit, Ferruh <Ferruh.Yigit at amd.com>; Anand, Harpreet
> >> <harpreet.anand at amd.com>; Agarwal, Nikhil <nikhil.agarwal at amd.com>
> >> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> >>
> >> Caution: This message originated from an External Source. Use proper
> >> caution when opening attachments, clicking links, or responding.
> >>
> >>
> >>> -----Original Message-----
> >>> From: Nipun Gupta <nipun.gupta at amd.com>
> >>> Sent: Monday, May 8, 2023 7:18 PM
> >>> To: dev at dpdk.org; thomas at monjalon.net; david.marchand at redhat.com
> >>> Cc: ferruh.yigit at amd.com; harpreet.anand at amd.com;
> >> nikhil.agarwal at amd.com;
> >>> Nipun Gupta <nipun.gupta at amd.com>
> >>> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
> >>>
> >>> CDX bus supports multiple type of devices, which can be
> >>> exposed to user-space via vfio-cdx.
> >>>
> >>> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
> >>> DMA interface for the device (IOMMU).
> >>>
> >>> This support aims to enable the DPDK to support the cdx
> >>> devices in user-space using VFIO interface.
> >>>
> >>> Signed-off-by: Nipun Gupta <nipun.gupta at amd.com>
> >>> ---
> >>>  MAINTAINERS                            |   5 +
> >>>  doc/guides/rel_notes/release_23_07.rst |   6 +
> >>>  drivers/bus/cdx/bus_cdx_driver.h       | 201 ++++++++++
> >>>  drivers/bus/cdx/cdx.c                  | 520
> +++++++++++++++++++++++++
> >>>  drivers/bus/cdx/cdx_logs.h             |  37 ++
> >>>  drivers/bus/cdx/cdx_vfio.c             | 437 +++++++++++++++++++++
> >>>  drivers/bus/cdx/meson.build            |  13 +
> >>>  drivers/bus/cdx/private.h              |  49 +++
> >>>  drivers/bus/cdx/version.map            |  11 +
> >>>  drivers/bus/meson.build                |   1 +
> >>>  10 files changed, 1280 insertions(+)
> >>>  create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
> >>>  create mode 100644 drivers/bus/cdx/cdx.c
> >>>  create mode 100644 drivers/bus/cdx/cdx_logs.h
> >>>  create mode 100644 drivers/bus/cdx/cdx_vfio.c
> >>>  create mode 100644 drivers/bus/cdx/meson.build
> >>>  create mode 100644 drivers/bus/cdx/private.h
> >>>  create mode 100644 drivers/bus/cdx/version.map
> >>>
> >>
> >> ...
> >>
> >>> --- /dev/null
> >>> +++ b/drivers/bus/cdx/cdx.c
> >>> @@ -0,0 +1,520 @@
> >>> +/* SPDX-License-Identifier: BSD-3-Clause
> >>> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
> >>> + */
> >>> +
> >>> +/*
> >>> + * Architecture Overview
> >>> + * =====================
> >>> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
> >>> + * consists of sophisticated mechanism for interaction between FPGA,
> >>> + * Firmware and the APUs (Application CPUs).
> >>> + *
> >>> + * Firmware resides on RPU (Realtime CPUs) which interacts with
> >>> + * the FPGA program manager and the APUs. The RPU provides memory-
> >> mapped
> >>> + * interface (RPU if) which is used to communicate with APUs.
> >>> + *
> >>> + * The diagram below shows an overview of the CDX architecture:
> >>> + *
> >>> + *          +--------------------------------------+
> >>> + *          |   DPDK                               |
> >>> + *          |                    DPDK CDX drivers  |
> >>> + *          |                             |        |
> >>> + *          |                      DPDK CDX bus    |
> >>> + *          |                             |        |
> >>> + *          +-----------------------------|--------+
> >>> + *                                        |
> >>> + *          +-----------------------------|--------+
> >>> + *          |    Application CPUs (APU)   |        |
> >>> + *          |                             |        |
> >>> + *          |                     VFIO CDX driver  |
> >>> + *          |     Linux OS                |        |
> >>> + *          |                      Linux CDX bus   |
> >>> + *          |                             |        |
> >>> + *          +-----------------------------|--------+
> >>> + *                                        |
> >>> + *                                        |
> >>> + *          +------------------------| RPU if |----+
> >>> + *          |                             |        |
> >>> + *          |                             V        |
> >>> + *          |          Realtime CPUs (RPU)         |
> >>> + *          |                                      |
> >>> + *          +--------------------------------------+
> >>> + *                                |
> >>> + *          +---------------------|----------------+
> >>> + *          |  FPGA               |                |
> >>> + *          |      +-----------------------+       |
> >>> + *          |      |           |           |       |
> >>> + *          | +-------+    +-------+   +-------+   |
> >>> + *          | | dev 1 |    | dev 2 |   | dev 3 |   |
> >>> + *          | +-------+    +-------+   +-------+   |
> >>> + *          +--------------------------------------+
> >>> + *
> >>> + * The RPU firmware extracts the device information from the loaded
> >> FPGA
> >>> + * image and implements a mechanism that allows the APU drivers to
> >>> + * enumerate such devices (device personality and resource details)
> via
> >>> + * a dedicated communication channel.
> >>
> >> What is APU? CPU resources that application uses? Then why cpu
> resources
> >> that DPDK
> >> Uses are not part of it?
> >
> > Hi Chenbo,
> >
> > APU's are application processor unit and are on-chip CPU's. So in short
> APUs
> > are CPUs only on which applications like DPDK are running.
> >
> 
> There are two sets of processing unit in the SoC. FW runs on RPU,
> everything else runs on APU (Arm cores), including OS, DPDK etc..

Cool, thanks Nipun and Ferruh for the explanation!