[PATCH v4 1/4] bus/cdx: introduce cdx bus

[Ferruh Yigit]

On 5/9/2023 12:09 PM, Gupta, Nipun wrote:
> [AMD Official Use Only - General]
> 
> 
> 
>> -----Original Message-----
>> From: Xia, Chenbo <chenbo.xia at intel.com>
>> Sent: Tuesday, May 9, 2023 12:25 PM
>> To: Gupta, Nipun <Nipun.Gupta at amd.com>; dev at dpdk.org;
>> thomas at monjalon.net; david.marchand at redhat.com
>> Cc: Yigit, Ferruh <Ferruh.Yigit at amd.com>; Anand, Harpreet
>> <harpreet.anand at amd.com>; Agarwal, Nikhil <nikhil.agarwal at amd.com>
>> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>>
>> Caution: This message originated from an External Source. Use proper
>> caution when opening attachments, clicking links, or responding.
>>
>>
>>> -----Original Message-----
>>> From: Nipun Gupta <nipun.gupta at amd.com>
>>> Sent: Monday, May 8, 2023 7:18 PM
>>> To: dev at dpdk.org; thomas at monjalon.net; david.marchand at redhat.com
>>> Cc: ferruh.yigit at amd.com; harpreet.anand at amd.com;
>> nikhil.agarwal at amd.com;
>>> Nipun Gupta <nipun.gupta at amd.com>
>>> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus
>>>
>>> CDX bus supports multiple type of devices, which can be
>>> exposed to user-space via vfio-cdx.
>>>
>>> vfio-cdx provides the MMIO IO_MEMORY regions as well as the
>>> DMA interface for the device (IOMMU).
>>>
>>> This support aims to enable the DPDK to support the cdx
>>> devices in user-space using VFIO interface.
>>>
>>> Signed-off-by: Nipun Gupta <nipun.gupta at amd.com>
>>> ---
>>>  MAINTAINERS                            |   5 +
>>>  doc/guides/rel_notes/release_23_07.rst |   6 +
>>>  drivers/bus/cdx/bus_cdx_driver.h       | 201 ++++++++++
>>>  drivers/bus/cdx/cdx.c                  | 520 +++++++++++++++++++++++++
>>>  drivers/bus/cdx/cdx_logs.h             |  37 ++
>>>  drivers/bus/cdx/cdx_vfio.c             | 437 +++++++++++++++++++++
>>>  drivers/bus/cdx/meson.build            |  13 +
>>>  drivers/bus/cdx/private.h              |  49 +++
>>>  drivers/bus/cdx/version.map            |  11 +
>>>  drivers/bus/meson.build                |   1 +
>>>  10 files changed, 1280 insertions(+)
>>>  create mode 100644 drivers/bus/cdx/bus_cdx_driver.h
>>>  create mode 100644 drivers/bus/cdx/cdx.c
>>>  create mode 100644 drivers/bus/cdx/cdx_logs.h
>>>  create mode 100644 drivers/bus/cdx/cdx_vfio.c
>>>  create mode 100644 drivers/bus/cdx/meson.build
>>>  create mode 100644 drivers/bus/cdx/private.h
>>>  create mode 100644 drivers/bus/cdx/version.map
>>>
>>
>> ...
>>
>>> --- /dev/null
>>> +++ b/drivers/bus/cdx/cdx.c
>>> @@ -0,0 +1,520 @@
>>> +/* SPDX-License-Identifier: BSD-3-Clause
>>> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc.
>>> + */
>>> +
>>> +/*
>>> + * Architecture Overview
>>> + * =====================
>>> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It
>>> + * consists of sophisticated mechanism for interaction between FPGA,
>>> + * Firmware and the APUs (Application CPUs).
>>> + *
>>> + * Firmware resides on RPU (Realtime CPUs) which interacts with
>>> + * the FPGA program manager and the APUs. The RPU provides memory-
>> mapped
>>> + * interface (RPU if) which is used to communicate with APUs.
>>> + *
>>> + * The diagram below shows an overview of the CDX architecture:
>>> + *
>>> + *          +--------------------------------------+
>>> + *          |   DPDK                               |
>>> + *          |                    DPDK CDX drivers  |
>>> + *          |                             |        |
>>> + *          |                      DPDK CDX bus    |
>>> + *          |                             |        |
>>> + *          +-----------------------------|--------+
>>> + *                                        |
>>> + *          +-----------------------------|--------+
>>> + *          |    Application CPUs (APU)   |        |
>>> + *          |                             |        |
>>> + *          |                     VFIO CDX driver  |
>>> + *          |     Linux OS                |        |
>>> + *          |                      Linux CDX bus   |
>>> + *          |                             |        |
>>> + *          +-----------------------------|--------+
>>> + *                                        |
>>> + *                                        |
>>> + *          +------------------------| RPU if |----+
>>> + *          |                             |        |
>>> + *          |                             V        |
>>> + *          |          Realtime CPUs (RPU)         |
>>> + *          |                                      |
>>> + *          +--------------------------------------+
>>> + *                                |
>>> + *          +---------------------|----------------+
>>> + *          |  FPGA               |                |
>>> + *          |      +-----------------------+       |
>>> + *          |      |           |           |       |
>>> + *          | +-------+    +-------+   +-------+   |
>>> + *          | | dev 1 |    | dev 2 |   | dev 3 |   |
>>> + *          | +-------+    +-------+   +-------+   |
>>> + *          +--------------------------------------+
>>> + *
>>> + * The RPU firmware extracts the device information from the loaded
>> FPGA
>>> + * image and implements a mechanism that allows the APU drivers to
>>> + * enumerate such devices (device personality and resource details) via
>>> + * a dedicated communication channel.
>>
>> What is APU? CPU resources that application uses? Then why cpu resources
>> that DPDK
>> Uses are not part of it?
> 
> Hi Chenbo,
> 
> APU's are application processor unit and are on-chip CPU's. So in short APUs
> are CPUs only on which applications like DPDK are running.
> 

There are two sets of processing unit in the SoC. FW runs on RPU,
everything else runs on APU (Arm cores), including OS, DPDK etc..