<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Exchange Server">  <style></style> </head> <body> <font face="Calibri" size="2"><span style="font-size:10pt;"> <div style="padding-right:5pt;padding-left:5pt;"><font color="blue">[AMD Official Use Only - AMD Internal Distribution Only]<br> </font></div> <div style="margin-top:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"><br> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"><snipped></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>>> Thank you Mattias for the information, as shared by in the reply</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>>> with</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> Anatoly we want expose a new API `rte_get_next_lcore_ex` which</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> intakes a extra argument `u32 flags`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>>> The flags can be RTE_GET_LCORE_L1 (SMT), RTE_GET_LCORE_L2,</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> RTE_GET_LCORE_L3, RTE_GET_LCORE_BOOST_ENABLED,</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> RTE_GET_LCORE_BOOST_DISABLED.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> Wouldn't using that API be pretty awkward to use?</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Current API available under DPDK is ` rte_get_next_lcore`, which is used</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> within DPDK example and in customer solution.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Based on the comments from others we responded to the idea of changing</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> the new Api from ` rte_get_next_lcore_llc` to ` rte_get_next_lcore_exntd`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> ></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Can you please help us understand what is `awkward`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> ></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> The awkwardness starts when you are trying to fit provide hwloc type</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> information over an API that was designed for iterating over lcores.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">I disagree to this point, current implementation of lcore libraries is only focused on iterating through list of enabled cores, core-mask, and lcore-map.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">With ever increasing core count, memory, io and accelerators on SoC, sub-numa partitioning is common in various vendor SoC. Enhancing or Augumenting lcore API to extract or provision NUMA, Cache Topology is not awkward.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">If memory, IO and accelerator can have sub-NUMA domain, why is it awkward to have lcore in domains? Hence I do not agree on the awkwardness argument.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> It seems to me that you should either have:</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> A) An API in similar to that of hwloc (or any DOM-like API), which would give a</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> low-level description of the hardware in implementation terms.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> The topology would consist of nodes, with attributes, etc, where nodes are</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> things like cores or instances of caches of some level and attributes are things</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> like CPU actual and nominal, and maybe max frequency, cache size, or memory</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> size.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Here is the catch, `rte_eal_init` internally invokes `get_cpu|lcores` and populates thread (lcore) to physical CPU. But there is more than just CPU mapping, as we have seeing in SoC architecture. The argument shared by many is `DPDK is not the place for such topology discovery`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">As per my current understanding, I have to disagree to the abive because </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">1. forces user to use external libraries example like hwloc</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">2. forces user to creating internal mapping for lcore, core-mask, and lcore-map with topology awareness code.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">My intention is to `enable end user to leverage the API format or similar API format (rte_get_next_lcore)` to get best results on any SoC (vendor agnostic).</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">I fail to grasp why we are asking CPU topology to exported, while NIC, PCIe and accelerators are not asked to be exported via external libraries like hwloc. </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Hence let us setup tech call in slack or teams to understand this better.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> or</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> B) An API to be directly useful for a work scheduler, in which case you should</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> abstract away things like "boost"</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Please note as shared in earlier reply to Bruce, I made a mistake of calling it boost (AMD SoC terminology). Instead it should DPDK_TURBO.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">There are use cases and DPDK examples, where cypto and compression are run on cores where TURBO is enabled. This allows end users to boost when there is more work and disable boost when there is less or no work.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> (and fold them into some abstract capacity notion, together with core "size" [in big-little/heterogeneous systems]), and</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> have an abstract notion of what core is "close" to some other core. This would</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> something like Linux'</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> scheduling domains.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">We had similar discussion with Jerrin on the last day of Bangkok DPDK summit. This RFC was intended to help capture this relevant point. With my current understanding on selected SoC the little core on ARM Soc shares L2 cache, while this analogy does not cover all cases. But this would be good start. </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> If you want B you probably need A as a part of its implementation, so you may</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> just as well start with A, I suppose.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> What you could do to explore the API design is to add support for, for</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> example, boost core awareness or SMT affinity in the SW scheduler. You could</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> also do an "lstopo" equivalent, since that's needed for debugging and</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> exploration, if nothing else.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Not following on this analogy, will discuss in detail in tech talk</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> One question that will have to be answered in a work scheduling scenario is</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> "are these two lcores SMT siblings," or "are these two cores on the same LLC",</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> or "give me all lcores on a particular L2 cache".</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Is not that we have been trying to address based on Anatoly request to generalize than LLC. Hence we agreed on sharing version-2 of RFC with `rte_get_nex_lcore_extnd` with `flags`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">May I ask where is the disconnect?</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> I mean, what you have is a topology, with nodes of different types</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> and with</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> different properties, and you want to present it to the user.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Let me be clear, what we want via DPDK to help customer to use an Unified</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> API which works across multiple platforms.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Example - let a vendor have 2 products namely A and B. CPU-A has all cores</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> within same SUB-NUMA domain and CPU-B has cores split to 2 sub-NUMA</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> domain based on split LLC.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > When `rte_get_next_lcore_extnd` is invoked for `LLC` on 1. CPU-A: it</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > returns all cores as there is no split 2. CPU-B: it returns cores from</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > specific sub-NUMA which is partitioned by L3</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> ></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> I think the function name rte_get_next_lcore_extnd() alone makes clear this is an awkward API. :)</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">I humbly disagree to this statement, as explained above.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> My gut feeling is to make it more explicit and forget about <rte_lcore.h>.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> <rte_hwtopo.h>? Could and should still be EAL.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">For me this is like adding a new level of library and more code. While the easiest way was to add an API similar to existing `get_next_lcore` style for easy adoption.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> In a sense, it's similar to XCM and DOM versus SAX. The above is</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> SAX-style,</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> and what I have in mind is something DOM-like.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >>> What use case do you have in mind? What's on top of my list is a scenario where a DPDK app gets a bunch of cores (e.g., -l <cores>) and tries to figure out how best make use of them.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Exactly.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> ></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > It's not going to "skip" (ignore, leave unused)</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> SMT siblings, or skip non-boosted cores, it would just try to be</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> >> clever in regards to which cores to use for what purpose.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > Let me try to share my idea on SMT sibling. When user invoked for</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> rte_get_next_lcore_extnd` is invoked for `L1 | SMT` flag with `lcore`; the API</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> identifies first whether given lcore is part of enabled core list.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > If yes, it programmatically either using `sysfs` or `hwloc library (shared the</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> version concern on distros. Will recheck again)` identify the sibling thread and</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> return.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> > If there is no sibling thread available under DPDK it will fetch next lcore </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> (probably lcore +1 ).</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> ></span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">> Distributions having old hwloc versions isn't an argument for a new DPDK library or new API. If only that was the issue, then it would be better to help the hwloc and/or distributions, rather than the DPDK project.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">I do not agree to terms of ` Distributions having old hwloc versions isn't an argument for a new DPDK library or new API.` Because this is not what my intention is. Let me be clear on Ampere & AMD Bios settings are 2</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">1. SLC or L3 as NUMA enable</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">2. Numa for IO|memory</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">With `NUMA for IO|memory` is set hwloc library works as expected. But when `L3 as NUMA` is set gives incorrect details. We have been fixing this and pushing to upstream. But as I clearly shared, version of distros having latest hwloc is almost nil.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Hence to keep things simple, in documentation of DPDK we pointed to AMD SoC tuning guide we have been recommending not to enable `L3 as NUMA`.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;">Now end goal for me is to allow vendor agnostic API which is easy to understand and use, and works irrespective of BIOS settings. I have enabled parsing of OS `sysfs` as a RFC. But if the comment is to use `hwloc` as shared with response for Stephen I am open to try this again.</span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></div> <div><font face="Calibri" size="2"><span style="font-size:11pt;"><snipped></span></font></div> </span></font> </body> </html>