<!DOCTYPE html><html data-lt-installed="true"><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="padding-bottom: 1px;">
<div class="moz-cite-prefix">On 2023/8/18 21:59, Honnappa
Nagarahalli wrote:<br>
</div>
<blockquote type="cite" cite="mid:DBAPR08MB58141E6BDA678F2DDDBF1B07981BA@DBAPR08MB5814.eurprd08.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Jack Min
<a class="moz-txt-link-rfc2396E" href="mailto:jackmin@nvidia.com"><jackmin@nvidia.com></a> <br>
<b>Sent:</b> Friday, August 18, 2023 12:57 AM<br>
<b>To:</b> Honnappa Nagarahalli
<a class="moz-txt-link-rfc2396E" href="mailto:Honnappa.Nagarahalli@arm.com"><Honnappa.Nagarahalli@arm.com></a>; Stephen Hemminger
<a class="moz-txt-link-rfc2396E" href="mailto:stephen@networkplumber.org"><stephen@networkplumber.org></a><br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:dev@dpdk.org">dev@dpdk.org</a>; Matan Azrad
<a class="moz-txt-link-rfc2396E" href="mailto:matan@nvidia.com"><matan@nvidia.com></a>; <a class="moz-txt-link-abbreviated" href="mailto:viacheslavo@nvidia.com">viacheslavo@nvidia.com</a>; Tyler
Retzlaff <a class="moz-txt-link-rfc2396E" href="mailto:roretzla@linux.microsoft.com"><roretzla@linux.microsoft.com></a>; Wathsala
Wathawana Vithanage <a class="moz-txt-link-rfc2396E" href="mailto:wathsala.vithanage@arm.com"><wathsala.vithanage@arm.com></a>;
nd <a class="moz-txt-link-rfc2396E" href="mailto:nd@arm.com"><nd@arm.com></a><br>
<b>Subject:</b> Re: MLX5 PMD access ring library private
data<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 2023/8/18 12:30, Honnappa
Nagarahalli wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>-----Original Message-----<o:p></o:p></pre>
<pre>From: Jack Min <a href="mailto:jackmin@nvidia.com" moz-do-not-send="true"><jackmin@nvidia.com></a><o:p></o:p></pre>
<pre>Sent: Thursday, August 17, 2023 9:32 PM<o:p></o:p></pre>
<pre>To: Stephen Hemminger <a href="mailto:stephen@networkplumber.org" moz-do-not-send="true"><stephen@networkplumber.org></a>; Honnappa<o:p></o:p></pre>
<pre>Nagarahalli <a href="mailto:Honnappa.Nagarahalli@arm.com" moz-do-not-send="true"><Honnappa.Nagarahalli@arm.com></a><o:p></o:p></pre>
<pre>Cc: <a href="mailto:dev@dpdk.org" moz-do-not-send="true" class="moz-txt-link-freetext">dev@dpdk.org</a>; Matan Azrad <a href="mailto:matan@nvidia.com" moz-do-not-send="true"><matan@nvidia.com></a>;<o:p></o:p></pre>
<pre><a href="mailto:viacheslavo@nvidia.com" moz-do-not-send="true" class="moz-txt-link-freetext">viacheslavo@nvidia.com</a>; Tyler Retzlaff <a href="mailto:roretzla@linux.microsoft.com" moz-do-not-send="true"><roretzla@linux.microsoft.com></a>;<o:p></o:p></pre>
<pre>Wathsala Wathawana Vithanage <a href="mailto:wathsala.vithanage@arm.com" moz-do-not-send="true"><wathsala.vithanage@arm.com></a>; nd<o:p></o:p></pre>
<pre><a href="mailto:nd@arm.com" moz-do-not-send="true"><nd@arm.com></a><o:p></o:p></pre>
<pre>Subject: Re: MLX5 PMD access ring library private data<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>On 2023/8/17 22:06, Stephen Hemminger wrote:<o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>On Thu, 17 Aug 2023 05:06:20 +0000<o:p></o:p></pre>
<pre>Honnappa Nagarahalli <a href="mailto:Honnappa.Nagarahalli@arm.com" moz-do-not-send="true"><Honnappa.Nagarahalli@arm.com></a> wrote:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Hi Matan, Viacheslav,<o:p></o:p></pre>
<pre> Tyler pointed out that the function<o:p></o:p></pre>
</blockquote>
</blockquote>
<pre>__mlx5_hws_cnt_pool_enqueue_revert is accessing the ring private structure<o:p></o:p></pre>
<pre>members (prod.head and prod.tail) directly. Even though ' struct rte_ring' is a<o:p></o:p></pre>
<pre>public structure (mainly because the library provides inline functions), the<o:p></o:p></pre>
<pre>structure members are considered private to the ring library. So, this needs to<o:p></o:p></pre>
<pre>be corrected.<o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>It looks like the function __mlx5_hws_cnt_pool_enqueue_revert is trying<o:p></o:p></pre>
</blockquote>
</blockquote>
<pre>to revert things that were enqueued. It is not clear to me why this<o:p></o:p></pre>
<pre>functionality is required. Can you provide the use case for this? We can<o:p></o:p></pre>
<pre>discuss possible solutions.<o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>How can reverting be thread safe? Consumer could have already looked at<o:p></o:p></pre>
</blockquote>
<pre>them?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hey,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>In our case, this ring is SC/SP, only accessed by one thread<o:p></o:p></pre>
<pre>(enqueue/dequeue/revert).<o:p></o:p></pre>
</blockquote>
<pre>You could implement a more simpler and more efficient (For ex: such an implementation would not need any atomic operations, would require less number of cache lines) ring for this.<o:p></o:p></pre>
<pre>Is this function being used in the dataplane?<o:p></o:p></pre>
</blockquote>
<p>Yes, we can have our own version of ring (no atomic
operations) but basic operation are still as same as
rte_ring.<o:p></o:p></p>
<p>Since rte ring has been well-designed and tested
sufficiently, so there is no strong reason to re-write a new
simple version of it until today :)<o:p></o:p></p>
<p><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>The scenario we have "revert" is:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> We use ring to manager our HW objects (counter in this case) and for each<o:p></o:p></pre>
<pre>core (thread) has "cache" (a SC/SP ring) for sake of performance.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>1. Get objects from "cache" firstly, if cache is empty, we fetch a bulk of free<o:p></o:p></pre>
<pre>objects from global ring into cache.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>2. Put (free) objects also into "cache" firstly, if cache is full, we flush a bulk of<o:p></o:p></pre>
<pre>objects into global ring in order to make some rooms in cache.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>However, this HW object cannot be immediately reused after free. It needs<o:p></o:p></pre>
<pre>time to be reset and then can be used again.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>So when we flush cache, we want to keep the first enqueued objects still stay<o:p></o:p></pre>
<pre>there because they have more chance already be reset than the latest<o:p></o:p></pre>
<pre>enqueued objects.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Only flush recently enqueued objects back into global ring, act as "LIFO"<o:p></o:p></pre>
<pre>behavior.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This is why we require "revert" enqueued objects.<o:p></o:p></pre>
</blockquote>
<pre>You could use 'rte_ring_free_count' API before you enqueue to check for available space.<o:p></o:p></pre>
</blockquote>
<p>Only when cache is full (rte_ring_free_count() is zero), we
revert X objects. <o:p></o:p></p>
<p>If there is still one free slot we will not trigger revert
(flush).<o:p></o:p></p>
<p><b><i>[Honnappa]</i></b> May be I was not clear in my
recommendation. What I am saying is, you could call
‘rte_ring_free_count’ to check if you have enough space on
the cache ring. If there is not enough space you can enqueue
the new objects on the global ring. Pseudo code below:<o:p></o:p></p>
<p>If (rte_ring_free_count(cache_ring) > n) {<o:p></o:p></p>
<p> <enqueue n objects on cache ring><o:p></o:p></p>
<p>} else {<o:p></o:p></p>
<p> <enqueue n objects on global ring><o:p></o:p></p>
<p>}</p>
</div>
</div>
</blockquote>
<p>Hey, <br>
</p>
<p>Then next n objects will still enqueue into global ring, not into
cache , right? ( we enqueue nnnn objects continually)<br>
</p>
<p>Our requirement is like this:<br>
</p>
<p>if (rte_ring_free_count(cache_ring) > 0) {</p>
<p> <enqueue this object on cache ring></p>
<p>} else { /* cache is full */<br>
</p>
<p> <enqueue this object into global ring> <br>
</p>
<p> <move the latest n objects into global ring too></p>
<p>}</p>
<p>It's not about if this enqueue on cache can success or not. </p>
<p>It's about we need "free" more room in advance so next n objects
can enqueue into cache.</p>
<p>-Jack<br>
</p>
<blockquote type="cite" cite="mid:DBAPR08MB58141E6BDA678F2DDDBF1B07981BA@DBAPR08MB5814.eurprd08.prod.outlook.com">
<div class="WordSection1">
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>-Jack<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
</blockquote>
</div>
</div>
</blockquote>
</body>
<lt-container></lt-container>
</html>