<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi all,<o:p></o:p> <o:p> </o:p> Going trough all notes on <a href="http://patches.dpdk.org/project/dpdk/patch/20220427092357.491720-1-marcinx.danilewicz@intel.com/"> http://patches.dpdk.org/project/dpdk/patch/20220427092357.491720-1-marcinx.danilewicz@intel.com/</a> please find my answers inline here.<o:p></o:p> <o:p> </o:p> First of all please take my apologies, for not checking url as above on regular basis. I was awaiting for mails with me on CC along to comments. <o:p></o:p> <o:p> </o:p> So lets go in to comments:<o:p></o:p> <o:p> </o:p> <o:p> </o:p> I don't see any note on the changes made in this version with respect to previous versions. Can you include them in future version? Also, I had some comments on the first version of this patch, I don't see any response.<o:p></o:p> @Marcin: hopefully all required changes are in new v5 patch<o:p></o:p> <o:p> </o:p> > int<o:p></o:p> > rte_sched_port_dequeue(struct rte_sched_port *port, struct rte_mbuf **pkts,<o:p></o:p> > uint32_t n_pkts);<o:p></o:p> > <o:p></o:p> > +/**<o:p></o:p> > + * Hierarchical scheduler subport TC OV enable/disable config.<o:p></o:p> <o:p> </o:p> The name of the feature should be fully stated here: traffic class oversubscription, not the abbreviation, please change.<o:p></o:p> <o:p> </o:p> @Marcin: please check v5 patch<o:p></o:p> <o:p> </o:p> > + * Note that this function is safe to use at runtime<o:p></o:p> > + * to enable/disable TC OV for subport.<o:p></o:p> <o:p> </o:p> We should actually forbit this rather than encourage it. Calling this function several times does not make sense, and it can create limitations that can come back and byte us in the future, whenever we might need to extend this code, for no reason.<o:p></o:p> <o:p> </o:p> Please actually replace with: "This function should be called at the time of subport initialization."<o:p></o:p> <o:p> </o:p> @Marcin: please check v5 patch<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > + *<o:p></o:p> > + * @param port<o:p></o:p> > + * Handle to port scheduler instance<o:p></o:p> > + * @param subport_id<o:p></o:p> > + * Subport ID<o:p></o:p> > + * @param tc_ov_enable<o:p></o:p> > + * Boolean flag to enable/disable TC OV<o:p></o:p> > + * @return<o:p></o:p> > + * 0 upon success, error code otherwise<o:p></o:p> > + */<o:p></o:p> > +__rte_experimental<o:p></o:p> > +int<o:p></o:p> > +rte_sched_subport_tc_ov_config(struct rte_sched_port *port, uint32_t<o:p></o:p> > subport_id, bool tc_ov_enable);<o:p></o:p> > +<o:p></o:p> > #ifdef __cplusplus<o:p></o:p> > }<o:p></o:p> > #endif<o:p></o:p> > diff --git a/lib/sched/version.map b/lib/sched/version.map<o:p></o:p> > index d22c07fc9f..c6e994d8df 100644<o:p></o:p> > --- a/lib/sched/version.map<o:p></o:p> > +++ b/lib/sched/version.map<o:p></o:p> > @@ -34,4 +34,7 @@ EXPERIMENTAL {<o:p></o:p> > # added in 21.11<o:p></o:p> > rte_pie_rt_data_init;<o:p></o:p> > rte_pie_config_init;<o:p></o:p> > +<o:p></o:p> > + # added in 22.03<o:p></o:p> <o:p> </o:p> This is not in 22.03, it will hopefully be in 22.07.<o:p></o:p> <o:p> </o:p> > + rte_sched_subport_tc_ov_config;<o:p></o:p> > };<o:p></o:p> > --<o:p></o:p> > 2.25.1<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > subport->profile;<o:p></o:p> > ><o:p></o:p> > > grinder_prefetch_tc_queue_arrays(subport, pos);<o:p></o:p> > > - grinder_credits_update(port, subport, pos);<o:p></o:p> > > +<o:p></o:p> > > + if (unlikely(subport->is_tc_ov_enabled))<o:p></o:p> > <o:p></o:p> > Please remove the "unlikely" from here, don't put any likely/unlikely here at all.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> @Marcin: as also noted below, I agree and all ours are removed.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > ><o:p></o:p> > > +/**<o:p></o:p> > > + * Hierarchical scheduler subport TC OV enable/disable config.<o:p></o:p> > <o:p></o:p> > The name of the feature should be fully stated here: traffic class<o:p></o:p> > oversubscription, not the abbreviation, please change.<o:p></o:p> > <o:p></o:p> > <o:p></o:p> <o:p> </o:p> @Marcin: please check changes<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > + * Note that this function is safe to use at runtime<o:p></o:p> > > + * to enable/disable TC OV for subport.<o:p></o:p> > <o:p></o:p> > We should actually forbit this rather than encourage it. Calling this function<o:p></o:p> > several times does not make sense, and it can create limitations that can come<o:p></o:p> > back and byte us in the future, whenever we might need to extend this code, for<o:p></o:p> > no reason.<o:p></o:p> > <o:p></o:p> > Please actually replace with: "This function should be called at the time of<o:p></o:p> > subport initialization."<o:p></o:p> <o:p> </o:p> <o:p> </o:p> @Marcin: please check changes<o:p></o:p> <o:p> </o:p> > > @@ -34,4 +34,7 @@ EXPERIMENTAL {<o:p></o:p> > > # added in 21.11<o:p></o:p> > > rte_pie_rt_data_init;<o:p></o:p> > > rte_pie_config_init;<o:p></o:p> > > +<o:p></o:p> > > + # added in 22.03<o:p></o:p> > <o:p></o:p> > This is not in 22.03, it will hopefully be in 22.07.<o:p></o:p> > <o:p></o:p> <o:p> </o:p> @Marcin: Well, nice spot!<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > Also the name of the feature should not be abbreviated in the patch title.<o:p></o:p> > ><o:p></o:p> > > I suggest you rework the title to:<o:p></o:p> > > [PATCH] sched: enable traffic class oversubscription conditionally<o:p></o:p> > ><o:p></o:p> @Marcin: already done<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > ><o:p></o:p> > > > Added new API to enable or disable TC over subscription for best<o:p></o:p> > > > effort traffic class at subport level.<o:p></o:p> > > > Added changes after review and increased throughput.<o:p></o:p> > > ><o:p></o:p> > > > By default TC OV is disabled.<o:p></o:p> > ><o:p></o:p> > > It should be the other way around, the TC_OV should be enabled by default.<o:p></o:p> > The<o:p></o:p> > > TC oversubscription is a more natural way to use this library, we usually want<o:p></o:p> > to<o:p></o:p> > > disable this feature just for better performance in case this functionality is<o:p></o:p> > not<o:p></o:p> > > needed. Please initialize the tc_ov flag accordingly.<o:p></o:p> > ><o:p></o:p> > <o:p></o:p> > In original code, this feature has always been disabled as it impacts<o:p></o:p> > performance.<o:p></o:p> > So, in my opinion we should keep it disabled by default and let user enable it<o:p></o:p> > when required.<o:p></o:p> > <o:p></o:p> <o:p> </o:p> In the original code, yes, it had to be explicitly enabled through a build-time flag. This was not the best option, and this is precisely what we are trying to fix with this patch.<o:p></o:p> <o:p> </o:p> But on the other hand all the users of these library that I know use it with the TC oversubscription turned on. Functionality is more important for them than performance. Hence my vote now is to enable it by default; those users that prefer performance over functionality can easily turn this feature off with no issues.<o:p></o:p> <o:p> </o:p> @Marcin: OK<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > > uint8_t memory[0] __rte_cache_aligned;<o:p></o:p> > > > +<o:p></o:p> > > > + /* TC oversubscription activation */<o:p></o:p> > > > + int is_tc_ov_enabled;<o:p></o:p> > ><o:p></o:p> > > How about we simplify the name of this variable to: tc_ov_enabled ?<o:p></o:p> @Marcin: agree 😊<o:p></o:p> <o:p> </o:p> > > > + s = port->subports[subport_id];<o:p></o:p> > > > + s->is_tc_ov_enabled = tc_ov_enable ? 1 : 0;<o:p></o:p> > > > +<o:p></o:p> > > > + if (s->is_tc_ov_enabled) {<o:p></o:p> > > > + /* TC oversubscription */<o:p></o:p> > > > + s->tc_ov_wm_min = port->mtu;<o:p></o:p> > > > + s->tc_ov_period_id = 0;<o:p></o:p> > > > + s->tc_ov = 0;<o:p></o:p> > > > + s->tc_ov_n = 0;<o:p></o:p> > > > + s->tc_ov_rate = 0;<o:p></o:p> > > > +<o:p></o:p> > > > + profile = port->subport_profiles + s->profile;<o:p></o:p> > > > + s->tc_ov_wm_max = rte_sched_time_ms_to_bytes(profile-<o:p></o:p> > > > >tc_period,<o:p></o:p> > > > + s->pipe_tc_be_rate_max);<o:p></o:p> > > > + s->tc_ov_wm = s->tc_ov_wm_max;<o:p></o:p> > > > + }<o:p></o:p> > > > + return 0;<o:p></o:p> > > > +}<o:p></o:p> > ><o:p></o:p> > > This function should not exist, please remove it and keep the initial code that<o:p></o:p> > > computes the tc_ov related variable regardless of whether tc_ov is enabled<o:p></o:p> > or<o:p></o:p> > > not.<o:p></o:p> > ><o:p></o:p> > > All the tc_ov related variables have the tc_ov particle in their name, so there<o:p></o:p> > is<o:p></o:p> > > no clash. This is initialization code, so no performance overhead. Let's keep<o:p></o:p> > the<o:p></o:p> > > code unmodified and compute both the tc_ov and the non-tc_ov varables at<o:p></o:p> > > initialization, regardless of whether the feature is enabled or not.<o:p></o:p> > ><o:p></o:p> > > This comment is applicable to all the initialization code, please adjust all the<o:p></o:p> > init<o:p></o:p> > > code accordingly. There should be no diff showing in the patch for any of the<o:p></o:p> > init<o:p></o:p> > > code!<o:p></o:p> > ><o:p></o:p> > > For this file "rte_sched.c", your patch should contain just two additional run-<o:p></o:p> > > time functions, i.e. the non-tc-ov version of functions<o:p></o:p> > grinder_credits_update()<o:p></o:p> > > and grindler_credits_check(), and the small code required to test when to use<o:p></o:p> > > the tc-ov vs. the non-tc_ov version, makes sense?<o:p></o:p> <o:p> </o:p> @Marcin: Yes, except setting tc ov enabled initially. Right?<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > > s->n_pipe_profiles = params->n_pipe_profiles;<o:p></o:p> > > > s->n_max_pipe_profiles = params->n_max_pipe_profiles;<o:p></o:p> > > ><o:p></o:p> > > > + /* TC over-subscription is disabled by default */<o:p></o:p> > > > + s->is_tc_ov_enabled = 0;<o:p></o:p> > > > +<o:p></o:p> > ><o:p></o:p> > > By default, this feature should be enabled:<o:p></o:p> > > s->is_tc_ov_enabled = 1;<o:p></o:p> > ><o:p></o:p> @Marcin: as below and here, ok feature is going to be enabled<o:p></o:p> <o:p> </o:p> > > ><o:p></o:p> > > > - if (!grinder_credits_check(port, subport, pos))<o:p></o:p> > > > - return 0;<o:p></o:p> > > > + switch (subport->is_tc_ov_enabled) {<o:p></o:p> > > > + case 1:<o:p></o:p> > > > + if (!grinder_credits_check_with_tc_ov(port, subport, pos))<o:p></o:p> > > > + return 0;<o:p></o:p> > > > + break;<o:p></o:p> > > > + case 0:<o:p></o:p> > > > + if (!grinder_credits_check(port, subport, pos))<o:p></o:p> > > > + return 0;<o:p></o:p> > > > + break;<o:p></o:p> > > > + }<o:p></o:p> > ><o:p></o:p> > > There should be no switch statement here, please replace with an if<o:p></o:p> > statement. I<o:p></o:p> > > suggest the following:<o:p></o:p> > ><o:p></o:p> > > int status;<o:p></o:p> > ><o:p></o:p> > > status = subport->tc_ov_enabled ? grinder_credits_check_with_tc_ov(port,<o:p></o:p> > > subport, pos) : grinder_credits_check(port, subport, pos); if (!status)<o:p></o:p> > > return 0;<o:p></o:p> > ><o:p></o:p> @Marcin: I disagree on that. Performance is decreased by that. Well any other thing than switch-case statement used there.<o:p></o:p> With code above I get:<o:p></o:p> -------+------------+------------+<o:p></o:p> | received | dropped |<o:p></o:p> -------+------------+------------+<o:p></o:p> RX | 5390751 | 0 |<o:p></o:p> QOS+TX | 5390784 | 0 | pps: 5390784<o:p></o:p> -------+------------+------------+<o:p></o:p> <o:p> </o:p> With switch case:<o:p></o:p> pps: 6733376<o:p></o:p> <o:p> </o:p> With TC oversubscription disabled or enabled, to my surprise. What to do with this now? I will be sending v5 patch with switch-case as of now ….<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > > /* Advance port time */<o:p></o:p> > > > port->time += pkt_len;<o:p></o:p> > > > @@ -2770,7 +2891,11 @@ grinder_handle(struct rte_sched_port *port,<o:p></o:p> > > > subport->profile;<o:p></o:p> > > ><o:p></o:p> > > > grinder_prefetch_tc_queue_arrays(subport, pos);<o:p></o:p> > > > - grinder_credits_update(port, subport, pos);<o:p></o:p> > > > +<o:p></o:p> > > > + if (unlikely(subport->is_tc_ov_enabled))<o:p></o:p> > ><o:p></o:p> > > Please remove the "unlikely" from here, don't put any likely/unlikely here at<o:p></o:p> > all.<o:p></o:p> <o:p> </o:p> @Marcin: Done. I didn’t not like “unlikely” from a first saw there.<o:p></o:p> <o:p> </o:p> > > > @@ -579,6 +579,24 @@ rte_sched_port_enqueue(struct rte_sched_port<o:p></o:p> > > > *port, struct rte_mbuf **pkts, uint int<o:p></o:p> > > > rte_sched_port_dequeue(struct rte_sched_port *port, struct rte_mbuf<o:p></o:p> > > > **pkts, uint32_t n_pkts);<o:p></o:p> > > ><o:p></o:p> > > > +/**<o:p></o:p> > > > + * Hierarchical scheduler subport TC OV enable/disable config.<o:p></o:p> > ><o:p></o:p> > > The name of the feature should be fully stated here: traffic class<o:p></o:p> > > oversubscription, not the abbreviation, please change.<o:p></o:p> <o:p> </o:p> @Marcin: done<o:p></o:p> <o:p> </o:p> > > > + * Note that this function is safe to use at runtime<o:p></o:p> > > > + * to enable/disable TC OV for subport.<o:p></o:p> > ><o:p></o:p> > > We should actually forbit this rather than encourage it. Calling this function<o:p></o:p> > > several times does not make sense, and it can create limitations that can<o:p></o:p> > come<o:p></o:p> > > back and byte us in the future, whenever we might need to extend this code,<o:p></o:p> > for no reason.<o:p></o:p> > ><o:p></o:p> > > Please actually replace with: "This function should be called at the time of<o:p></o:p> > > subport initialization."<o:p></o:p> > ><o:p></o:p> <o:p> </o:p> @Marcin: done<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > > > + # added in 22.03<o:p></o:p> > ><o:p></o:p> > > This is not in 22.03, it will hopefully be in 22.07.<o:p></o:p> > ><o:p></o:p> <o:p> </o:p> @Marcin: done<o:p></o:p> <o:p> </o:p> <o:p> </o:p> > Yes, I agree this would be the ideal way to drive this change, but the problem is that modifying the existing subport parameter structure would represent an API change. This would require a deprecation notice, and the patch would be blocked until 22.11 release. Are you willing to wait until 22.11? If not, then adding the configuration function for this flag is the next best thing.<o:p></o:p> <o:p> </o:p> Are we making any plans for that?<o:p></o:p> <o:p> </o:p> No, definitely not.<o:p></o:p> <o:p> </o:p> > > Also the name of the feature should not be abbreviated in the patch title.<o:p></o:p> > > In original code, this feature has always been disabled as it impacts<o:p></o:p> >> performance.<o:p></o:p> > > So, in my opinion we should keep it disabled by default and let user enable it<o:p></o:p> > > when required.<o:p></o:p> > > <o:p></o:p> <o:p> </o:p> > In the original code, yes, it had to be explicitly enabled through a build-time flag. This was not the best option, and this is precisely what we are trying to fix with this patch.<o:p></o:p> <o:p> </o:p> > But on the other hand all the users of these library that I know use it with the TC oversubscription turned on. Functionality is more important for them than performance. Hence my vote now is to enable it by default; those users that prefer performance over functionality can easily turn this feature off with no issues.<o:p></o:p> <o:p> </o:p> @Marcin: OK, so we enable it by default.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Intel Research and Development Ireland Limited<o:p></o:p> Registered in Ireland<o:p></o:p> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare<o:p></o:p> Registered Number: 308263<o:p></o:p> <o:p> </o:p> <o:p> </o:p> This e-mail and any attachments may contain confidential material for the sole<o:p></o:p> use of the intended recipient(s). Any review or distribution by others is<o:p></o:p> strictly prohibited. If you are not the intended recipient, please contact the<o:p></o:p> sender and delete all copies.<o:p></o:p> Stephen HemmingerMay 24, 2022, 2:52 p.m. UTC | #6<o:p></o:p> On Tue, 24 May 2022 13:33:31 +0000<o:p></o:p> Marcin Danilewicz <<a href="mailto:marcinx.danilewicz@intel.com">marcinx.danilewicz@intel.com</a>> wrote:<o:p></o:p> <o:p> </o:p> > /Marcin<o:p></o:p> > --------------------------------------------------------------<o:p></o:p> > Intel Research and Development Ireland Limited<o:p></o:p> > Registered in Ireland<o:p></o:p> > Registered Office: Collinstown Industrial Park, Leixlip, County Kildare<o:p></o:p> > Registered Number: 308263<o:p></o:p> > <o:p></o:p> > <o:p></o:p> > This e-mail and any attachments may contain confidential material for the sole<o:p></o:p> > use of the intended recipient(s). Any review or distribution by others is<o:p></o:p> > strictly prohibited. If you are not the intended recipient, please contact the<o:p></o:p> > sender and delete all copies.<o:p></o:p> > <o:p></o:p> <o:p> </o:p> <o:p> </o:p> Please talk to your management/lawyers. This kind of auto-footer violates the<o:p></o:p> required discussion properties of open source.<o:p></o:p> <o:p> </o:p> @Marcin: I send answer in separate thread. But in a short, that message was added by mail server or something in between. Patch using git email engine was sent as per usual. <o:p></o:p> <o:p> </o:p> <o:p> </o:p> BR,<o:p></o:p> /Marcin<o:p></o:p> </div> -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. </body> </html>