[PATCH v2 02/10] app/test: fix undefined behaviour in red autotest

[Bruce Richardson]

The shift of a negative number (or very large positive) is undefined
behaviour which causes errors when run with UBSan. Fix this by making
the behaviour explicit for the edge case of n being zero in the
calculation.

Fixes: de3cfa2c9823 ("sched: initial import")
Cc: stable at dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson at intel.com>
---
 lib/sched/rte_red.h | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/sched/rte_red.h b/lib/sched/rte_red.h
index e62abb9295..3b90cc46a9 100644
--- a/lib/sched/rte_red.h
+++ b/lib/sched/rte_red.h
@@ -172,8 +172,15 @@ __rte_red_calc_qempty_factor(uint8_t wq_log2, uint16_t m)
 	f = (n >> 6) & 0xf;
 	n >>= 10;
 
-	if (n < RTE_RED_SCALING)
+	if (n < RTE_RED_SCALING) {
+		/* When n == 0, no rounding or shifting needed.
+		 * For n > 0, add 2^(n-1) for rounding before right shift.
+		 * This avoids UB from (1 << -1) when n == 0.
+		 */
+		if (n == 0)
+			return (uint16_t) rte_red_pow2_frac_inv[f];
 		return (uint16_t) ((rte_red_pow2_frac_inv[f] + (1 << (n - 1))) >> n);
+	}
 
 	return 0;
 }
-- 
2.51.0