[PATCH v2 02/10] app/test: fix undefined behaviour in red autotest

[Marat Khalili]

> -----Original Message-----
> From: Bruce Richardson <bruce.richardson at intel.com>
> Sent: Thursday 4 December 2025 18:21
> To: dev at dpdk.org
> Cc: Bruce Richardson <bruce.richardson at intel.com>; stable at dpdk.org
> Subject: [PATCH v2 02/10] app/test: fix undefined behaviour in red autotest
> 
> The shift of a negative number (or very large positive) is undefined
> behaviour which causes errors when run with UBSan. Fix this by making
> the behaviour explicit for the edge case of n being zero in the
> calculation.
> 
> Fixes: de3cfa2c9823 ("sched: initial import")
> Cc: stable at dpdk.org
> 
> Signed-off-by: Bruce Richardson <bruce.richardson at intel.com>
> ---
>  lib/sched/rte_red.h | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/sched/rte_red.h b/lib/sched/rte_red.h
> index e62abb9295..3b90cc46a9 100644
> --- a/lib/sched/rte_red.h
> +++ b/lib/sched/rte_red.h
> @@ -172,8 +172,15 @@ __rte_red_calc_qempty_factor(uint8_t wq_log2, uint16_t m)
>  	f = (n >> 6) & 0xf;
>  	n >>= 10;
> 
> -	if (n < RTE_RED_SCALING)
> +	if (n < RTE_RED_SCALING) {
> +		/* When n == 0, no rounding or shifting needed.
> +		 * For n > 0, add 2^(n-1) for rounding before right shift.
> +		 * This avoids UB from (1 << -1) when n == 0.
> +		 */
> +		if (n == 0)
> +			return (uint16_t) rte_red_pow2_frac_inv[f];
>  		return (uint16_t) ((rte_red_pow2_frac_inv[f] + (1 << (n - 1))) >> n);
> +	}
> 
>  	return 0;
>  }
> --
> 2.51.0

Acked-by: Marat Khalili <marat.khalili at huawei.com>