[dpdk-dev] [PATCH 2/2] ixgbe: check mbuf refcnt when clearing RX/TX ring

[Ananyev, Konstantin]

Hi Bruce,

> -----Original Message-----
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Bruce Richardson
> Sent: Friday, July 03, 2015 4:40 PM
> To: dev at dpdk.org
> Subject: [dpdk-dev] [PATCH 2/2] ixgbe: check mbuf refcnt when clearing RX/TX ring
> 
> The function to clear the TX ring when a port was being closed, e.g. on
> exit in testpmd, was not checking the mbuf refcnt before freeing it.
> Since the function in the vector driver to clear the ring after TX does
> not set the pointer to NULL post-free, this caused crashes if mbuf
> debugging was turned on.
> 
> To reproduce the issue, ensure the follow config variables are set:
> RTE_IXGBE_INC_VECTOR
> RTE_LIBRTE_MBUF_DEBUG
> Then compile up and run testpmd using 10G ports with the vector driver.
> Start traffic and let some flow through, then type "stop" and "quit" at
> the testpmd prompt, and crash will occur. Output below:
> 
> 	testpmd> quit
> 	Stopping port 0...done
> 	Stopping port 1...PANIC in rte_mbuf_sanity_check():
> 	bad ref cnt
> 	[New Thread 0x7fffabfff700 (LWP 145312)]
> 	[New Thread 0x7fffb47fe700 (LWP 145311)]
> 	[New Thread 0x7fffb4fff700 (LWP 145310)]
> 	[New Thread 0x7ffff6cd5700 (LWP 145309)]
> 	18: [/home/bruce/dpdk.org/x86_64-native-linuxapp-gcc/app/testpmd(_start+0x29)
> 	<....snip for brevity...>
> 	Program received signal SIGABRT, Aborted.
> 	0x00007ffff7120a98 in raise () from /lib64/libc.so.6
> 
> A similar error occurs when clearing the RX ring, which is also fixed by
> this patch.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson at intel.com>
> ---
>  drivers/net/ixgbe/ixgbe_rxtx.c     | 3 ++-
>  drivers/net/ixgbe/ixgbe_rxtx_vec.c | 8 +++++++-
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
> index 41a062e..12e25b7 100644
> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> @@ -2108,7 +2108,8 @@ ixgbe_rx_queue_release_mbufs(struct ixgbe_rx_queue *rxq)
> 
>  	if (rxq->sw_ring != NULL) {
>  		for (i = 0; i < rxq->nb_rx_desc; i++) {
> -			if (rxq->sw_ring[i].mbuf != NULL) {
> +			if (rxq->sw_ring[i].mbuf != NULL &&
> +					rte_mbuf_refcnt_read(rxq->sw_ring[i].mbuf)) {
>  				rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf);
>  				rxq->sw_ring[i].mbuf = NULL;
>  			}


Sorry for late review, but I am afraid your changes don't fix the problem.
After sw_ring[].mbuf was freed by RX path, entity that manages that RX queue shouldn't touch that mbuf.
(unless it was allocated  by the same RX queue again).
As same mbuf could be already allocated by something else.
As an example by another RX/TX queue and is in active use. 
Same story for TX below.

As I can see the proper fix could be one of 2:
1. Make RX/TX vector functions to reset sw_ring[].mbuf to 0.
2. At queue_release_mbufs(), don't go through all sw_ring[] entries, but only though ones
which might contain valid mbufs.
For RX: entries between rx_tail and rxrearm_start only
(which implies a special queue_release_mbufs() for vector rx).
For TX: from tx_next_dd - (tx_rs_thresh - 1) and no more then nb_tx_desc - nb_tx_free
  
Konstantin

> diff --git a/drivers/net/ixgbe/ixgbe_rxtx_vec.c b/drivers/net/ixgbe/ixgbe_rxtx_vec.c
> index 0edac82..7e633d3 100644
> --- a/drivers/net/ixgbe/ixgbe_rxtx_vec.c
> +++ b/drivers/net/ixgbe/ixgbe_rxtx_vec.c
> @@ -665,7 +665,13 @@ ixgbe_tx_queue_release_mbufs(struct ixgbe_tx_queue *txq)
>  		     nb_free < max_desc && i != txq->tx_tail;
>  		     i = (i + 1) & max_desc) {
>  			txe = (struct ixgbe_tx_entry_v *)&txq->sw_ring[i];
> -			if (txe->mbuf != NULL)
> +			/*
> +			 *check for already freed packets.
> +			 * Note: ixgbe_tx_free_bufs does not NULL after free,
> +			 * so we actually have to check the reference count.
> +			 */
> +			if (txe->mbuf != NULL &&
> +					rte_mbuf_refcnt_read(txe->mbuf) != 0)
>  				rte_pktmbuf_free_seg(txe->mbuf);
>  		}
>  		/* reset tx_entry */
> --
> 2.4.3