<div dir="ltr"><div dir="ltr"><p>Hi, <br><br>I hope this is the correct place to report these issues since it seems to be related to DPDK codes. I've reported this to Nvidia a few days ago but have yet to receive any response from them.<br><br>My server is currently using ConnectX5 MT27800 (mlx5_core 5.7-1.0.2) on firmware 16.35.4506 (MT_0000000011). My DPDK library version is 22.11.</p><p>I ran the following testpmd command which resulted in segmentation fault (I am currently running on filtered traffic with packets >1000 bytes to increase the odds of hitting the segmentation fault):</p><pre><code>dpdk-testpmd -l 1-5 -n 4 -a 0000:1f:00.0,rxq_comp_en=1,rxq_pkt_pad_en=1,rxqs_min_mprq=1,mprq_en=1,mprq_log_stride_num=6,mprq_log_stride_size=9,mprq_max_memcpy_len=64,rx_vec_en=1 -- -i --rxd=8192 --max-pkt-len=1700 --rxq=1 --total-num-mbufs=16384 --mbuf-size=3000 --enable_drop_en –enable_scatter</code></pre><p>This segmentation fault goes away when I disable vectorization (rx_vec_en=0). (Note that the segmentation fault does not occur in forward-mode=rxonly). The segmentation fault also seems to happen with higher chances when there is a rxnombuf.<br><br>Upon some investigation, I noticed that in DPDK’s source codes <code>drivers/net/mlx5/mlx5_rxtx_vec.c (function rxq_copy_mprq_mbuf_v())</code>, there is a possibility where the consumed stride exceeds the stride number (64 in this case) which should not be happening. I'm suspecting there's some CQE misalignment here upon encountering rxnombuf.</p><pre><code>rxq_copy_mprq_mbuf_v(...) {
...
if(rxq->consumed_strd == strd_n) {
// replenish WQE
}
...
strd_cnt = (elts[i]->pkt_len / strd_sz) +
((elts[i]->pkt_len % strd_sz) ? 1 : 0);
rxq_code = mprq_buf_to_pkt(rxq, elts[i], elts[i]->pkt_len, buf, rxq->consumed_strd, strd_cnt);
rxq->consumed_strd += strd_cnt; // encountering cases where rxq->consumed_strd > strd_n
...
}</code></pre><p>In addition, there were also cases in mprq_buf_to_pkt() where the allocated seg address is exactly the same as the pkt (elts[i]) address passed in which should not happen.<br></p><pre><code>mprq_buf_to_pkt(...) {
...
if(hdrm_overlap > 0) {
MLX5_ASSERT(rxq->strd_scatter_en);<pre><code> struct rte_mbuf *seg = rte_pktmbuf_alloc(rxq->mp);
if (unlikely(seg == NULL)) return MLX5_RXQ_CODE_NOMBUF;
SET_DATA_OFF(seg, 0);
// added debug statement
DRV_LOG(DEBUG, "pkt %p seg %p", (void *)pkt, (void *)seg);<br></code></pre> rte_memcpy(rte_pktmbuf_mtod(seg, void *), RTE_PTR_ADD(addr, len - hdrm_overlap), hdrm_overlap);
...
}
}</code></pre><p>I have tried upgrading my DPDK version to 24.11 but the segmentation fault still persists. <br><br>In addition, there were also a few other issues that I've noticed:<br></p><ul><li>max-pkt-len does not seem to work for values < 1500 even though "show port info X" showed that the MTU was set to the value I've passed in<br></li><li>In mprq_buf_to_pkt():<br> - uint32_t seg_len = RTE_MIN(len, (uint32_t)(pkt->buf_len - RTE_PKTMBUF_HEADROOM)) --> seems unnecessary as to hit this code, len has to be greater than (uint32_t)(pkt->buf_len - RTE_PKTMBUF_HEADROOM) due to the if condition<br> - If the allocation struct rte_mbuf *next = rte_pktmbuf_alloc(rxq->mp) fails and packet has more than 2 segs, the segs that were allocated previously do not get freed <br></li></ul><p></p><pre><code> mprq_buf_to_pkt(...) {
...
</code> } else if (rxq->strd_scatter_en) {</pre></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace"> struct rte_mbuf *prev = pkt;</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace"> uint32_t seg_len = RTE_MIN(len, (uint32_t)</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace"> (pkt->buf_len - RTE_PKTMBUF_HEADROOM));</font></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace"> uint32_t rem_len = len - seg_len;</font></div></blockquote><div><font face="monospace"><br></font></div><div><font face="monospace"> rte_memcpy(rte_pktmbuf_mtod(pkt, void *), </font><span style="font-family:monospace">addr, seg_len);</span></div><div><font face="monospace"> DATA_LEN(pkt) = seg_len;</font></div><div><font face="monospace"> while (rem_len) {</font></div><div><font face="monospace"> struct rte_mbuf *next =</font><span style="font-family:monospace"> rte_pktmbuf_alloc(rxq->mp);</span></div></blockquote><div dir="ltr"><font face="monospace"><br> if (unlikely(next == NULL))<br> return MLX5_RXQ_CODE_NOMBUF;<br> ...<br></font> - In the external buffer attach case where hdrm_overlap > 0, the code did not decrement the buffer refcnt if allocation struct rte_mbuf *next = rte_pktmbuf_alloc(rxq->mp) fails<br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div dir="ltr"><div><pre><code>mprq_buf_to_pkt(...) {
...
</code> if (hdrm_overlap > 0) {</pre><pre> __atomic_add_fetch(&buf->refcnt, 1, __ATOMIC_RELAXED);
...
MLX5_ASSERT(rxq->strd_scatter_en);
struct rte_mbuf *seg = rte_pktmbuf_alloc(rxq->mp);
if (unlikely(seg == NULL))
return MLX5_RXQ_CODE_NOMBUF;
SET_DATA_OFF(seg, 0);
...</pre></div></div></blockquote><div dir="ltr"><pre></pre><br>Hope to hear from you soon!<br><br>With regards,<br>Joni</div>
</div>