<div dir="ltr">Hello all,<div> </div><div>I never stop testing to see results. It has been 10 days. After patching, no leak.</div><div> </div><div>MBUF_POOL 82 10,317 0.79% [|....................] MBUF_POOL 83 10,316 0.80% [|....................] MBUF_POOL 93 10,306 0.89% [|....................] </div><div> </div><div>Sometimes, it takes time to get back to mempool. In my opinion, it is about the OVS-DPDK/openstack environment issue. If I have a chance, try to run an Intel Bare-metal environment.</div><div> </div><div>After meeting with Ferruh, he explained concerns about performance issues so I decided to continue manual patching for my application. </div><div> </div><div>It is removed from bugzilla.</div><div> </div><div>For your information.</div><div>Best regards.</div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">Ferruh Yigit <<a href="mailto:ferruh.yigit@amd.com">ferruh.yigit@amd.com</a>>, 19 May 2023 Cum, 21:43 tarihinde şunu yazdı: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 5/19/2023 6:47 PM, Yasin CANER wrote: > Hello, > Hi, Can you please bottom-post, combination of both makes discussion very hard to follow? > I tested all day both before and after patching. > > I could not understand that it is a memory leak or not. Maybe it needs > optimization. You lead, I follow. > > 1-) You are right, alloc_q is never bigger than 1024. But it always > allocates 32 units then more than 1024 are being freed. Maybe it takes > time, I don't know. > At least alloc_q is only freed on kni release, so mbufs in that fifo can sit there as long as application is running. > 2-) I tested tx_rs_thresh via ping. After 210 sec , allocated memories > are back to mempool (most of them). (driver virtio and eth-devices are > binded via igb_uio) . It really takes time. So it is better to increase > the size of the mempool. > (<a href="https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html</a> > <<a href="https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html</a>>) > > 3-) try to list mempool state in randomly > It looks number of mbufs used seems increasing, but in worst case both alloc_q and free_q can be full, which makes 2048 mbufs, and in below tests used mbufs number is not bigger than this value, so looks OK. If you run your test for a longer duration, do you observe that used mbufs going much above this number? Also what are the 'num' parameter to 'rte_kni_tx_burst()' API? If it is bigger than 'MAX_MBUF_BURST_NUM', that may lead mbufs accumulate at free_q fifo. As experiment, it is possible to decrease KNI fifo sizes, and observe the result. > Test -1 -) (old code) ICMP testing. The whole mempool size is about > 10350. So after FIFO reaches max-size -1024, %10 of the size of the > mempool is in use. But little by little memory is waiting in use and > doesn't go back to the pool. I could not find the reason. > > MBUF_POOL 448 9,951 > 4.31% [|....................] > MBUF_POOL 1,947 8,452 > 18.72% [||||.................] > MBUF_POOL 1,803 8,596 > 17.34% [||||.................] > MBUF_POOL 1,941 8,458 > 18.67% [||||.................] > MBUF_POOL 1,900 8,499 > 18.27% [||||.................] > MBUF_POOL 1,999 8,400 > 19.22% [||||.................] > MBUF_POOL 1,724 8,675 > 16.58% [||||.................] > MBUF_POOL 1,811 8,588 > 17.42% [||||.................] > MBUF_POOL 1,978 8,421 > 19.02% [||||.................] > MBUF_POOL 2,008 8,391 > 19.31% [||||.................] > MBUF_POOL 1,854 8,545 > 17.83% [||||.................] > MBUF_POOL 1,922 8,477 > 18.48% [||||.................] > MBUF_POOL 1,892 8,507 > 18.19% [||||.................] > MBUF_POOL 1,957 8,442 > 18.82% [||||.................] > > Test-2 -) (old code) run iperf3 udp testing that from Kernel to eth > device. Waited to see what happens in 4 min. memory doesn't go back to > the mempool. little by little, memory usage increases. > > MBUF_POOL 512 9,887 > 4.92% [|....................] > MBUF_POOL 1,411 8,988 > 13.57% [|||..................] > MBUF_POOL 1,390 9,009 > 13.37% [|||..................] > MBUF_POOL 1,558 8,841 > 14.98% [|||..................] > MBUF_POOL 1,453 8,946 > 13.97% [|||..................] > MBUF_POOL 1,525 8,874 > 14.66% [|||..................] > MBUF_POOL 1,592 8,807 > 15.31% [||||.................] > MBUF_POOL 1,639 8,760 > 15.76% [||||.................] > MBUF_POOL 1,624 8,775 > 15.62% [||||.................] > MBUF_POOL 1,618 8,781 > 15.56% [||||.................] > MBUF_POOL 1,708 8,691 > 16.42% [||||.................] > iperf is STOPPED to tx_fresh for 4 min > MBUF_POOL 1,709 8,690 > 16.43% [||||.................] > iperf is STOPPED to tx_fresh for 4 min > MBUF_POOL 1,709 8,690 > 16.43% [||||.................] > MBUF_POOL 1,683 8,716 > 16.18% [||||.................] > MBUF_POOL 1,563 8,836 > 15.03% [||||.................] > MBUF_POOL 1,726 8,673 > 16.60% [||||.................] > MBUF_POOL 1,589 8,810 > 15.28% [||||.................] > MBUF_POOL 1,556 8,843 > 14.96% [|||..................] > MBUF_POOL 1,610 8,789 > 15.48% [||||.................] > MBUF_POOL 1,616 8,783 > 15.54% [||||.................] > MBUF_POOL 1,709 8,690 > 16.43% [||||.................] > MBUF_POOL 1,740 8,659 > 16.73% [||||.................] > MBUF_POOL 1,546 8,853 > 14.87% [|||..................] > MBUF_POOL 1,710 8,689 > 16.44% [||||.................] > MBUF_POOL 1,787 8,612 > 17.18% [||||.................] > MBUF_POOL 1,579 8,820 > 15.18% [||||.................] > MBUF_POOL 1,780 8,619 > 17.12% [||||.................] > MBUF_POOL 1,679 8,720 > 16.15% [||||.................] > MBUF_POOL 1,604 8,795 > 15.42% [||||.................] > MBUF_POOL 1,761 8,638 > 16.93% [||||.................] > MBUF_POOL 1,773 8,626 > 17.05% [||||.................] > > Test-3 -) (after patching) run iperf3 udp testing that from Kernel to > eth device. looks stable. > After patching , > > MBUF_POOL 76 10,323 > 0.73% [|....................] > MBUF_POOL 193 10,206 > 1.86% [|....................] > MBUF_POOL 96 10,303 > 0.92% [|....................] > MBUF_POOL 269 10,130 > 2.59% [|....................] > MBUF_POOL 102 10,297 > 0.98% [|....................] > MBUF_POOL 235 10,164 > 2.26% [|....................] > MBUF_POOL 87 10,312 > 0.84% [|....................] > MBUF_POOL 293 10,106 > 2.82% [|....................] > MBUF_POOL 99 10,300 > 0.95% [|....................] > MBUF_POOL 296 10,103 > 2.85% [|....................] > MBUF_POOL 90 10,309 > 0.87% [|....................] > MBUF_POOL 299 10,100 > 2.88% [|....................] > MBUF_POOL 86 10,313 > 0.83% [|....................] > MBUF_POOL 262 10,137 > 2.52% [|....................] > MBUF_POOL 81 10,318 > 0.78% [|....................] > MBUF_POOL 81 10,318 > 0.78% [|....................] > MBUF_POOL 87 10,312 > 0.84% [|....................] > MBUF_POOL 252 10,147 > 2.42% [|....................] > MBUF_POOL 97 10,302 > 0.93% [|....................] > iperf is STOPPED to tx_fresh for 4 min > MBUF_POOL 296 10,103 > 2.85% [|....................] > MBUF_POOL 95 10,304 > 0.91% [|....................] > MBUF_POOL 269 10,130 > 2.59% [|....................] > MBUF_POOL 302 10,097 > 2.90% [|....................] > MBUF_POOL 88 10,311 > 0.85% [|....................] > MBUF_POOL 305 10,094 > 2.93% [|....................] > MBUF_POOL 88 10,311 > 0.85% [|....................] > MBUF_POOL 290 10,109 > 2.79% [|....................] > MBUF_POOL 84 10,315 > 0.81% [|....................] > MBUF_POOL 85 10,314 > 0.82% [|....................] > MBUF_POOL 291 10,108 > 2.80% [|....................] > MBUF_POOL 303 10,096 > 2.91% [|....................] > MBUF_POOL 92 10,307 > 0.88% [|....................] > > > Best regards. > > > Ferruh Yigit <<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a> <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a>>>, 18 > May 2023 Per, 17:56 tarihinde şunu yazdı: > > On 5/18/2023 9:14 AM, Yasin CANER wrote: > > Hello Ferruh, > > > > Thanks for your kind response. Also thanks to Stephen. > > > > Even if 1 packet is consumed from the kernel , each time rx_kni > > allocates another 32 units. After a while all mempool is used in > alloc_q > > from kni. there is not any room for it. > > > > What you described continues until 'alloc_q' is full, by default fifo > length is 1024 (KNI_FIFO_COUNT_MAX), do you allocate less mbuf in your > mempool? > > You can consider either increasing mempool size, or decreasing 'alloc_q' > fifo length, but reducing fifo size may cause performance issues so you > need to evaluate that option. > > > Do you think my mistake is using one and common mempool usage both kni > > and eth? > > > > Using same mempool for both is fine. > > > If it needs a separate mempool , i'd like to note in docs. > > > > Best regards. > > > > Ferruh Yigit <<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a> <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a>> > <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a> <mailto:<a href="mailto:ferruh.yigit@amd.com" target="_blank">ferruh.yigit@amd.com</a>>>>, 17 > > May 2023 Çar, 20:53 tarihinde şunu yazdı: > > > > On 5/9/2023 12:13 PM, Yasin CANER wrote: > > > Hello, > > > > > > I draw a flow via asciiflow to explain myself better. > Problem is after > > > transmitting packets(mbufs) , it never puts in the > kni->free_q to back > > > to the original pool. Each cycle, it allocates another 32 > units that > > > cause leaks. Or I am missing something. > > > > > > I already tried the rte_eth_tx_done_cleanup() function but it > > didn't fix > > > anything. > > > > > > I am working on a patch to fix this issue but I am not sure > if there > > > is another way. > > > > > > Best regards. > > > > > > <a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a> > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a>> > > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a> > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a>>> > > > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a> > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a>> > > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a> > <<a href="https://pastebin.ubuntu.com/p/s4h5psqtgZ/" rel="noreferrer" target="_blank">https://pastebin.ubuntu.com/p/s4h5psqtgZ/</a>>>> > > > > > > > > > unsigned > > > rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, > > unsigned > > > int num) > > > { > > > unsigned int ret = kni_fifo_get(kni->tx_q, (void **)mbufs, num); > > > > > > /* If buffers removed, allocate mbufs and then put them into > > alloc_q */ > > > /* Question, how to test buffers is removed or not?*/ > > > if (ret) > > > kni_allocate_mbufs(kni); > > > > > > return ret; > > > } > > > > > > > Selam Yasin, > > > > > > You can expect 'kni->alloc_q' fifo to be full, this is not a > memory > > leak. > > > > As you pointed out, number of mbufs consumed by kernel from > 'alloc_q' > > and number of mbufs added to 'alloc_q' is not equal and this is > > expected. > > > > Target here is to prevent buffer underflow from kernel > perspective, so > > it will always have available mbufs for new packets. > > That is why new mbufs are added to 'alloc_q' at worst same or > sometimes > > higher rate than it is consumed. > > > > You should calculate your mbuf requirement with the assumption > that > > 'kni->alloc_q' will be full of mbufs. > > > > > > 'kni->alloc_q' is freed when kni is removed. > > Since 'alloc_q' holds physical address of the mbufs, it is a > little > > challenging to free them in the userspace, that is why first > kernel > > tries to move mbufs to 'kni->free_q' fifo, please check > > 'kni_net_release_fifo_phy()' for it. > > > > If all moved to 'free_q' fifo, nothing left to in 'alloc_q', > but if not, > > userspace frees 'alloc_q' in 'rte_kni_release()', with > following call: > > `kni_free_fifo_phy(kni->pktmbuf_pool, kni->alloc_q);` > > > > > > I can see you have submitted fixes for this issue, although as I > > explained above I don't think a defect exist, I will review them > > today/tomorrow. > > > > Regards, > > Ferruh > > > > > > > Stephen Hemminger <<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a> > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> > > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a> > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>>> > > > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a> > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>> > > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a> > <mailto:<a href="mailto:stephen@networkplumber.org" target="_blank">stephen@networkplumber.org</a>>>>>, 8 May 2023 Pzt, 19:18 tarihinde > > > şunu yazdı: > > > > > > On Mon, 8 May 2023 09:01:41 +0300 > > > Yasin CANER <<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a> > <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a>> > > <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a> <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a>>> > <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a> <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a>> > > <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a> <mailto:<a href="mailto:yasinncaner@gmail.com" target="_blank">yasinncaner@gmail.com</a>>>>> > > > wrote: > > > > > > > Hello Stephen, > > > > > > > > Thank you for response, it helps me a lot. I > understand problem > > > better. > > > > > > > > After reading mbuf library ( > > > > > <a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a> > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a>> > > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a> > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a>>> > > > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a> > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a>> > > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a> > <<a href="https://doc.dpdk.org/guides/prog_guide/mempool_lib.html" rel="noreferrer" target="_blank">https://doc.dpdk.org/guides/prog_guide/mempool_lib.html</a>>>>) i > > > realized that > > > > 31 units allocation memory slot doesn't return to pool! > > > > > > If receive burst returns 1 mbuf, the other 31 pointers > in the > > array > > > are not valid. They do not point to mbufs. > > > > > > > 1 unit mbuf can be freed via rte_pktmbuf_free so it > can back > > to pool. > > > > > > > > Main problem is that allocation doesn't return to > original pool, > > > act as > > > > used. So, after following rte_pktmbuf_free > > > > > > > > > > <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a>>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a1215458932900b7cd5192326fa4a6902</a>>>>> > > > > function, > > > > i realized that there is 2 function to helps to mbufs back > > to pool. > > > > > > > > These are rte_mbuf_raw_free > > > > > > > > > > <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a>>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a9f188d53834978aca01ea101576d7432</a>>>>> > > > > and rte_pktmbuf_free_seg > > > > > > > > > > <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a>>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a>> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a> <<a href="http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37" rel="noreferrer" target="_blank">http://doc.dpdk.org/api/rte__mbuf_8h.html#a006ee80357a78fbb9ada2b0432f82f37</a>>>>>. > > > > I will focus on them. > > > > > > > > If there is another suggestion, I will be very pleased. > > > > > > > > Best regards. > > > > > > > > Yasin CANER > > > > Ulak > > > > > > </blockquote></div>