<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> <a name="_MailEndCompose">The issue can be resolved by allocating the mb_mgr only for the primary process in the PMD probe/create function of the aesni_mb_pmd. <o:p></o:p></a> Disabling the scanning/probe is just a work around, I guess that is the reason why the fix in this <a href="https://urldefense.com/v3/__https:/review.spdk.io/gerrit/c/spdk/dpdk/*/1056__;Kw!!ACWV5N9M2RV99hQ!YF30Qsfu_2n30Fh2UhUdVH-1-72sWzP0kFWqtxkp1w3jzijsVGrk6w6v8C1gKCMGq9Bu$">https://review.spdk.io/gerrit/c/spdk/dpdk/+/1056</a> is canceled as the latest DPDK of 21.11 redesigned this PMD. The mb_mgr is removed from the device data and put to the each creation of the crypto session (wondering it has performance degradation). <o:p></o:p> <o:p> </o:p> <div> <o:p> </o:p> Thanks,<o:p></o:p> Changchun (Alex)<o:p></o:p> </div> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: Changchun Zhang [mailto:changchun.zhang@oracle.com] Sent: Wednesday, February 2, 2022 11:08 AM To: users@dpdk.org; dev@dpdk.org Subject: [External] : cryto_aesni_mb device data contaminated and causing crash when supporting vdev_scan/vdev_action <o:p></o:p> </div> </div> <o:p> </o:p> <div> <div> Hi, <o:p></o:p> <o:p> </o:p> Has anyone noticed that crypto_aesni_mb virtual crypto device has issue of memory crash caused by the scanning and probe on secondary process. Can anyone cast any lights on it.<o:p></o:p> What I encountered is: <o:p></o:p> On the primary process, the crypto_aesni_mb device is probed and created successfully and I got the mb_mgr set in the device private data. But during the packet process, the application crashes on accessing the mb_mgr. The deugging shows this mb_mgr address has been changed to an invalid address (non-NULL). Further digging shows this memory contamination occurs after the vdev_action replies the scan request. <o:p></o:p> In below code, the crash is gone by either disable sending message on VDEV_SCAN_REQ or skip processing the VDEV_SCAN_ONE. It seems the insert_vdev() on secondary process triggers another probe and break the existing device data?<o:p></o:p> It is also noticed there was an issue which was fixed by this patch <a href="https://urldefense.com/v3/__https:/review.spdk.io/gerrit/c/spdk/dpdk/*/1056__;Kw!!ACWV5N9M2RV99hQ!YF30Qsfu_2n30Fh2UhUdVH-1-72sWzP0kFWqtxkp1w3jzijsVGrk6w6v8C1gKCMGq9Bu$"> https://review.spdk.io/gerrit/c/spdk/dpdk/+/1056</a> but this patch is cancelled. This patch was complaining the similar memory issue found during scanning and probing on the secondary process. <o:p></o:p> <o:p> </o:p> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse"> <tbody> <tr> <td width="612" valign="top" style="width:688.1pt;border:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt"> static int<o:p></o:p> vdev_action(const struct rte_mp_msg *mp_msg, const void *peer)<o:p></o:p> {<o:p></o:p> struct rte_vdev_device *dev;<o:p></o:p> struct rte_mp_msg mp_resp;<o:p></o:p> struct vdev_param *ou = (struct vdev_param *)&mp_resp.param;<o:p></o:p> const struct vdev_param *in = (const struct vdev_param *)mp_msg->param;<o:p></o:p> const char *devname;<o:p></o:p> int num;<o:p></o:p> int ret;<o:p></o:p> <o:p> </o:p> strlcpy(mp_resp.name, VDEV_MP_KEY, sizeof(mp_resp.name));<o:p></o:p> mp_resp.len_param = sizeof(*ou);<o:p></o:p> mp_resp.num_fds = 0;<o:p></o:p> <o:p> </o:p> switch (in->type) {<o:p></o:p> case VDEV_SCAN_REQ:<o:p></o:p> VDEV_LOG(INFO, "changczh skip vdev, %s", devname);<o:p></o:p> ou->type = VDEV_SCAN_ONE;<o:p></o:p> ou->num = 1;<o:p></o:p> num = 0;<o:p></o:p> <o:p> </o:p> rte_spinlock_recursive_lock(&vdev_device_list_lock);<o:p></o:p> TAILQ_FOREACH(dev, &vdev_device_list, next) {<o:p></o:p> devname = rte_vdev_device_name(dev);<o:p></o:p> if (strlen(devname) == 0) {<o:p></o:p> VDEV_LOG(INFO, "vdev with no name is not sent");<o:p></o:p> continue;<o:p></o:p> }<o:p></o:p> VDEV_LOG(INFO, "send vdev, %s", devname);<o:p></o:p> strlcpy(ou->name, devname, RTE_DEV_NAME_MAX_LEN);<o:p></o:p> if (rte_mp_sendmsg(&mp_resp) < 0)<o:p></o:p> VDEV_LOG(ERR, "send vdev, %s, failed, %s",<o:p></o:p> devname, strerror(rte_errno));<o:p></o:p> num++;<o:p></o:p> }<o:p></o:p> rte_spinlock_recursive_unlock(&vdev_device_list_lock);<o:p></o:p> ou->type = VDEV_SCAN_REP;<o:p></o:p> ou->num = num;<o:p></o:p> if (rte_mp_reply(&mp_resp, peer) < 0)<o:p></o:p> VDEV_LOG(ERR, "Failed to reply a scan request");<o:p></o:p> break;<o:p></o:p> case VDEV_SCAN_ONE:<o:p></o:p> VDEV_LOG(INFO, "receive vdev, %s", in->name);<o:p></o:p> ret = insert_vdev(in->name, NULL, NULL, false); <o:p></o:p> if (ret == -EEXIST)<o:p></o:p> VDEV_LOG(DEBUG, "device already exist, %s", in->name);<o:p></o:p> else if (ret < 0)<o:p></o:p> VDEV_LOG(ERR, "failed to add vdev, %s", in->name);<o:p></o:p> break;<o:p></o:p> default:<o:p></o:p> VDEV_LOG(ERR, "vdev cannot recognize this message");<o:p></o:p> }<o:p></o:p> <o:p> </o:p> return 0;<o:p></o:p> }<o:p></o:p> </td> </tr> </tbody> </table> <o:p> </o:p> <o:p> </o:p> Thanks,<o:p></o:p> Alex<o:p></o:p> </div> </div> </div> </body> </html>