[dpdk-dev] [dpdk-stable] [PATCH 2/2] app/testpmd: fix invalid port detaching

Ferruh Yigit ferruh.yigit at intel.com
Thu Jan 23 19:14:06 CET 2020


On 1/23/2020 3:29 PM, Matan Azrad wrote:
> 
> Hi
> 
> From: Ferruh Yigit
>> On 1/23/2020 2:05 PM, Matan Azrad wrote:
>>> Hi
>>>
>>> From: Yigit, Ferruh
>>>> On 11/12/2019 8:47 AM, Matan Azrad wrote:
>>>>> The port was not validated before detaching.
>>>>>
>>>>> Ignore port detach operation when the port is not valid.
>>>>>
>>>>> Fixes: f8e5baa2662d ("app/testpmd: check not detaching device
>>>>> twice")
>>>>> Cc: thomas at monjalon.net
>>>>> Cc: stable at dpdk.org
>>>>>
>>>>> Signed-off-by: Matan Azrad <matan at mellanox.com>
>>>>> ---
>>>>>  app/test-pmd/testpmd.c | 3 +++
>>>>>  1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index
>>>>> 4444346..370eefe 100644
>>>>> --- a/app/test-pmd/testpmd.c
>>>>> +++ b/app/test-pmd/testpmd.c
>>>>> @@ -2545,6 +2545,9 @@ struct extmem_param {
>>>>>
>>>>>  	printf("Removing a device...\n");
>>>>>
>>>>> +	if (port_id_is_invalid(port_id, ENABLED_WARN))
>>>>> +		return;
>>>>> +
>>>>>  	dev = rte_eth_devices[port_id].device;
>>>>>  	if (dev == NULL) {
>>>>>  		printf("Device already removed\n");
>>>>>
>>>>
>>>> The patch is already in 19.11 [1] but it is breaking the testpmd
>>>> hotplug support.
>>>> Before 'detach_port_device()' called, the port has been stopped and
>>>> closed [2], which will make port fail from 'port_id_is_invalid()'
>>>> check and the device removal path never fully called.
>>>> The implication is, since device not detached, vfio request interrupt
>>>> keeps triggered continuously and re-starts the detach path, but
>>>> because of the half cleaned device it fails and app gets stuck with a
>> continuous log [3].
>>>>
>>>> I wonder if the actual hotplug has been tested with this patch, the
>>>> commit log is not clear about the motivation and implication of the
>>>> patch, I am not clear why this check is added but I am sending a
>>>> patch soon to remove it back.
>>>
>>> The motivation of this patch was to prevent double detach on same port,
>> so the user cannot call detach of invalid port.
>>
>> What is the definition of the 'invalid port', if you mean device already
>> detached case, in the second call of the function "if (dev == NULL)" check
>> should prevent it going forward.
> 
> No, ethdev doesn't zero the device pointer when it release a port.

As far as I can see it does, please see below.

> So even if the port is in unused state already - means invalid, the device pointer still may be valid and point to the last port that used the same id.

If the port is closed, it is unused state, and ethdev layer resources freed but
as you said device related structures are still there, device pointer is still
valid and it is still in probed device list etc.. We need to able to detach the
device even after it is unused state.

"stop -> close -> detach" is a normal order, we shouldn't prevent it, but your
check does prevent it.

I am not very clear about your concern here, "point to the last port that used
the same id", can you please clarify?

> 
> 
>> But according the 'port_id_is_invalid()' API, a closed port is an invalid port, I
>> think that is wrong in this context.
> 
> Why?

Closed port is 'invalid' for using it, because ethdev resources are freed. But
it is not 'invalid' to detach it, why a port being closed should prevent freeing
its device layer resources?

> 
> You are going to look on ethdev portid structure, don't you think we should valid the port before using its structure?

Is your main concern "rte_eth_devices[port_id].device" can be dangling pointer?

1) It is not.
2) The check you added to replace it is not correct check.

> 
> 
>>>
>>> I agree this patch is not good and we need a fix but I think the bug is
>> conceptual.
>>>
>>> Testpmd tries to do detach by port_id which is derived by ethdev port id
>> while detach work with rte_device.
>>>
>>> For example:
>>> you can see in the line above after +++: dev =
>>> rte_eth_devices[port_id].device, Testpmd may access invalid  or
>> reallocated ethdev structure to get the device name and may even detach
>> unwanted rte_device.
>>
>> I thinks whichever function calling 'detach_port_device()' should check the
>> port validity.
>> 'detach_port_device()' doesn't know if port reallocated or not, it will free the
>> given port_id, and when freeing done 'rte_eth_devices[port_id].device' will
>> be NULL, this looks to me a valid check.
> 
> Please validate me, check ethdev, I don't think so, 'rte_eth_devices[port_id].device still valid after detach.

This is a long stack trace, but what happens is:

rte_dev_remove
  bus unpug
    driver remove
      rte_eth_dev_pci_release
        eth_dev->device = NULL;

Please check the driver you are testing remove() ops (rte_pci_driver.remove())
does cleans the ethdev fields.

A little more detailed stack trace for my environment:
#0  rte_eth_dev_pci_release (eth_dev=..) at  rte_ethdev_pci.h:143
#1  rte_eth_dev_pci_generic_remove (pci_dev=.., dev_uninit=..) at
rte_ethdev_pci.h:199
#2  eth_i40e_pci_remove (pci_dev=..) at i40e_ethdev.c:710
#3  rte_pci_detach_dev (dev=..) at pci_common.c:243
#4  pci_unplug (dev=..) at pci_common.c:537
#5  local_dev_remove (dev=..) at eal_common_dev.c:321
#6  rte_dev_remove (dev=..) at eal_common_dev.c:402
#7  detach_port_device (port_id=0) at testpmd.c:2663
#8  cmd_operate_detach_port_parsed (parsed_result=.., cl=.., data=0x0) at
cmdline.c:1501
#9  cmdline_parse (cl=.., buf=.."port detach 0\n") at cmdline_parse.c:295
#10 cmdline_valid_buffer (rdl=.., buf="port detach 0\n", size=15) at  cmdline.c:31
#11 rdline_char_in (rdl=.., c=10 '\n') at  cmdline_rdline.c:421
#12 cmdline_in (cl=.., buf=.."\n", size=1) at cmdline.c:148
#13 cmdline_interact (cl=..) at cmdline.c:227
#14 prompt () at cmdline.c:19644
#15 main (argc=3, argv=..) at testpmd.c:3617

> 
>> The caller of the 'detach_port_device()' should ensure correct port_id
>> passed to the function.
> 
> What is correct port id, if the port was released , is it correct?

You are right, there is no good answer for it, I was thinking application state
information can be used but no ethdev should able to provide this information,
we need 'is_freed' kind of check for it, currently
'rte_eth_devices[port_id].device' is used for that purpose.

> 
>>>
>>> So, detach is broken with and without this patch.
>>
>> I can't see how it is broken without the check, how the problem you
>> mentioned can be reproduced? Or is it a theoretical issue?
>> But with this check hotplug support is %100 reproducible broken.
>>
>>>
>>>
>>> I think Testpmd should change the concept of rte_device mapping and put
>> attention to next:
>>> 1. Don't detach by ethdev port ID.
>>> 2. Multiple ethdev port IDs may related to the same rte_device.
>>>
>>> The Testpmd user should be sure that all the port IDs of the rte_device are
>> released before the detach call and Testpmd maybe need to validate it.
>>> And like attach, detach should be triggered by PCI address \ rte_device
>> name.
>>>
>>
>> We need to know about port_id too to be able to stop/close it.
>> And sure no objection to improve the hotplug support but it is broken now,
>> lets fix it first.
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Regards,
>>>> ferruh
>>>>
>>>>
>>>> [1]
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
>>>> .dp
>>>>
>> dk.org%2Fdpdk%2Fcommit%2F%3Fid%3D43d0e304980a1527bcac92dc679057
>>>>
>> b189e2545a&data=02%7C01%7Cmatan%40mellanox.com%7Cc3f40356d
>>>>
>> d124e20faf708d7a006e68c%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7
>>>>
>> C0%7C637153823809699996&sdata=dBy9m%2BxCA%2Bme1IpX2LqPARa
>>>> 62giznKi8Xbtu220GA%2Bg%3D&reserved=0
>>>>
>>>> [2]
>>>> rmv_port_callback
>>>>   stop_port(port_id);
>>>>   close_port(port_id);
>>>>   detach_port_device(port_id);
>>>>
>>>> [3]
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> EAL: can not get port by device 0000:00:05.0!
>>>> ...
> 



More information about the dev mailing list