[dpdk-dev] 答复: 答复: [PATCH v2] bus/pci: resolve multiple NICs address conflicts

[Gavin Hu (Arm Technology China)]

Hi Eric,

> -----Original Message-----
> From: dev <dev-bounces at dpdk.org> On Behalf Of Wangyu (Eric)
> Sent: Monday, November 11, 2019 5:38 PM
> To: Burakov, Anatoly <anatoly.burakov at intel.com>; David Marchand
> <david.marchand at redhat.com>
> Cc: dev at dpdk.org; ferruh.yigit at intel.com; Linuxarm
> <linuxarm at huawei.com>; humin (Q) <humin29 at huawei.com>; Liyuan
> (Larry) <Larry.T at huawei.com>; dengxiaofeng <dengxiaofeng at huawei.com>
> Subject: [dpdk-dev] 答复: 答复: [PATCH v2] bus/pci: resolve multiple NICs
> address conflicts
> 
> 
> Sorry, I didn't explain it clearly, and I will explain this problem step by step.
> 
> Precondition, we have a 64K page size system and two 82599 NICs. The
> memory required for each NIC is as follows:
> Map0 : size = 0x0000000000400000
> Map1 : size = 0x0000000000004000
> 
> 1. Primary process start, process mmap() first NIC map0, mmap()'s input
> address is 0x8202000000, and output address is 0x8202000000, size is
> 0x0000000000400000, next_addr is 0x8202400000, mmap() executed
> correctly.
> 
> 2. Primary mmap() first NIC map1, mmap()'s input address is 0x8202400000,
> and output address is 0x8202400000, size is 0x0000000000004000,
> next_addr is 0x8202404000, now mmap() applied from 0x8202400000 to
> 0x8202410000 actually(because page size is 64K), and next_addr is
> 0x8202404000, but mmap() executed correctly.
So the problem begins to happen here, next_addr should be equal to: 0x8202400000 + 64K(other than 16K) = 0x8202410000, taking account of the real mapping size(page size).  
> 
> 3. Primary mmap() second NIC map0, mmap()'s input address is
> 0x8202404000, but it's conflict, so output address is 0xffffbcdc0000(system
> assigned), size is 0x0000000000400000, next_addr is 0xffffbd1c0000, now
> the address is abnormal, and mmap() executed correctly.
If the step 2) is correct with the next_addr, this step should go on correctly with addresses.
> 
> 4. Primary mmap() second NIC map1, mmap()'s input address is
> 0xffffbd1c0000, and output address is 0xffffbcdb0000 (system assigned),
> size is 0x0000000000004000, now the address is abnormal, and mmap()
> executed correctly.
> 
> 5. Secondary process start, process mmap() first NIC map0, it's normal.
> 
> 6. Secondary process mmap() first NIC map1, it's normal.
> 
> 7. Secondary process mmap() second NIC map0, mmap()'s input address is
> 0xffffbcdc0000, but it's conflict on secondary process, so we get another
> address, but secondary will check if the input address is equal with output
> address, it's not equal, so secondary will exit with " Cannot mmap device
> resource file %s to address: %p ".
> 
> 
> Now I use (next_addr = RTE_PTR_ALIGN(current.addr + current.len,
> pagesize)) to solve the problem, and it worked. If it is right, I will submit a
> patch later.
> 
> By the way, I made a mistake, the problem won't happen on VFIO, because
> VFIO don't apply for 16K memory, only apply for 4M size(map0).But I think
> VFIO also needs to be modified.
> 
> 
> -----邮件原件-----
> 发件人: Burakov, Anatoly [mailto:anatoly.burakov at intel.com]
> 发送时间: 2019年11月7日 20:25
> 收件人: Wangyu (Eric) <seven.wangyu at huawei.com>; David Marchand
> <david.marchand at redhat.com>
> 抄送: dev at dpdk.org; ferruh.yigit at intel.com; Linuxarm
> <linuxarm at huawei.com>; humin (Q) <humin29 at huawei.com>; Liyuan
> (Larry) <Larry.T at huawei.com>; dengxiaofeng <dengxiaofeng at huawei.com>
> 主题: Re: 答复: [dpdk-dev] [PATCH v2] bus/pci: resolve multiple NICs
> address conflicts
> 
> On 07-Nov-19 5:44 AM, Wangyu (Eric) wrote:
> > Hi, Anatoly
> >
> > Thank you for advices. This problem will happen in both VFIO and UIO, I
> will modify both according to your advices and test them.
> >
> > I did some tests with mmap() on my system, when I provided address not
> page-aligned, mmap() could return page-aligned address too, but the code
> will return fault because mmap() return address was not equal with address
> I provided(problem occurs in pci_uio_map_secondary()).
> >
> 
> I still don't understand how do you get addresses aligned on a 16K boundary
> with 64K page size.
> 
> The mapping process is as follows:
> 
> 0) start with max_va_end, or with previous addres + previous len
> 1) reserve virtual area with mmap() (accepts any return address)
> 2) map the BAR with MAP_FIXED (checks return address, but should work
> because we already have that area reserved)
> 
> The error you're referring to would've happened at step 2 (MAP_FIXED with
> unaligned addresses will cause the mmap() to fail), but at that point we
> already have a valid virtual area for the bar. If you get a 16K-aligned page
> address for the BAR, you get it on step 1, not step 2.
> 
> So, if, by your own admission, your mmap() implementation does return a
> 64K-aligned address... What exactly is the issue then? How does your BAR
> end up with an invalid address?
> 
> --
> Thanks,
> Anatoly