[dpdk-dev] [PATCH 6/6] virtio: arm/arm64: memory mapped IO support in pmd driver

Santosh Shukla sshukla at mvista.com
Wed Dec 9 19:59:30 CET 2015


On Tue, Dec 8, 2015 at 6:23 PM, Santosh Shukla <sshukla at mvista.com> wrote:
>
>
> On Mon, Dec 7, 2015 at 10:38 PM, Stephen Hemminger
> <stephen at networkplumber.org> wrote:
>>
>> On Fri,  4 Dec 2015 23:05:19 +0530
>> Santosh Shukla <sshukla at mvista.com> wrote:
>>
>> >
>> > +#ifdef RTE_EXEC_ENV_LINUXAPP
>> > +/* start address of first pci_iobar slot (user-space virtual-addres) */
>> > +void *ioport_map;
>> > +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
>> > +
>> > +#include <sys/mman.h>
>> > +#define DEV_NAME             "/dev/igb_ioport"
>> > +
>> > +/* Keeping pci_ioport_size = 4k.
>> > + * So maximum mmaped pci_iobar supported =
>> > + * (ioport_size/pci_dev->mem_resource[0].len)
>> > + *
>> > + * note: kernel could allow maximum 32 virtio-net-pci interface, that
>> > mean
>> > + * maximum 32 PCI_IOBAR(s) where each PCI_IOBAR_LEN=0x20, so
>> > virtio_map_ioport()
>> > + * func by theory gonna support 4k/0x20 ==> 128 PCI_IOBAR(s), more than
>> > + * max-virtio-net-pci interface.
>> > + */
>> > +#define PAGE_SIZE            4096
>> > +#define PCI_IOPORT_SIZE              PAGE_SIZE
>> > +#define PCI_IOPORT_MAX               128 /* 4k / 0x20 */
>> > +
>> > +int ioport_map_cnt;
>> > +#endif /* ARM, ARM64 */
>> > +#endif /* RTE_EXEC_ENV_LINUXAPP */
>>
>> These variables should be static.
>>
>
> (Sorry for delayed follow, Was travelling..)
> Right,
>
>>
>> Also, it is should be possible to extract the I/O bar stuff in a generic
>> way through sysfs
>> and not depend on a character device. The long term goal for DPDK
>> acceptance is to
>> eliminate (or at least reduce to a minumum) any requirement for special
>> kernel drivers.
>
>
> I agree. Existing implementation does read pci_iobar for start address and
> size, But for non-x86 arch, we need someway to map pci_iobar and thats-why
> thought of adding device file for a purpose, as archs like arm lack iopl()
> privileged io syscall support, However iopl() too quite native driver design
> assumption.
>
> I have few idea in my mind such that - Right now I am updating ioport_mapped
> addr {kernel-virtual-addr-io-memory} to /sys/bus/pci/pci_bus_xxxx/xxx/map
> field, instead of mapping their, I'll try to map to uio's pci interface and
> then use existing pci_map_resource() api to mmap kernel-virtual-io-address
> to user-space-virtual-ioaddr. We'll come back on this.
>


Spent sometime digging dpdk's uio/pci source code, Intent was to map
pci ioport region via uio-way. In order to achieve that I tried to
hack the virtio-net-pci pmd driver. Right now in virtio-net-pci case,
It creates two sysfs entry for pci bars: resource0 /1.

Resource0; is ioport region
Resource1; is iomem region.

By appending a RTE_PCI_DRV_NEED_MAPPING flag to drv_flag and passing
hw->io_base = resource1 type pci.mem_resource[slot].addr; where slot
=1. Resource1 is IORESOURCE_MEM type so uio/pci driver able to mmap.
That way I could get the valid user-space virtual address. However
this hack did not worked for me because at qemu side: virtio-pxe.rom
has virtio_headers located at ioport pci region and guest driver
writing at iomem region, that's why driver init failed. Note that
default driver doesn't use resource1 memory.

This made me think that either I had to add dependent code in kernel
or something similar proposed in this patch.
It is because:
- uio driver and dependent user-space pci api's in dpdk mmaps
IORESOURCE_MEM types address only {refer igbuio_setup_bars() and in
particular function pci_parse_sysfs_resource()}.
- That mmap in userspace backed by arch specific api
pci_mmap_page_range() in kernel.
- pci_mmap_page_range() does not support mmaping to IO_RESOURCE_IO type memory.
- Having said that, we need routine or a way to to map pci_iobar
region from kernel virtual-address to user-space virtual address.

In x86 case; iopl() exports the ioport address to userspace. But for
non-x86 case, we need special device file to mmap ioport kernel
address space to user space.  Also I grepped dpdk source code little
more and noticed that xen dom0 implementation is quite similar too.
They are too using /dev/dom0_mem special device file.

IMO, there is approach-2 to achieve desired which I mentioned in
cover-letter too. By adding /dev/ioport device file support in kernel,
so that user could do byte/word/halfword rd/wr, rightnow kernel
support only byte long rd/wr (i.e.. /dev/port file driver/char/mem.c).
In that way; We care to do file open/rd/wr/close for /dev/ioport. No
low level extra arch level api introduced in this patch series. Pretty
clean code. But then we do have kernel device file dependency which is
a cons. I personally like this approach as we are not dependant on
kernel side driver, all desired dependency is in dpdk source base. I
am going to send out approach-2 patch series, perhaps in this week for
comment.

Pl. let me know if you have better approach in mind, Otherwise I am
planning to work on V2 patch revision, incorporating other review
comment.


More information about the dev mailing list