[dpdk-dev] [RFC 0/4] SocketPair Broker support for vhost and virtio-user.

[Ilya Maximets]

On 3/19/21 9:51 AM, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Mar 18, 2021 at 11:47 PM Ilya Maximets <i.maximets at ovn.org <mailto:i.maximets at ovn.org>> wrote:
> 
>     On 3/18/21 6:52 PM, Stefan Hajnoczi wrote:
>     > On Wed, Mar 17, 2021 at 09:25:26PM +0100, Ilya Maximets wrote:
>     > Hi,
>     > Some questions to understand the problems that SocketPair Broker solves:
>     >
>     >> Even more configuration tricks required in order to share some sockets
>     >> between different containers and not only with the host, e.g. to
>     >> create service chains.
>     >
>     > How does SocketPair Broker solve this? I guess the idea is that
>     > SocketPair Broker must be started before other containers. That way
>     > applications don't need to sleep and reconnect when a socket isn't
>     > available yet.
>     >
>     > On the other hand, the SocketPair Broker might be unavailable (OOM
>     > killer, crash, etc), so applications still need to sleep and reconnect
>     > to the broker itself. I'm not sure the problem has actually been solved
>     > unless there is a reason why the broker is always guaranteed to be
>     > available?
> 
>     Hi, Stefan.  Thanks for your feedback!
> 
>     The idea is to have the SocketPair Broker running right from the
>     boot of the host.  If it will use a systemd socket-based service
>     activation, the socket should persist while systemd is alive, IIUC.
>     OOM, crash and restart of the broker should not affect existence
>     of the socket and systemd will spawn a service if it's not running
>     for any reason without loosing incoming connections.
> 
> 
> Since the solution relies on systemd, why not use DBus to perform authentication, service discovery and setup the socketpair between peers? You don't need an extra service broker in this case.
> 
> When the org.foo service shows up, call org.foo.Connect() to return the fd of the client end (or throw an error etc)

Yes, that's a possible solution.  And I even thought about running
SocketPair Broker as a dbus service (it's in a 'todo' list with
a question mark).  However, there are few issues with DBus-based
solution:

1. I'd like to not bind the solution to systemd just because it's
   not always required.  There are cases where you don't really
   need a persistent and never-changing socket file.  For example,
   dpdk implementation for vhost and virtio-user in this patch set
   will work just fine, i.e. will re-connect as soon as socket is
   available.
   Also, socket-based activation is a cool feature, but it's not the
   only solution on how to make socket file appear before starting
   a certain application.

2. It should be easier for a developer of an existing client-server
   application to just use a different socket in compare to learning
   how to use DBus and how to integrate it into application.
   Especially, it's much easier to use another socket if they want
   to keep traditional way of connection as alternative to connection
   with a SocketPair Broker.

3. Unclear security implication.  I tried to research how to use
   host DBus from the container and I didn't find a good convenient
   way to do that.  Please, point me to a correct documentation if
   I missed something.  Solution that I managed to google includes
   mounting of a /run/user/<user-id> or dbus sessions directory into
   the container and copying a dbus configuration files.
   Some articles also points out that communication is only possible
   from a privileged containers.  To be clear, I know very little
   about DBus, so any pointers on how to use it in a convenient way
   from the inside of a container will be appreciated.

> 
> I don't think establishing socketpair connection between process peers sharing some ID, without any other context, is going to be so useful. The relation is usually not symmetrical, and you usually have associated setup/configuration details.

There is an "operation mode" that user can specify in a GET_PAIR
request to the SocketPair Broker.  It might be NONE, CLIENT or
SERVER.  Broker will pair users that provided NONE together as
they likely want to have a symmetrical connection.  And it will
pair users that declared themselves as CLIENTs with users that
specified SEVER.  This is to ensure that in asymmetrical connections
there will be no two "clients" or two "servers".
See:
  https://github.com/igsilya/one-socket/blob/main/doc/socketpair-broker-proto-spec.rst

If you have any idea what else could be added to the protocol to
make it better, I'd love to hear. 

Best regards, Ilya Maximets.