Discussion of technical issues relating to NetBSD's networking subsystem

Darren Reed | 2 May 2012 08:01

Thinking about "branes" for netbsd...

After spending some time thinking about what would be required
to implement branes as part of the SMP networking project,
I've put together the notes below. There are some aspects of
this where I'm not sure yet what the bike shed should look
like (e.g. managing branes) so if you would like to argue
about its colour, please construct it first.

At this point I don't know if it will turn into a formal project
proposal but I thought it would be wise to seek feedback from
others anyway to see if there are aspects of this problem that
I've missed or over designed.

Darren

Assumptions
===========
In the one line brief for branes, there's no stated requirements
about attributes such as their longevity. For instance, should a
brane exist for the duration of a process or longer? This proposal
assumes the former - that a brane is to be created without there
necessarily being any process that is assigned to it.

Phase 1 (Implement default brane)
=================================
* Analyse NetBSD networking stack to determine what data
  structures can or cannot be removed from global context
  - Include in this analysis what the required scope is for
    those that are currently global

* Design and implement structures that describe the external

(Continue reading)

headers

Mouse | 2 May 2012 08:49

Re: Thinking about "branes" for netbsd...

> After spending some time thinking about what would be required to
> implement branes as part of the SMP networking project, [...]

What's a brane in this context?  The only meaning I'm familiar with for
the term is from particle physics and makes no sense here.  I did a
little searching, and, while my Web-fu is admittedly weak, I didn't
find anything the least bit helpful.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Dennis Ferguson | 3 May 2012 09:48

Re: Thinking about "branes" for netbsd...

On 2 May, 2012, at 14:49 , Mouse wrote:
>> After spending some time thinking about what would be required to
>> implement branes as part of the SMP networking project, [...]
> 
> What's a brane in this context?  The only meaning I'm familiar with for
> the term is from particle physics and makes no sense here.  I did a
> little searching, and, while my Web-fu is admittedly weak, I didn't
> find anything the least bit helpful.

I think I've only seen the term used here.  It seems to refer to
a multiple routing table, or routing instance, mechanism, which I
understand, but appears to be laden with additional, implied policy
semantics concerning the use of the mechanism, which I don't quite
understand.  The term "VRF" is used by router vendors for essentially
the identical mechanism (i.e. more than one routing table), but that
term is loaded with what seem to be an entirely different set of policy
assumptions concerning what the mechanism will be used for, assumptions
also different from, say, NAT, which is another router thing which is
usefully (but not necessarily) structured around a multiple routing
instances.

Assuming this is correct, I'll admit that I much prefer thinking about
the bare underlying mechanism and its associated configuration, trying to
make the latter rich enough that a user can configure his own way to
whatever policy he would like to use the mechanism to implement without
strong a priori assumptions about what that policy will be.  I stumble over
the "processes get chroot'd into branes" thing that always seems to arise
in conversations about branes since, for most of the applications of
multiple routing instances that I understand, this is not very useful.

(Continue reading)

headers

Mouse | 3 May 2012 17:05

Re: Thinking about "branes" for netbsd...

>> What's a brane in this context?
> I think I've only seen the term used here.  It seems to refer to a
> multiple routing table, or routing instance, mechanism, [...]

Ah.  Thanks for explaining.

> I stumble over the "processes get chroot'd into branes" thing that
> always seems to arise in conversations about branes since, for most
> of the applications of multiple routing instances that I understand,
> this is not very useful.

I'm not even sure what it could mean, since chroot is about pathname
interpretation and IP packet routing has nothing to do with pathnames
as far as I can see.  Perhaps something is being extended
metaphorically, but it's unclear to me what or how that could be.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Gert Doering | 3 May 2012 17:33

Re: Thinking about "branes" for netbsd...

Hi,

On Thu, May 03, 2012 at 11:05:12AM -0400, Mouse wrote:
> > I stumble over the "processes get chroot'd into branes" thing that
> > always seems to arise in conversations about branes since, for most
> > of the applications of multiple routing instances that I understand,
> > this is not very useful.
> 
> I'm not even sure what it could mean, since chroot is about pathname
> interpretation and IP packet routing has nothing to do with pathnames
> as far as I can see.  Perhaps something is being extended
> metaphorically, but it's unclear to me what or how that could be.

"Give processes a different view to the network than 'the rest of the
machine' has" - not that different to "give processes a different view to 
the file system", no?

In a way similar to jails on FreeBSD, where you can restrict which IP 
addresses can be used by the jailed processes.

gert
--

-- 
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             gert <at> greenie.muc.de
fax: +49-89-35655025                        gert <at> net.informatik.tu-muenchen.de

headers

Mouse | 3 May 2012 18:23

Re: Thinking about "branes" for netbsd...

>>> [...] "processes get chroot'd into branes" [...]
>> I'm not even sure what it could mean, [...].  Perhaps something is
>> being extended metaphorically, but it's unclear to me what or how
>> that could be.
> "Give processes a different view to the network than 'the rest of the
> machine' has" - not that different to "give processes a different
> view to the file system", no?

Yeah, but that involves (or, at least, I would expect that to involve)
more than just the routing table.  Or do branes cover more than just
routing?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Gert Doering | 3 May 2012 19:41

Re: Thinking about "branes" for netbsd...

Hi,

On Thu, May 03, 2012 at 12:23:36PM -0400, Mouse wrote:
> >>> [...] "processes get chroot'd into branes" [...]
> >> I'm not even sure what it could mean, [...].  Perhaps something is
> >> being extended metaphorically, but it's unclear to me what or how
> >> that could be.
> > "Give processes a different view to the network than 'the rest of the
> > machine' has" - not that different to "give processes a different
> > view to the file system", no?
> 
> Yeah, but that involves (or, at least, I would expect that to involve)
> more than just the routing table.  Or do branes cover more than just
> routing?

I'm not sure how that particular bikeshed is planned, but I'd consider
"IP addresses on Interfaces + Routing + Firewall rules" to be all I
need to define a network environment.  If that's a "brane", I could
very well see a process being "chroot()"ed into that.

Sort of like a VRF on a Cisco router, with some of the services on
the box (like "telnet") only operating within a certain routing
instance, but not in others.  Or particularily, telnet://192.168.0.1 
only working in a particular VRF, and not in another that happens
to use 192.168.0.1 as well  [not that this works overly well on Cisco
today, but some bits already do, and the idea is sane].

gert
--

-- 
USENET is *not* the non-clickable part of WWW!

(Continue reading)

headers

Darren Reed | 4 May 2012 01:46

Re: Thinking about "branes" for netbsd...

On 4/05/2012 3:41 AM, Gert Doering wrote:
> Hi,
> 
> On Thu, May 03, 2012 at 12:23:36PM -0400, Mouse wrote:
>>>>> [...] "processes get chroot'd into branes" [...]
>>>> I'm not even sure what it could mean, [...].  Perhaps something is
>>>> being extended metaphorically, but it's unclear to me what or how
>>>> that could be.
>>> "Give processes a different view to the network than 'the rest of the
>>> machine' has" - not that different to "give processes a different
>>> view to the file system", no?
>>
>> Yeah, but that involves (or, at least, I would expect that to involve)
>> more than just the routing table.  Or do branes cover more than just
>> routing?
> 
> I'm not sure how that particular bikeshed is planned, but I'd consider
> "IP addresses on Interfaces + Routing + Firewall rules" to be all I
> need to define a network environment.  If that's a "brane", I could
> very well see a process being "chroot()"ed into that.

That is my interpretation as well.

The proposal at this point in time has the following goals:
- make the kernel able to support different networking domains
- make it possible to manage those domains
- make it possible for processes to change domain into a brane

There is additional work required to:
- determine how and when to load the firewall configuration for a brane

(Continue reading)

headers

Mouse | 4 May 2012 03:37

Re: Thinking about "branes" for netbsd...

> ... the topic of privileges is something that I'd like to avoid for
> now, except to say that changing brane is like changing root - you
> can only do it once.

Huh?  How is changing root something you can do only once?  Has chroot
been broken when you're already chrooted?  That would strike me as a
bit of a regression....

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

David Young | 4 May 2012 05:37

Re: Thinking about "branes" for netbsd...

On Fri, May 04, 2012 at 09:46:24AM +1000, Darren Reed wrote:
> On 4/05/2012 3:43 AM, David Young wrote:
> > The general idea is to have more than one forwarding domain per router.
> > Belonging to each forwarding domain are the routes for that domain and
> > some interfaces.  Each route/interface can belong to just one domain.
> > Packets cannot cross from one forwarding domain to another except by
> > going through an interface.  We can imagine a virtual interface that has
> > two "ends," each end in a different forwarding domain, for shuttling
> > packets from domain to domain.  More commonly we will have a hardware
> > interface that attaches a NetBSD router's forwarding domain to the
> > forwarding domain of a router/switch that's connected with an ethernet
> > cable.
> 
> Except for the notion that a route can belong to one domain,
> this is otherwise in agreement with what's proposed. More
> than one domain may have a specific route, for example, two
> domains may have the same default route.

How/why do two domains share a route, even the default?

It might help to see a made-up routing table for a couple of domains.

> > ISTM that it will be useful sometimes for a tunnel interface to straddle
> > two domains, sending/receiving encapsulated packets on one domain and
> > sending/receiving decap'd packets on the other.
> 
> And what if you have three domains?
> Do you then need n-1 tunnels in each domain to route between them?
> That doesn't scale.

(Continue reading)

headers

Gert Doering | 4 May 2012 09:45

Re: Thinking about "branes" for netbsd...

Hi,

On Thu, May 03, 2012 at 10:37:35PM -0500, David Young wrote:
> > Except for the notion that a route can belong to one domain,
> > this is otherwise in agreement with what's proposed. More
> > than one domain may have a specific route, for example, two
> > domains may have the same default route.
> 
> How/why do two domains share a route, even the default?
> 
> It might help to see a made-up routing table for a couple of domains.

In Cisco IOS, you have <n> independent routing tables, but you can
have routes pointing out of a routing instance by specifying a target
interface that belongs to another routing instance.  Like:

  ip route vrf BLUE 0.0.0.0 0.0.0.0 gige3/6 192.0.2.1

so if "gige3/6" belongs to "vrf RED", the default route for packets 
in "vrf BLUE" will make them change the vrf.

What you cannot easily do in IOS is "keep the packets on the same box,
but have them change vrf" - but that's something, for example, Juniper
ScreenOS can do with their "virtual routers" - you can point a route 
at another vrouter, to make it jump routing tables

set vrouter "trust-vr"
  set route 172.18.0.0/16 vrouter "untrust-vr"

(a "vrouter" is, basically, a collection of routing table entries plus

(Continue reading)

headers

David Young | 4 May 2012 15:46

Re: Thinking about "branes" for netbsd...

On Fri, May 04, 2012 at 09:45:32AM +0200, Gert Doering wrote:
> Hi,
> 
> On Thu, May 03, 2012 at 10:37:35PM -0500, David Young wrote:
> > > Except for the notion that a route can belong to one domain,
> > > this is otherwise in agreement with what's proposed. More
> > > than one domain may have a specific route, for example, two
> > > domains may have the same default route.
> > 
> > How/why do two domains share a route, even the default?
> > 
> > It might help to see a made-up routing table for a couple of domains.
> 
> In Cisco IOS, you have <n> independent routing tables, but you can
> have routes pointing out of a routing instance by specifying a target
> interface that belongs to another routing instance.  Like:
> 
>   ip route vrf BLUE 0.0.0.0 0.0.0.0 gige3/6 192.0.2.1
> 
> so if "gige3/6" belongs to "vrf RED", the default route for packets 
> in "vrf BLUE" will make them change the vrf.
> 
> What you cannot easily do in IOS is "keep the packets on the same box,
> but have them change vrf" - but that's something, for example, Juniper
> ScreenOS can do with their "virtual routers" - you can point a route 
> at another vrouter, to make it jump routing tables
> 
> set vrouter "trust-vr"
>   set route 172.18.0.0/16 vrouter "untrust-vr"
>

(Continue reading)

headers

Darren Reed | 4 May 2012 14:09

Re: Thinking about "branes" for netbsd...

David Young wrote:
> On Fri, May 04, 2012 at 09:46:24AM +1000, Darren Reed wrote:
>> On 4/05/2012 3:43 AM, David Young wrote:
>>> The general idea is to have more than one forwarding domain per router.
>>> Belonging to each forwarding domain are the routes for that domain and
>>> some interfaces.  Each route/interface can belong to just one domain.
>>> Packets cannot cross from one forwarding domain to another except by
>>> going through an interface.  We can imagine a virtual interface that has
>>> two "ends," each end in a different forwarding domain, for shuttling
>>> packets from domain to domain.  More commonly we will have a hardware
>>> interface that attaches a NetBSD router's forwarding domain to the
>>> forwarding domain of a router/switch that's connected with an ethernet
>>> cable.
>> Except for the notion that a route can belong to one domain,
>> this is otherwise in agreement with what's proposed. More
>> than one domain may have a specific route, for example, two
>> domains may have the same default route.
>
> How/why do two domains share a route, even the default?

Sharing a route is not the same as having the same route.

So I do not see routes being shared but I do see more than
one brane needing to have the same route. So perhaps "belong"
is the wrong word to use, but the same route can be present
in more than one brane at any one time.

A very basic instance of this would be different interfaces,
with different IP addresses, each in their own brane. Thus
each would necessarily require the network route that uses

(Continue reading)

headers

Dennis Ferguson | 4 May 2012 11:22

Re: Thinking about "branes" for netbsd...

On 4 May, 2012, at 07:46 , Darren Reed wrote:
> Further, if bridge(4) was to support TRILL then it may not make sense
> to use VLANs.

How does TRILL help?  As I understand it, TRILL is an effort to replace
spanning tree routing with a real link-state routing protocol so you can
use all the links you have configured between switches to carry traffic,
rather than turning off links until there's only one way to reach any
destination a la spanning tree.

Whether spanning tree or IS-IS is used to route a bridged ethernet doesn't
seem to have much effect on what VLANs do, though (I'll also admit I don't
understand how the use of VLANs would eliminate bridge(4) in any case).  Is
there something else that TRILL changes?

Dennis Ferguson

headers

David Young | 3 May 2012 19:43

Re: Thinking about "branes" for netbsd...

On Thu, May 03, 2012 at 12:23:36PM -0400, Mouse wrote:
> >>> [...] "processes get chroot'd into branes" [...]
> >> I'm not even sure what it could mean, [...].  Perhaps something is
> >> being extended metaphorically, but it's unclear to me what or how
> >> that could be.
> > "Give processes a different view to the network than 'the rest of the
> > machine' has" - not that different to "give processes a different
> > view to the file system", no?
> 
> Yeah, but that involves (or, at least, I would expect that to involve)
> more than just the routing table.  Or do branes cover more than just
> routing?

Matt Thomas describes branes here,
<http://www.netbsd.org/~matt/smpnet.html#branes>.

I should say, before I launch into my own explanation, that I have a
particular understanding of "branes" that may be different from Matt's
and others'.

The general idea is to have more than one forwarding domain per router.
Belonging to each forwarding domain are the routes for that domain and
some interfaces.  Each route/interface can belong to just one domain.
Packets cannot cross from one forwarding domain to another except by
going through an interface.  We can imagine a virtual interface that has
two "ends," each end in a different forwarding domain, for shuttling
packets from domain to domain.  More commonly we will have a hardware
interface that attaches a NetBSD router's forwarding domain to the
forwarding domain of a router/switch that's connected with an ethernet
cable.

(Continue reading)

headers

Dennis Ferguson | 4 May 2012 10:16

Re: Thinking about "branes" for netbsd...

On 4 May, 2012, at 01:43 , David Young wrote:
> The general idea is to have more than one forwarding domain per router.
> Belonging to each forwarding domain are the routes for that domain and
> some interfaces.  Each route/interface can belong to just one domain.

Okay, but maybe I can reword what I understand this to say in terms that
(to me) more clearly describe mechanism.  It would be possible to configure more
than one route table for a protocol, with routes being added to, changed
deleted from each table independently.  This actually requires replication
of several address-bearing structures in addition to the route table (e.g.
tables for PCB lookups), so let me call this group of structures a "routing
instance".

The basic configurables around this are that each interface (or "interface",
for some logical definition of this) in the machine running this network
protocol needs to be told which of these tables it should use to route
incoming packets, with the interface routes generated as a side effect
of address configuration on that interface being stored in the same table.
In addition, each open socket needs to be configured (or somehow come to
understand) which table it should use to route outbound packets and
which "routing instance" its socket binding should be stored in.  The
collection of a routing instance, the interfaces configured to forward
packets through and install routes in that routing instance and the
sockets which are using the instance are what I think you are calling
a "domain".  Beyond this, what each of these routing instances does that
is "special" is defined by the routes installed in each table, and perhaps
the filters operating around the table.

> Packets cannot cross from one forwarding domain to another except by

(Continue reading)

headers

David Young | 4 May 2012 18:38

Re: Thinking about "branes" for netbsd...

On Fri, May 04, 2012 at 04:16:48PM +0800, Dennis Ferguson wrote:
> 
> On 4 May, 2012, at 01:43 , David Young wrote:
> > The general idea is to have more than one forwarding domain per router.
> > Belonging to each forwarding domain are the routes for that domain and
> > some interfaces.  Each route/interface can belong to just one domain.
> 
> Okay, but maybe I can reword what I understand this to say in terms that
> (to me) more clearly describe mechanism.  It would be possible to configure more
> than one route table for a protocol, with routes being added to, changed
> deleted from each table independently.  This actually requires replication
> of several address-bearing structures in addition to the route table (e.g.
> tables for PCB lookups), so let me call this group of structures a "routing
> instance".
> 
> The basic configurables around this are that each interface (or "interface",
> for some logical definition of this) in the machine running this network
> protocol needs to be told which of these tables it should use to route
> incoming packets, with the interface routes generated as a side effect
> of address configuration on that interface being stored in the same table.
> In addition, each open socket needs to be configured (or somehow come to
> understand) which table it should use to route outbound packets and
> which "routing instance" its socket binding should be stored in.  The
> collection of a routing instance, the interfaces configured to forward
> packets through and install routes in that routing instance and the
> sockets which are using the instance are what I think you are calling
> a "domain".

The collection you describe sounds like a domain to me.

(Continue reading)

headers

Mouse | 4 May 2012 20:24

Re: Thinking about "branes" for netbsd...

>> By not stopping at "I can configure this policy", but rather going
>> all the way to "No other policy can possibly be configured" I think
>> you make the mechanism less generally useful and I'd sort of like to
>> see the justification for that.
> You're right, it isn't desirable to be so restrictive.

I agree.  It strikes me as rather restrictive to mandate that which
routing table is used is tied to which process is responsible for the
traffic - especially since sockets can be shared by multiple processes,
meaning that either you have to somehow forbid the same socket from
appearing in the open file tables of processes using different branes,
or you have to tag the data in the socket's send queue with which brane
the writing process was in at the time of writing, or you have to
attach branes to sockets as well as processes, or you have to make the
brane association work only for immediately-sent traffic (eg, UDP but
not TCP), or something of the sort.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Darren Reed | 5 May 2012 01:32

Re: Thinking about "branes" for netbsd...

On 5/05/2012 4:24 AM, Mouse wrote:
>>> By not stopping at "I can configure this policy", but rather going
>>> all the way to "No other policy can possibly be configured" I think
>>> you make the mechanism less generally useful and I'd sort of like to
>>> see the justification for that.
>> You're right, it isn't desirable to be so restrictive.
> 
> I agree.  It strikes me as rather restrictive to mandate that which
> routing table is used is tied to which process is responsible for the
> traffic - especially since sockets can be shared by multiple processes,
> meaning that either you have to somehow forbid the same socket from
> appearing in the open file tables of processes using different branes,
> or you have to tag the data in the socket's send queue with which brane
> the writing process was in at the time of writing, or you have to
> attach branes to sockets as well as processes, or you have to make the
> brane association work only for immediately-sent traffic (eg, UDP but
> not TCP), or something of the sort.

It would appear that various people did not fully read what I proposed
or perhaps misunderstood it. What I wrote is that a brane would become
an entire instance of networking and that global structures, etc, would
disappear. If it helps you (or others) to understand it by thinking of
branes in terms of support for virtual routers then I'm happy for you
to do so but it is by no means limited to that.

For the purposes of this and other discussion, I consider the default
brane to be the networking environment that is created at boot.

In that context, if a process belongs to a brane then all of its sockets,
etc, would also belong to the new brane. Given that it is proposed that

(Continue reading)

headers

David Young | 5 May 2012 03:56

Re: Thinking about "branes" for netbsd...

On Sat, May 05, 2012 at 09:32:36AM +1000, Darren Reed wrote:
> On 5/05/2012 4:24 AM, Mouse wrote:
> >>> By not stopping at "I can configure this policy", but rather going
> >>> all the way to "No other policy can possibly be configured" I think
> >>> you make the mechanism less generally useful and I'd sort of like to
> >>> see the justification for that.
> >> You're right, it isn't desirable to be so restrictive.
> > 
> > I agree.  It strikes me as rather restrictive to mandate that which
> > routing table is used is tied to which process is responsible for the
> > traffic - especially since sockets can be shared by multiple processes,
> > meaning that either you have to somehow forbid the same socket from
> > appearing in the open file tables of processes using different branes,
> > or you have to tag the data in the socket's send queue with which brane
> > the writing process was in at the time of writing, or you have to
> > attach branes to sockets as well as processes, or you have to make the
> > brane association work only for immediately-sent traffic (eg, UDP but
> > not TCP), or something of the sort.
> 
> It would appear that various people did not fully read what I proposed
> or perhaps misunderstood it. What I wrote is that a brane would become
> an entire instance of networking and that global structures, etc, would
> disappear. If it helps you (or others) to understand it by thinking of
> branes in terms of support for virtual routers then I'm happy for you
> to do so but it is by no means limited to that.

I think we are all finding out that your ideas and enthusiasm for
branes resonate with others' ideas and enthusiasm about similar but
not-quite-alike overhauls of NetBSD networking.  Let's take that
resonance and run with it---not all in different directions, I hope!

(Continue reading)

headers

Mouse | 5 May 2012 04:29

Re: Thinking about "branes" for netbsd...

> There may be less enthusiasm for branes in particular than there is
> for multiple routing tables and for a network stack with a more
> rational structure.

For example, I would like to see multiple routing tables, but if the
design requires that the mechanism which selects which table gets used
is driven off solely which process sent the data, or which process
created the socket...then it's of little value to me.  I would prefer
multiple routing tables with less policy attached; I want to select the
routing table based on other things.  (The principal one I have in mind
at the moment is, loosely put, ip_src, but I'm fairly confident there
are other alternatives as well; to name three that come to mind
immediately, time-of-day, load average, and anything computable by a
bpf bytecode program.)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Ted Lemon | 5 May 2012 04:37

Re: Thinking about "branes" for netbsd...

It might be worth looking at the work that the MIF working group at IETF is doing with respect to provisioning
domains when talking about this problem space.   I can go into more detail if people are interested, but the
essential focus of what they are doing has to do with the problem of a device that has N interfaces, where N is
typically greater than 1, and they are connected to different physical networks operated by different
operators.   It also addresses the case where you have a dual-stack network.

My theory as how this should be made to work sounds a little bit like the brane proposal, and a little bit like
what you are saying the brane proposal isn't, Mouse.

headers

Dennis Ferguson | 5 May 2012 06:32

Re: Thinking about "branes" for netbsd...

On 5 May, 2012, at 07:32 , Darren Reed wrote:
> It would appear that various people did not fully read what I proposed
> or perhaps misunderstood it. What I wrote is that a brane would become
> an entire instance of networking and that global structures, etc, would
> disappear. If it helps you (or others) to understand it by thinking of
> branes in terms of support for virtual routers then I'm happy for you
> to do so but it is by no means limited to that.

The point was not that the mechanism should be thought of in terms of
a VRF implementation, it is only that the mechanism should not have arbitrary
constraints that prevent you from using it for a VRF implementation, or any
other job that the mechanism might be appropriate to use for.

> In that context, if a process belongs to a brane then all of its sockets,
> etc, would also belong to the new brane.

But, speaking of a topic that came up a week or two ago, what if the thing
I want to use the multiple tables for is to implement strict host behavior
on a multi-homed host?  That is, suppose I am running a server application
on a host with two interfaces, and I just want the reply packets to connections
that came in one or the other of the interfaces to go back out the same
interface (say the interfaces connect to different service providers).  Two
routing tables, one per interface, are required for this since the outbound
routing from the host needs to be done through a table that only has routes
pointing out the interface the packet needs to leave on.

For this to work I want my server process to be able to send and receive
packets in either brane (or, at least, using either table) at will.  In fact,
an ideal implementation might allow a single listening socket to receive

(Continue reading)

headers

Darren Reed | 5 May 2012 07:43

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 5 May, 2012, at 07:32 , Darren Reed wrote:
>> In that context, if a process belongs to a brane then all of its sockets,
>> etc, would also belong to the new brane.
>
> But, speaking of a topic that came up a week or two ago, what if the thing
> I want to use the multiple tables for is to implement strict host behavior
> on a multi-homed host?  That is, suppose I am running a server application
> on a host with two interfaces, and I just want the reply packets to connections
> that came in one or the other of the interfaces to go back out the same
> interface (say the interfaces connect to different service providers).  Two
> routing tables, one per interface, are required for this since the outbound
> routing from the host needs to be done through a table that only has routes
> pointing out the interface the packet needs to leave on.

Branes are the wrong solution for that problem.

What is better is to be able to issue an ioctl on the socket
and in that fashion specify which network interface to use
for sending out packets.

Darren

headers

Dennis Ferguson | 5 May 2012 07:15

Re: Thinking about "branes" for netbsd...


On 5 May, 2012, at 13:43 , Darren Reed wrote:
> Dennis Ferguson wrote:
>> On 5 May, 2012, at 07:32 , Darren Reed wrote:
>>> In that context, if a process belongs to a brane then all of its sockets,
>>> etc, would also belong to the new brane.
>> 
>> But, speaking of a topic that came up a week or two ago, what if the thing
>> I want to use the multiple tables for is to implement strict host behavior
>> on a multi-homed host?  That is, suppose I am running a server application
>> on a host with two interfaces, and I just want the reply packets to connections
>> that came in one or the other of the interfaces to go back out the same
>> interface (say the interfaces connect to different service providers).  Two
>> routing tables, one per interface, are required for this since the outbound
>> routing from the host needs to be done through a table that only has routes
>> pointing out the interface the packet needs to leave on.
> 
> Branes are the wrong solution for that problem.

Probably, but multiple forwarding tables are essential to solve that
problem.  This is the problem with "multiple forwarding tables" == "branes".

> What is better is to be able to issue an ioctl on the socket
> and in that fashion specify which network interface to use
> for sending out packets.

The problem is that, no matter what the ioctl tells the socket
to do, the kernel cannot send a packet out an interface if it
does not have a route to the packet's destination pointing out
that interface.  If the destination is being routed with a

(Continue reading)

headers

Darren Reed | 5 May 2012 09:33

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 5 May, 2012, at 13:43 , Darren Reed wrote:
>> Dennis Ferguson wrote:
>>> On 5 May, 2012, at 07:32 , Darren Reed wrote:
>>>> In that context, if a process belongs to a brane then all of its sockets,
>>>> etc, would also belong to the new brane.
>>> But, speaking of a topic that came up a week or two ago, what if the thing
>>> I want to use the multiple tables for is to implement strict host behavior
>>> on a multi-homed host?  That is, suppose I am running a server application
>>> on a host with two interfaces, and I just want the reply packets to connections
>>> that came in one or the other of the interfaces to go back out the same
>>> interface (say the interfaces connect to different service providers).  Two
>>> routing tables, one per interface, are required for this since the outbound
>>> routing from the host needs to be done through a table that only has routes
>>> pointing out the interface the packet needs to leave on.
>> Branes are the wrong solution for that problem.
>
> Probably, but multiple forwarding tables are essential to solve that
> problem.  This is the problem with "multiple forwarding tables" == "branes".
>
>> What is better is to be able to issue an ioctl on the socket
>> and in that fashion specify which network interface to use
>> for sending out packets.
>
> The problem is that, no matter what the ioctl tells the socket
> to do, the kernel cannot send a packet out an interface if it
> does not have a route to the packet's destination pointing out
> that interface.  If the destination is being routed with a
> default route then there need to be two default routes (if it
> is being routed with some other route there need to be two

(Continue reading)

headers

Dennis Ferguson | 5 May 2012 10:11

Re: Thinking about "branes" for netbsd...


On 5 May, 2012, at 15:33 , Darren Reed wrote:
> Dennis Ferguson wrote:
>> 
>> 
>> The problem is that, no matter what the ioctl tells the socket
>> to do, the kernel cannot send a packet out an interface if it
>> does not have a route to the packet's destination pointing out
>> that interface.  If the destination is being routed with a
>> default route then there need to be two default routes (if it
>> is being routed with some other route there need to be two
>> of those), one for each interface, which means you need two
>> forwarding tables to store the different routes in.  It is a
>> multiple forwarding table problem even if it isn't one that
>> a "brane" works for.
> 
> For reference, you might like to investigate the following
> IP socket options that are available on OpenSolaris:
> IP_NEXTHOP
> IP_BOUND_IF
> The latter of which is similar to Linux's SO_BINDTODEVICE.
> I don't know if Linux has an equivalent of IP_NEXTHOP but
> as an ioctl, it works in a similar way (for a particular
> socket) as does policy routing with ipfilter. I believe
> that either one or both of those are a solution to your
> problem without implementing virtual routing tables.

I'm aware of both of those.  I think you may be confused about
what IP_BOUND_IF does (hint: last I looked it only effects
where multicasts and broadcasts go).  IP_NEXTHOP does help if

(Continue reading)

headers

Darren Reed | 5 May 2012 11:55

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 5 May, 2012, at 15:33 , Darren Reed wrote:
>> Dennis Ferguson wrote:
>>>
>>> The problem is that, no matter what the ioctl tells the socket
>>> to do, the kernel cannot send a packet out an interface if it
>>> does not have a route to the packet's destination pointing out
>>> that interface.  If the destination is being routed with a
>>> default route then there need to be two default routes (if it
>>> is being routed with some other route there need to be two
>>> of those), one for each interface, which means you need two
>>> forwarding tables to store the different routes in.  It is a
>>> multiple forwarding table problem even if it isn't one that
>>> a "brane" works for.
>> For reference, you might like to investigate the following
>> IP socket options that are available on OpenSolaris:
>> IP_NEXTHOP
>> IP_BOUND_IF
>> The latter of which is similar to Linux's SO_BINDTODEVICE.
>> I don't know if Linux has an equivalent of IP_NEXTHOP but
>> as an ioctl, it works in a similar way (for a particular
>> socket) as does policy routing with ipfilter. I believe
>> that either one or both of those are a solution to your
>> problem without implementing virtual routing tables.
>
> I'm aware of both of those.  I think you may be confused about
> what IP_BOUND_IF does (hint: last I looked it only effects
> where multicasts and broadcasts go).  IP_NEXTHOP does help if
> you think this is solved by having each application do its
> own routing (maybe the application could run DHCP to find out

(Continue reading)

headers

Dennis Ferguson | 5 May 2012 13:20

Re: Thinking about "branes" for netbsd...


On 5 May, 2012, at 17:55 , Darren Reed wrote:
> Dennis Ferguson wrote:
>> 
>> I'm aware of both of those.  I think you may be confused about
>> what IP_BOUND_IF does (hint: last I looked it only effects
>> where multicasts and broadcasts go).  IP_NEXTHOP does help if
>> you think this is solved by having each application do its
>> own routing (maybe the application could run DHCP to find out
>> the next hop for that interface's default route too); if all
>> applications did this then the kernel could get even simpler
>> by eliminating all forwarding tables.
> 
> You want an application to send a packet back out the same
> interface that the packet was received on. That amounts to
> the application doing its own routing.

See RFC 1122, section 3.3.4.2, the Strong ES model.  The routing
operation is well-defined, the application doesn't have to be
aware it is happening, but it needs a routing table per interface
to implement it.

This has reached the point of absurdity, however.  I'm done.

Dennis Ferguson

headers

Darren Reed | 5 May 2012 18:05

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 5 May, 2012, at 17:55 , Darren Reed wrote:
>> Dennis Ferguson wrote:
>>> I'm aware of both of those.  I think you may be confused about
>>> what IP_BOUND_IF does (hint: last I looked it only effects
>>> where multicasts and broadcasts go).  IP_NEXTHOP does help if
>>> you think this is solved by having each application do its
>>> own routing (maybe the application could run DHCP to find out
>>> the next hop for that interface's default route too); if all
>>> applications did this then the kernel could get even simpler
>>> by eliminating all forwarding tables.
>> You want an application to send a packet back out the same
>> interface that the packet was received on. That amounts to
>> the application doing its own routing.
>
> See RFC 1122, section 3.3.4.2, the Strong ES model.  The routing
> operation is well-defined, the application doesn't have to be
> aware it is happening, but it needs a routing table per interface
> to implement it.

No, it does not.

The problem as stated is that the routing decision should
include the source address as part of the key for deciding
where to route a packet.

I'll grant that it is possible to read that as suggesting
that using a routing table per interface is the solution
but it is not the only solution to that problem.

(Continue reading)

headers

Darren Reed | 5 May 2012 05:22

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 4 May, 2012, at 01:43 , David Young wrote:
>> The general idea is to have more than one forwarding domain per router.
>> Belonging to each forwarding domain are the routes for that domain and
>> some interfaces.  Each route/interface can belong to just one domain.
>
> Okay, but maybe I can reword what I understand this to say in terms that
> (to me) more clearly describe mechanism.  It would be possible to configure more
> than one route table for a protocol, with routes being added to, changed
> deleted from each table independently.  This actually requires replication
> of several address-bearing structures in addition to the route table (e.g.
> tables for PCB lookups), so let me call this group of structures a "routing
> instance".
>
> The basic configurables around this are that each interface (or "interface",
> for some logical definition of this) in the machine running this network
> protocol needs to be told which of these tables it should use to route
> incoming packets, with the interface routes generated as a side effect
> of address configuration on that interface being stored in the same table.
> In addition, each open socket needs to be configured (or somehow come to
> understand) which table it should use to route outbound packets and
> which "routing instance" its socket binding should be stored in.  The
> collection of a routing instance, the interfaces configured to forward
> packets through and install routes in that routing instance and the
> sockets which are using the instance are what I think you are calling
> a "domain".  Beyond this, what each of these routing instances does that
> is "special" is defined by the routes installed in each table, and perhaps
> the filters operating around the table.

Roughly speaking, more or less yes.

(Continue reading)

headers

Mouse | 5 May 2012 04:38

Re: Thinking about "branes" for netbsd...

> The goal of branes is to support network virtualisation.  One aspect
> of network virtualisation is the ability for the kernel to support
> multiple routing tables as a result.

Indeed.

But, by choosing a design that has that sort of isolation wired into
it, you end up ensuring that they cannot be used for anything else,
rather than creating a general-purpose mechanism which can, among other
things, be used for virtualization.

As the Jaron File notes (see the entry for `uninteresting'), there is a
strong tendency towards general-purpose tools which solve the original
problem as a special case.  I'd much prefer to see this done that way
(though, admittedly, that constitutes armchair quarterbacking).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Darren Reed | 5 May 2012 05:48

Re: Thinking about "branes" for netbsd...

Mouse wrote:
>> The goal of branes is to support network virtualisation.  One aspect
>> of network virtualisation is the ability for the kernel to support
>> multiple routing tables as a result.
>
> Indeed.
>
> But, by choosing a design that has that sort of isolation wired into
> it, you end up ensuring that they cannot be used for anything else,
> rather than creating a general-purpose mechanism which can, among other
> things, be used for virtualization.

I think that the virtualisation approach is superior because it
means that the tools that work with a routing table don't need
to be told that there are different types of routing tables.
In the virtual environment, everything just works without being
aware of which environment it is in.

To my way of thinking, doing something special just for virtual
routing tables means that any tools that work with routing need
to be modified in special ways. If a tool hasn't been modified
then it doesn't work. Thus the solution becomes more frail. As
a quick example of this, what happens to routing socket messages?
Do they get special tags? Do they need to change in a way that
makes them incompatible? And so on. There's hidden complexity
that I believe makes it more trouble than it is worth.

Darren

(Continue reading)

headers

Dennis Ferguson | 5 May 2012 07:58

Re: Thinking about "branes" for netbsd...


On 5 May, 2012, at 11:22 , Darren Reed wrote:

> Dennis Ferguson wrote:
>> 
>>> Packets cannot cross from one forwarding domain to another except by
>>> going through an interface.
>> 
>> And this is a place where I stumble.  If you want to implement a
>> policy which prevents packets from crossing from one domain to another
>> except by going through an interface (and if I understand what that
>> means), why is it not sufficient that you can configure each of these
>> tables with routes which only route packets from one domain to the other
>> via an interface?  Why is it necessary to go the extra step and prohibit
>> someone else from installing a route in one table with a next hop which
>> directly routes out an interface whose inbound packets are forwarded via
>> another table (i.e. an interface in another domain), if that is what is
>> required for that someone else to get their work done?
> 
> To allow cross references between data structures from one brane to
> another or cross talk where packets can move from one brane to another
> would be a very dirty implementation. The goal of branes is to support
> network virtualisation. One aspect of network virtualisation is the
> ability for the kernel to support multiple routing tables as a result.

I learned a long time ago that there is no accounting for taste, and
what looks to me like dirt is often someone else's very excellent and
essential feature.  While the goal of branes might be network virtualization
that is only one of many possible applications for multiple forwarding
tables, so it is bothering me a bit that you seem to want to implement

(Continue reading)

headers

Darren Reed | 5 May 2012 09:50

Re: Thinking about "branes" for netbsd...

Dennis Ferguson wrote:
> On 5 May, 2012, at 11:22 , Darren Reed wrote:
>
>> Dennis Ferguson wrote:
>>>> Packets cannot cross from one forwarding domain to another except by
>>>> going through an interface.
>>> And this is a place where I stumble.  If you want to implement a
>>> policy which prevents packets from crossing from one domain to another
>>> except by going through an interface (and if I understand what that
>>> means), why is it not sufficient that you can configure each of these
>>> tables with routes which only route packets from one domain to the other
>>> via an interface?  Why is it necessary to go the extra step and prohibit
>>> someone else from installing a route in one table with a next hop which
>>> directly routes out an interface whose inbound packets are forwarded via
>>> another table (i.e. an interface in another domain), if that is what is
>>> required for that someone else to get their work done?
>> To allow cross references between data structures from one brane to
>> another or cross talk where packets can move from one brane to another
>> would be a very dirty implementation. The goal of branes is to support
>> network virtualisation. One aspect of network virtualisation is the
>> ability for the kernel to support multiple routing tables as a result.
>
> I learned a long time ago that there is no accounting for taste, and
> what looks to me like dirt is often someone else's very excellent and
> essential feature.  While the goal of branes might be network virtualization
> that is only one of many possible applications for multiple forwarding
> tables, so it is bothering me a bit that you seem to want to implement
> multiple forwarding tables in a way which excludes those other applications
> rather than designing a more general underlying implementation of the
> multiple forwarding table mechanism, and then figuring out how to specialize

(Continue reading)

headers

David Holland | 3 May 2012 20:25

Re: Thinking about "branes" for netbsd...

On Wed, May 02, 2012 at 02:49:43AM -0400, Mouse wrote:
 > > After spending some time thinking about what would be required to
 > > implement branes as part of the SMP networking project, [...]
 > 
 > What's a brane in this context?  The only meaning I'm familiar with for
 > the term is from particle physics and makes no sense here.  I did a
 > little searching, and, while my Web-fu is admittedly weak, I didn't
 > find anything the least bit helpful.

As far as I can tell so far, the meaning has the same relationship to
the meaning from particle physics that, say, "functors" in ML have to
functors in category theory. Which is also similar to the relationship
that "functions" in C have to functions in math.

Can we come up with another term?

--

-- 
David A. Holland
dholland <at> netbsd.org

headers

Darren Reed | 5 May 2012 05:52

Re: Thinking about "branes" for netbsd...

Now that folks are paying attention, I would like to
draw people's attention back to this issue. I would
seriosly like feedback from people on their thoughts
on two topics.

Firstly, there is this issue:
>
> * New tool to manage branes is required that does the following:
>  - create brane
>  - delete brane
>  - list branes
>  - assign a network interface to a brane
>  - remove a network interface from a brane
>  - this should potentially be a generic tool, such as nicctl,
>    or branectl or ... that is dedicated to managing network
>    interfaces and not IP, etc.
>  - ifconfig is not an appropriate tool for managing branes
>    or which brane an interface belongs to because it cannot
>    see all branes. This means that "ifconfig -a" will not
>    report all interfaces when branes are in use.

Next there is the issue of interface naming.

I will add that it may be that "struct devnet" can
be of help to SMP networking, I don't know...

> ...
> Interface naming
> ================
> For each brane to have a lo0 accessible within it requires that

(Continue reading)