Multiple default routes (was: Networking help)

[bscott at ntisys.com]

On Sat, 14 Dec 2002, at 1:26pm, gnhlug at sophic.org wrote:
> While true, historically this feature of IP has been provided by routers.
> it has not been the case with host routing.  That feature in hosts is
> reletively new.  In the case of most host IP stacks, only the first
> default route is ever checked.

  Did you read my post?  **There is nothing special about a default route.**
It is simply a route that matches all packets.  Most implementations do not
do anything special in the case of more than one route.  The code simply
finds the best matching route.  If more than one route matches, the first
route is used because there is nothing in the code to do otherwise.  This
applies to routes that are not default routes as well.

  The fact that other matching routes are ignored does not mean that it
routing is broken.  (It may mean you're not utilizing your network to its
fullest potential, but that's not what we're talking about.)  I have even
used this behavior intentionally to provide a sort of poor man's automatic
fail-over.

  Yes, some implementations will not even let you add multiple routes to a
given destination.  We're not talking about that.  I've even heard of an
implementation that could only handle *one* route, period.  We're not
talking about that, either.

  Load balancing, traffic shaping, policy routing, and other advanced
features are available in some implementations, but because they consume
additional resources -- in particular, because they turn routing into a
stateful operation -- they are often disabled by default.

> Also historically, some routing protocols DID NOT allow multiple routes to
> the same target.  RIP is an example.

  Yes.  RIP's deficiencies are many.  I confess I do not understand the
collection of hacks used to try and get RIP to do things it was never
designed to do.  But, AFAIK, routing protocols were not under discussion.  
I did ask Paul if he was running a routing daemon, because I do know they
often become even more confused than the admins.  :-)

>>   As for the theory that Linux, or a distribution thereof, "automatically
>> adds a default route for every interface", that is incredibly bogus.
> 
> Yes, and no.  If you specify a gateway for all of your interfaces at
> install, the system will automatically configure each of them as a default
> route, rather than allowing you to specify what networks can be routed via
> that gateway.  IMO, this behavior is wrong, unless you know it's what you
> want.

  Why is it wrong?  It may not be especially creative, and it likely is not
optimal, but as long as all the gateways involved are configured correctly,
it is a perfectly valid configuration.  Granted, I don't think the people
writing the install program were really thinking that way.  I suspect they
were thinking that you would only enter a gateway into the "Default Gateway"  
field for one interface.  That does not make it invalid, however.  (Of
course, this is largely due to the fact that so many OS vendors incorrectly
impose the view that gateway routes must be associated with a particular
interface.  Windows even associates DNS servers with interfaces!  Talk about
bogus.  But I digress...)

>> Trying to add a default route which specifies a gateway to which there is
>> no route already will result in a "SIOCADDRT: Network is unreachable"
>> error.
>
> This is what you get if you use /sbin/route on a Linux system... /sbin/ip
> uses a different mechanism to configure the routing tables, and does not
> use the SIOC* ioctls.

  If you try to add a gateway route via an unreachable gateway with the
ip(8) command, you get "RTNETLINK answers: Network is unreachable"  
instead.  It's the same error, just a different mechanism giving it.  
What's your point?

> I have not tried to configure routes to networks which my interfaces were
> not directly connected on Linux 2.4 systems, but on at least some systems
> I've used (notably HP-UX 10.something) when I tried to do this, I got an
> error, even if I DID already have a route to the router I was trying to
> use as a gateway.

  Well, Derek, as you yourself noted in another post, there are
implementations which may impose (often arbitrary) limits which have nothing
to do with IP.  And it is good to be aware of those limits in your
implementation.  But if we list every brain damage in every broken IP stack
out there, we'll be here all day.  :-)

> I thought that it was similar on pre-2.4 kernels with linux, but I can't
> say that I recall to a certainty, and have no 2.2 systems to check.

  I know for a fact it worked.

>>   Most of the time, routing works like this: When an interface is brought
>> up, a route for that interface's directly-connected network(s) or peer(s)
>> is automatically added to the kernel routing table.
> 
> With pre-2.4 kernels, this is not the case.  Those routes had to be added
> manually.

  Hmmmm.  Well, I don't have any 2.2 systems that I can test that on right
this second (I can reach some, but removing the interface I'm SSH'ed in via
would be troublesome).  However, if the kernel did not add the routes
automatically, then Red Hat must have patched their ifconfig(8) utility to
add the routes automatically, because I sure never had to.

> ... my own experience with pre-2.4 kernels is that while having two
> default routes did not always cause problems, it sometimes did.  Getting
> rid of them fixed the problem.

  Again: That is more likely an indication that the routing plan at a given
site is incomplete.  Which is not uncommon.  But that doesn't mean multiple
default routes are in any way a problem.  It does, alas, mean that many
networks are full of problems that one must work around.  In the case of
Paul's problem, removing one of those routes would definitely be worth
trying.  You'll note that he did try that, and it did not help.  However,
there were so many people jumping in saying "You cannot have two default
routes" that I felt it important to explain that it is quite legal.

> In the course of trying to find out more about this topic, I came across
> posts to mailing lists which suggest that the above problems might be
> caused by having rp_filter turned on.

  Short answer: Agreed.

  Longer answer: I have encountered different answers as to what rp_filter
actually *does*.  I tried deciphering the source code, but could never find
the function that actually implemented the check.  (I am most definitely
*not* a kernel hacker.)  The most common explanation for what rp_filter does
I have encountered is: rp_filter drops packets received on an interface
that, if source and destination addresses were reversed, would never be
routed out that interface.  Now, if there really is a default gateway that
is reachable via the interface in question, that means the rp_filter check
*should* pass, given that definition.  However, given the general confusion
surrounding what rp_filter does, and given my own experience with rp_filter
and routing tables that deviate from the common cases, I would say that it
is likely the possibility of multiple routes to a given destination is not
considered for rp_filter, at least in some versions of the kernel.

> I also saw posts that suggested that using /sbin/route to configure
> routing tables on 2.4 kernels may possibly lead to having routing tables
> left in a funky state.

  If true, that is rather bogus on the part of the kernel.  The kernel
should not let a userland utility corrupt its internal state.

  A Google search did not find anything relevant.  Perhaps this was the case
in early 2.4, or even 2.3, versions of the kernel, and the problem has since
been fixed?

> What I'm getting at here is that almost nothing you've said is
> universally true ...

  *Nothing* anyone ever says is universally true, Derek.  However, it seemed
that a great many people on this list thought that having multiple default
routes was universally wrong.  That is incorrect, and *that* is what *I* was
getting at.  I mean, come on.  If Windows 98 can handle it, you have got to
be pretty embarrassed if the IP stack you wrote cannot.  :-)

-- 
Ben Scott <bscott at ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |