Subject: Re: The internet architecture

From: John Leslie <[email-masked]>
Date: Thu, 1 Jan 2009 09:26:59 -0500

Tony Finch <[email-masked]> wrote:
> 
> In fact the use of the term "multihoming" to describe multiple IP
> addresses on the same link is a serious problem with practical
> consequences.

   I see no benefit of setting up "multi-homing" without separate links.
Routing based on source address is a dangerous practice, even when the
source address is trusted.

> The Internet's idea of multihoming as supported by the architecture is
> NOT what I consider to be multihoming. (The lack of support for my
> idea of multihoming makes Internet connections klunky, fragile, and
> immobile, but we all know that, and should be embarrassed by it.)

   I'm not at all clear what "support" of "multihoming" Tony is asking
for...

> The Internet's idea of multihoming is reasonably precisely
> encapsulated by RFC 3484. This specification breaks existing practice
> and fails to do what it is designed to.

   RFC 3484, of course, is "Default Address Selection for IPv6". I
guess that Tony is referring to Section 10.5 (which, frankly, I have
never succeeded in deciphering). If anyone actually does what 10.5
suggests, I have not stumbled upon it.

> OK, so what is Internet multihoming? If the DNS resolves a hostname
> to multiple IP addresses then those are assumed to be multiple links
> to the same host.

   That is sometimes true, and often not.

   We indeed do that in a few cases, to have a path which bypasses
routing or connectivity problems. More often, we use separate DNS names
to point to the different interfaces on the same host.

   Typically, multiple A)ddress records for the same domain name will
point to different hosts configured to return identical responses to
service requests.

> Unfortunately there is no way for a client to make an informed
> decision about which IP address to choose.

   Our attitude is that clients _should_ be expected to choose blindly.

> RFC 3484 specifies how to make an UNINFORMED decision, or at best,
> how one could in principle inform the decision. However in order to be
> informed, a host needs to be hooked into the routing infrastructure -
> but the Internet architecture says that the "dumb" network need not
> explain its workings to the intelligent but ignorant hosts.

   ... which seems about right. Layer 3 is supposed to find an
interconnection from one network in the Internet to another. There
seems to be little point in "explaining" how it does this to the
endpoints.

> As a result, having multiple addreses for the same hostname on
> different links does not work.

   There's some logical inference missing here. It certainly does work,
though presumably it doesn't do something which Tony thinks might be
expected of it.

> It never worked before RFC 3484, and though RFC 3484 tries to fix it,
> it fails because of the lack of routing information.

   I don't follow...

> What is worse is it breaks DNS round robin - which I admit is a hack,
> but it's a hack with 15 years of deployment, therefore not to be
> broken without warning. Na?ve implementations have broken round-robin
> DNS because RFC 3484 ignores it.

   Round-robin seems mostly unrelated -- it was never guaranteed to be
particularly good at load-balancing.

> So to summarize:
> 
> A host has no way to use multiple links to provide redundancy or
> resilience, without injecting a PI route into the DFZ - like the
> anycasted root name servers.

   This would be nice to fix, but it's not clear there's a sufficient
constituency interested in fixing it.

> Given multiple addresses for the same hostname, a client has no way
> to make an informed decision about which is the best to connect to.
> This is why hosts that support IPv6 do not work as well as IPv4-only
> hosts.

   I guess I don't follow. Indeed, IPv6 hosts that follow RFC 3483
will defeat some attempts at load-balancing, but it would seem that
this would only affect server farms using IPv6 -- which ought to know
better than to depend on those load-balancing tricks.

> The Internet addressing architecture has a built-in idea that there
> is one instance of each application per host, and applications are
> identified by protocols (port numbers).

   This is a broken idea. It should be abandoned.

> There is no support for multiple instances of the same application
> per host (i.e. virtual hosting) unless the application has its own
> addressing.

   I'm not clear what Tony might see as such "support".

> There is no support for distributing an application across multiple
> hosts (or multiple links to the same host) because address selection
> is blind to availability and reachability - whether you consider them
> to be binary or fractional values.

   Again, I'm not clear.

> If you try to use it you are either relying on the non-kosher
> round-robin DNS, or you are likely to suffer failed or sub-optimal
> connections.

   RFC 3484 specifies that implementations of getaddrinfo() should sort
the list of IPv6 and IPv4 addresses that they return. (This has never
seemed to me a particularly good idea.) It goes on to state that
applications should iterate through the list until they find a working
address. (This need not imply a delay, as I read it.)

> In practice multihomed services (services with multiple redundant
> links to the public Internet) do not use any of the techniques
> described in RFCs as host multihoming, and often use techniques
> that are contrary to the architecture or outright protocol
> violations (e.g. Akamai's use of CNAME chains).

   Does Tony have an alternative to suggest?

--
John Leslie <[email-masked]>
_______________________________________________
Ietf mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: [email-masked]
Date: Thu, 01 Jan 2009 08:00:43 -0800 (PST)

> I've been asked twice now in private email to clarify what I mean, so this
> is going to turn into a massive rant about how the current Internet
> architecture - as it is deployed, and as it seems to be developing - has a
> completely broken idea of how to address endpoints. The multiple meanings
> of the word "multihoming" relate directly to the multiple points in the
> rant.

Tony, I pretty much agree with everything you say here. There really is a
pretty serious disconnect between what we seem to be able to build
and what applications actually need.

> > This is rather more common for machines that mostly accept connections
> > (web servers), as you can do relatively simple source address based
> > policy routing.

> In my experience policy routing is not what people use multiple addresses
> on the same link for. Multiple addresses on the same link are used for
> virtual hosting for application protocols that don't signal
> application-level addresses within the application protocol. The canonical
> example is HTTP over SSL, but POP and IMAP have the same problem. (People
> sometimes hack around this design error in POP and IMAP by embedding the
> virtual domain in the username, which is yet another example of why every
> application protocol needs its own addressing architecture.)

Indeed. But of course this approach doesn't work when a diffferent POP or IMAP
server is needed for different virtual hosts, so you throw in a proxy to deal
with that case. The proxy in turn can make authentication fairly ... exciting.

But you're still stuck the minute you hit the need for different virtual hosts
to use different certificates. More recent protocols, starting, I believe, with
MTQP (RFC 3887) have a domain parameter on the STARTTLS command. And I think
there's a TLS extension to do this. But older protocols (which means most of
them) don't have the parameter, and TLS-level support is nonexistant in 
practice, so in this case you're stuck with certificate selection based on IP
addresses. In some very unusual scenarios with disjoint client pools or
unusually flexible clients you may be able to get away with doing it based on
source address or destination port respectively, but most of the time you end
up with multiple destination IPs to handle this.

And even the parameter is clunky - it feels like what it is: A tacked on field.
You're absolutely right that this highlights our failure to give application
addressing design the attention it deserves.

> To
> re-iterate, one computer providing multiple different services on the same
> IP address is NOT multihoming, in the same way that one computer providing
> multiple services on different port numbers is not multihoming.

> The dual scenario is multiple computers providing the same service on
> different IP addresses. The simplest deployment is to scale up on the
> cheap by relying on round-robin DNS. In this case there are multiple
> links, but they are often on the same layer 2 segment, so again not
> multihoming in the sense that I meant. If you scale up further, the first
> thing you do is get proper resilience with a load-balancing router (which
> probably requires the architecturally impure NAT).

There are also lots of commonly-ussed solution points between round-robin DNS
and load-balancing routers where more than one host effectively shares
an IP address. These tricks are often, but not always, associated with
various clustering schemes.

One unfortunate side effect of all this is that applications end up having to
know all sorts of stuff about IP addresses - both internal and external. This
in turn contributes to the renumbering problem in ways that IPv6 support  for
multiple addresses per interfaces cannot address.

> If you require more
> than one point of presence, the next step (beyond layer 2 techniques like
> trunking ethernet vlans between multiple sites) is IP routing tricks, i.e.
> anycast. For serious wide-area services, you are likely to use location-
> and availibity-sensitive DNS to deirect users to the right instance.

> Note that NONE of these techniques use the architecturally-supported idea
> of multihoming. In fact most of them deliberately avoid it because it does
> not work!

> The Internet's idea of multihoming as supported by the architecture is NOT
> what I consider to be multihoming. (The lack of support for my idea of
> multihoming makes Internet connections klunky, fragile, and immobile, but
> we all know that, and should be embarrassed by it.) The Internet's idea of
> multihoming is reasonably precisely encapsulated by RFC 3484. This
> specification breaks existing practice and fails to do what it is designed
> to.

> OK, so what is Internet multihoming? If the DNS resolves a hostname to
> multiple IP addresses then those are assumed to be multiple links to the
> same host.

> Unfortunately there is no way for a client to make an informed decision
> about which IP address to choose. RFC 3484 specifies how to make an
> UNINFORMED decision, or at best, how one could in principle inform the
> decision. However in order to be informed, a host needs to be hooked into
> the routing infrastructure - but the Internet architecture says that the
> "dumb" network need not explain its workings to the intelligent but
> ignorant hosts.

And what ends up happening in practice is applications grow a handful of knobs
for tuning this stuff. But since there's no specification for any of this
different implementations end up with knobs with different semantics and
different levels of applicability. (In the latter case I'm referring to, say,
an application that uses multiple protocols having the same or different knobs
for each protocol.)

> As a result, having multiple addreses for the same hostname on different
> links does not work. It never worked before RFC 3484, and though RFC 3484
> tries to fix it, it fails because of the lack of routing information. What
> is worse is it breaks DNS round robin - which I admit is a hack, but it's
> a hack with 15 years of deployment, therefore not to be broken without
> warning. Naïve implementations have broken round-robin DNS because RFC
> 3484 ignores it.

> Given multiple addresses for the same hostname, a client has no way to
> make an informed decision about which is the best to connect to. This is
> why hosts that support IPv6 do not work as well as IPv4-only hosts.

In my experience this is often the initial reason for adding
application-specific configuation settings to control this stuff.

> The Internet addressing architecture has a built-in idea that there is one
> instance of each application per host, and applications are identified by
> protocols (port numbers). There is no support for multiple instances of
> the same application per host (i.e. virtual hosting) unless the
> application has its own addressing.

And the more virtualization there is the more of a problem this becomes.

> There is no support for distributing an application across multiple hosts
> (or multiple links to the same host) because address selection is blind to
> availability and reachability - whether you consider them to be binary or
> fractional values. If you try to use it you are either relying on the
> non-kosher round-robin DNS, or you are likely to suffer failed or
> sub-optimal connections.

In some cases you also end up with applications breaking another set of
rules and maintaining caches of past connectivity results.

> In practice multihomed services (services with multiple redundant links to
> the public Internet) do not use any of the techniques described in RFCs as
> host multihoming, and often use techniques that are contrary to the
> architecture or outright protocol violations (e.g. Akamai's use of CNAME
> chains).

Yep, that's exactly where things end up.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 01 Jan 2009 12:48:49 -0500

+1 (or +several).  From my point of view, we are trying to use 
the same concepts of "multiple addresses per host" and "multiple 
hosts per address" to handle a whole series of unrelated things, 
perhaps with various sorts of virtualization and clustering at 
one extreme and "one address per interface/link" at the other. 
We haven't gotten any of them right for today's world and 
attempts at most of them interfere with the others.

  john

--On Thursday, January 01, 2009 8:00 AM -0800 
[email-masked] wrote:

>> I've been asked twice now in private email to clarify what I
>> mean, so this is going to turn into a massive rant about how
>> the current Internet architecture - as it is deployed, and as
>> it seems to be developing - has a completely broken idea of
>> how to address endpoints. The multiple meanings of the word
>> "multihoming" relate directly to the multiple points in the
>> rant.
>
> Tony, I pretty much agree with everything you say here. There
> really is a pretty serious disconnect between what we seem to
> be able to build and what applications actually need.
>...

---

From: Tony Finch <[email-masked]>
Date: Thu, 1 Jan 2009 21:01:58 +0000

>    I'm not at all clear what "support" of "multihoming" Tony is asking

I'm not clear either, because I don't know what mechanisms could make it
work, especially not mechanisms that are deployable.

But what we need is an addressing architecture that allows us to tell the
difference between a hostname that has multiple addresses because they are
required for application addressing, or because the destination has
multiple points of connection. (I think IPv4 vs IPv6 is a special case of
the latter.) This is another way of looking at the id/loc split.

Then there must be a way for an endpoint to make an informed decision
about which of its links to use (source address) and which of the possible
destination links to use. There also needs to be a way to migrate
connections between the available links either pro-actively for seamless
mobility or re-actively for fail-over.

> guess that Tony is referring to Section 10.5

I was thinking of the whole thing, actually, because it specifies an
uninformed (therefore broken) version of what I wrote above.

>    That is sometimes true, and often not.

Right. My point is that the above seems to be the architectural model, but
actual practice is almost always different.

> seems to be little point in "explaining" how it does this to the

The endpoint doesn't need to know how: it needs to know if a link is
working, or even better, how well it is working compared to the other
alternatives.

> particularly good at load-balancing.

True, but it often works well enough in practice and has been widely used
for 15 years. The reason for talking about it is it's an example of a
widespread practice that goes against the architecture. It is not
documented in an RFC and is not supported by the IETF to the extent that
the IETF feels free to break it (in RFC 3484 section 6 rule 9).

> > resilience

Almost every bit of portable network-capable consumer electronics has the
hardware to benefit from this fix.

>    I guess I don't follow. Indeed, IPv6 hosts that follow RFC [3484]
> will defeat some attempts at load-balancing,

This isn't about load balancing. One example is that RFC 3484 prefers IPv6
to IPv4 even when IPv6 connectivity relies on sub-optimal tunnels. Another
is section 6 rule 1 says a client should "avoid unusable destinations"
without specifying any way for a client to find out which destinations are
usable.

>    I'm not clear what Tony might see as such "support".

Ned's message had some examples. I suppose that SRV records go some way to
fixing the problems with well-known port numbers, so "no support" is an
exaggeration - but we've failed to back-port this fix to older protocols.

> > to be binary or fractional values.

This is RFC 3484 section 6 rule 1 again. It doesn't work in practice which
is why the real world uses load-balancing routers or anycast or whatever.

>    Does Tony have an alternative to suggest?

As I said, it was a rant and not intended to be constructive :-) Far
better minds than mine are working on the problem and I'm following their
work with interest - especially whether the proposed improvements to
addressing and routing help with these application-level problems.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
LUNDY FASTNET: EAST OR SOUTHEAST 5 TO 7. MODERATE OR ROUGH. MAINLY FAIR, RAIN
AT FIRST IN FASTNET. MODERATE OR GOOD, OCCASIONALLY POOR AT FIRST.
_______________________________________________
Ietf mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Rémi Després <[email-masked]>
Date: Fri, 02 Jan 2009 11:14:14 +0100

Tony Finch  -  le (m/j/a) 1/1/09 10:01 PM:

In my understanding, SCTP has a promising  approach for this:

- The DNS provides as usual  a set of locators that may be tried
successively to start an SCTP association, until one succeeds. They may
be those of different hosts (e.g. for load sharing on a per connection
basis).

- Once an SCTP association is established with an SCTP endpoint, both
ends may exchange their lists of alternative locators. These locators
being exclusive of endpoint physical hosts, this is adequate for
multihoming support.

SRV resource records do provide indications for weighted load
balancing, nicely distinct from normal vs backup.

IMO, their use could be extended for multihoming.

For this, alternative locators of a multihomed endpoint would be
compared to those obtained in  SRV RRs that , in the DNS, are given
for the name of this endpoint.

For each address that matches an SRV RR, the weight it indicates can be
used for intelligent load balancing.

Same view.

In addition, if applications use Connect By Name, and resolvers make
SRV queries when names have the service-name format, then:

- applications need not to be concerned

- transport modules can get the right remote application ports.

RD

---

From: "Fleischman, Eric" <[email-masked]>
Date: Mon, 29 Dec 2008 20:42:13 -0800

I spent several years trying to discern what the IETF's Internet
Architecture became when we abandoned our historic architecture and
started growing the middle of the hour glass through middleboxes and
telephony-originated techniques. For several years in the early 2000s I
became convinced that we had no architecture and I lamented that fact.
Recently I realized that we had stumbled upon a common architecture but
hadn't yet realized it -- map-and-encaps is our current Internet
architecture. I believe that the techniques described in Fred Templin's
current I-D trilogy should be re-written to state that those techniques
describe our current de facto Internet architecture.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 30 Dec 2008 16:55:58 -0500

>>>>> "Tony" == Tony Finch <[email-masked]> writes:
    >>> Most networkable consumer electronics ships with at least two
    >>> network interfaces. Host multihoming is only rare because it
    >>> doesn't work.

    >> Why do you say it doesn't work ?

    Tony> The kind of multihoming I am talking about is where you are
    Tony> using multiple links for redundncy or resilience - i.e. the
    Tony> same reasons for site multihoming. I did not mean other kinds

  I think it says something to the extreme *rarity* of host multihoming
(for systems that initiate connections) that there is even this confusion.
  You can be multihomed with one physical interface.  This is rather
more common for machines that mostly accept connections (web servers),
as you can do relatively simple source address based policy routing.
("If it's ISP-A's address in the source field, use ISP-A")
  
  I haven't read every one of the emails in this thread: but where are
the shim6 people?  Maybe they are busy writing drafts + code.

- -- 
]     Y'avait une poule de jammé dans l'muffler!!!!!!!!!        |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [