Subject: Re: ISP failures and site multihoming [Re: Enforcing unreachability of site local addresses]

From: Pekka Savola <[email-masked]>
Date: Thu, 20 Feb 2003 21:53:47 +0200 (EET)

On Thu, 20 Feb 2003, Iljitsch van Beijnum wrote:
> On Thu, 20 Feb 2003, Pekka Savola wrote:
> > > We are _very_ far from a situation where even the best ISP provides a
> > > service level that is better then the one you get from multihoming even
> > > if you consider failover delays.
> 
> > In some cases, this may be better.  In some others, not.
> 
> > It's not IMO necessary to get significantly better but "roughly equal".
> 
> Don't forget that failover is just one of the benefits of multihoming.
> Two important others are protection against losing service when ISPs
> go out of business 

Sure, independence is another issue -- not related to ISP failures, and 
there are ways to protect against this.

> and more optimal traffic flow.

I agree, but other than really simple traffic flow balancing (read: 
multiple addresses) is a difficult thing to do, and people don't often 
really do it.
 
> > > And the single service provider thing doesn't scale anyway: the end
> > > result would have to be a single global ISP.
> 
> > It does scale, pretty well actually.  I'm not talking about your average
> > neighborhood ISP's with 100 customers, though.  Currently in DFZ, there
> > are about 3500 (ONLY!) AS numbers which transit at least one other AS
> > number.
> 
> I don't see your point. What you were saying is that using a single
> reliable ISP would be better than multihoming.

Considering the tradeoffs, yes.

> Now obviously an ISP can
> only control the QoS parameters inside its own network so if you want to
> do reliable and high quality VoIP you need to use the same ISP
> end-to-end. In other words: just one ISP.

This seems like a ridiculous argument or I'm missing something.  You could
just use VoIP up to the ISP, have ISP's coordinate the parameters (if you
need any), etc.  This is no different than multihomed scenario.
 
> > > Has the end-to-end principle failed to teach us anything? Reliability
> > > begins and ends in the end hosts. If each host is connected over two
> > > service providers there are four possible paths the hosts can switch
> > > between on a per-packet basis. Then the only problem becomes detecting
> > > failure. The end hosts are in an excellent position to do this without
> > > having to generate keepalive messages; a well designed protocol could
> > > switch to an alternate path within a few round trip times when a path
> > > failure occurs.
> 
> > Compare this to a solution where the site has two connections to the same
> > ISP, and you're left with major ISP backbone failure or upstream failure
> > (any relevant ISP's have only one upstream)?
> 
> So ISPs get to multihome but not end-users?

Yes.
 
> There are many ways in which an ISP network can fail, as the large scale
> Worldcom and AT&T outages six months ago illustrate. 

I'm not aware of the case, so I can't really comment.  Pointers?

> More common is the
> situation that an ISP network has trouble reaching a certain destination
> because the only link to that destination has failed (which in itself
> may not be their fault) or there is congestion somewhere. 

This seems no different than the case when using BGP site multihoming -- 
unless you want the fine-tune your policy per destination -- a 
non-starter.

> And don't
> forget maintenance windows.

No real ISP has maintenance windows that seriously affect all
communications at once.
 
> > A solution without multi-connecting, ie. only one L1 connection to one
> > ISP, is naturally out of question.
> 
> Ok, so why is multihoming to a single ISP better than multihoming to
> several ISPs?

Fewer tradeoffs: you can protect against most failure modes, while not 
having to hae AS, your own IP addresses etc -- that is, it doesn't require 
bloating the DFZ!
 
> > > Multi6 has been gravitating towards multi-address multihoming solutions
> > > for a while now, but unfortunately it seems impossible to move foward.
> 
> > Multi-address solutions solve certain problems well, but leave some
> > unsolved.
> 
> Like what?

Solves: operator independence

Doesn't solve: connection survivability, short-term failures, more than 
basic TE [mostly a non-issue IMO]

Mixing multiconnecting to one ISP and having a backup to the second one
gives you mostly everything.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------

---

From: "Bound, Jim" <[email-masked]>
Date: Fri, 21 Feb 2003 12:47:39 -0500

Folks,

I am on multi6 and just listen and send clarification questions to
various participants.  But a short comment here.

> Has the end-to-end principle failed to teach us anything? 
> Reliability begins and ends in the end hosts. If each host is 
> connected over two service providers there are four possible 
> paths the hosts can switch between on a per-packet basis. 
> Then the only problem becomes detecting failure. The end 
> hosts are in an excellent position to do this without having 
> to generate keepalive messages; a well designed protocol 
> could switch to an alternate path within a few round trip 
> times when a path failure occurs.

I agree with the above view and much of the problem can be solved by
intelligent code and configuration being done by the end system for new
IPv6 development and some existing network application infrastructure
that is a direct port from IPv4 that has been ported and deployed.  SCTP
is a clear winner for us here over TCP and it's a new API so we could go
there.  But more importantly the transition to this requires code on the
end systems from suppliers, and then either apps change or shims are
built on the end systems. Doing it with TCP is possible till SCTP is
more dominant.  This is solving the problem at layer 7 and 4. And that
means we need new platform code releases and development to make it
happen.  Which is fine but will take time.

Layer 1 and 3 may be able to do something too but that will take time
and require new code and platform releases.

I think we need to first pick which way we want to specify this or both
and provide technical spec on what each mean.

It is really transparent to IPv6 but IPv6 has a better chance of getting
this right.  

> Multi6 has been gravitating towards multi-address multihoming 
> solutions for a while now, but unfortunately it seems 
> impossible to move foward.

Hmmm.  At some point this needs to get done folks.  Not sure what will
cause it, but I believe it may be external forces resolve it for us.

P.S. Mutli6 should not MUST SCTP as Diameter tried (well was forced)
that and all the Diameter products shipping are doing it with TCP and
same for some other things like RDMA. But Multi6 could do the community
a big favor by stressing that SCTP really helps this problem within its
inherent failover capabilities via the network association for the
connections.

Regards,
/jim

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Mon, 24 Feb 2003 09:27:21 +0100

>>> There is no technical reason why a single service provider network 
>>> can
>>> do better than a similar network that consists of several smaller
>
>> See Abha and Craigs paper on convergence of BGP. Personally I would go
>> for a large provider with multiple connections.
>
> Based on this paper? What I see is rarely as bad as what they describe.
> However, I had the chance to experiment a little with revoking a
> longer prefix and then see how soon the shorter prefix would "catch" 
> the
> traffic a while back, and this was certainly interesting: the state 
> goes
> back and forth between "working" and "not working" several times over
> the course of two minutes. But simple failover is usually pretty fast.

My experience is different, and I believe many others share that. But 
this will always be different for everyone.

>> Last fall I was invited to a conference in Sweden to debate 
>> multihoming
>> and the enterprise. Before me was this enterprise IT manager who 
>> showed
>> how much more resilient his network was with two BGP sessions. While 
>> he
>> talked I checked his announcements just to find that one of the
>> providers bought transit from the other. You can't buy clue.
>
> You can buy a good book that explains it all.  :-)
>
> Did you check to see if the second ISPs also had additional upstreams?

Yes they did. And they bought transit from the first provider.

>
>>> But IGPs have the same
>>> fundamental problem (although the details may differ). OSPF for
>>> instance takes 40 seconds to detect a dead circuit.
>
>> there was a fix proposed in San Diego (although for IS-IS) but that 
>> was
>> voted down. There was pros and cons.
>
> Just type:
>
>  ip ospf hello-interval 1
>  ip ospf dead-interval 3
>
> But do it on ALL your boxes in the subnet or you'll live to regret it.

This I thought was more or less standard. I was talking about less than 
100ms convergence.

- kurtis -

---

From: "Tom Petch" <[email-masked]>
Date: Mon, 24 Feb 2003 15:55:16 -0000

To talk about resilience is a snare for the unwary until you specify
what you are providing resilience against.

The large enterprises I work with have a simple policy, set by the
non-technicians who run the organisation; no single point of failure.
That doesn't mean no single point of failure in the ISP - that is up
to the management of the ISP - but no single failure in the parts the
enterprise management is accountable for, such as the links to the
ISP.  So if one ISP piggy backs on another, that is the choice of the
management of the ISP and so not their responsibility.

In the non-IP world, that invariably means taking bandwidth, basic
carrier services, to two different telcos, coming out of different
parts of the site, to different exchanges, perhaps with different
media (copper and fibre, fibre and wireless), not having a power
supply (eg local electricity supplier) in common.  If in the centre of
the world these feeds share a fibre, that is the telco's concern, not
the enterprise's.  The enterprise should ensure that the telcos do not
share a local exchange or fibre to the site but no more than that.

This is a system that has evolved over decades with much pain but it
does work well because the bulk of failures -  some surveys put it as
high as 95% - occur within the 'last mile' and that is what this gives
resilience against.

IP? same difference.  It is the links to the ISPs'  points of presence
(obviously not co-located) that are the primary concern of the
enterprise management.  If ISPs share a resource, which in a sense
every ISP does because there is a single world-wide BGP RIB, then that
is the responsibility of the ISP.

So multi-homing is a must.

Tom Petch, Network Consultant
[email-masked]

-----Original Message-----
From: Kurt Erik Lindqvist <[email-masked]>
To: Iljitsch van Beijnum <[email-masked]>
Cc: Pekka Savola <[email-masked]>; Alan E. Beard
<[email-masked]>; Tim Chown <[email-masked]>;
[email-masked] <[email-masked]>;
[email-masked] <[email-masked]>
Date: 24 February 2003 08:29
Subject: Re: ISP failures and site multihoming [Re: Enforcing
unreachability of site local addresses]