Subject: RE: Death of the Internet - details at 11

From: [email-masked] (Noel Chiappa)
Date: Mon, 12 Jan 2004 22:03:37 -0500 (EST)

> From: "Tony Hain" <[email-masked]>

    > You seem to have missed the point. ... You will never hear a consumer
    > demanding IPv6 .. You won't hear ISP's demanding IPv6 unless their
    > customers are demanding apps that run over IPv6 (even then the consumer
    > is more likely to use an automated tunnel and make the clueless ISP
    > irrelevant). You won't get new apps unless the development community
    > sees a viable path to personal riches.

Tony, I think you're the one missing the point - although you're half-way
there in your later comments above.

IPv6 simply isn't going to get deployed "as a replacement for IPv4". It's just
not enough better to make it worth switching - and you can flame all day about
how NAT's are preventing deployment of new applications, but I can't run an
SMTP or HTTP server in my house because my provider blocks incoming SMTP
connections unless I pay for business service, and I personally find that a
lot more problematic than the limitations of NAT.

IPv6's only hope of some modest level of deployment is, as the latter part of
your message points out, as the substrate for some hot application(s). Somehow
I doubt anything the IETF does or does not do is going to have any affect on
whether or not that happens.

Of course, that still doesn't get you to the "general replacement for IPv4"
stage. I don't think that goal is reachable - IPv6 just isn't enough better
(i.e. more functionality) than IPv4. It's ironic that one of IPv6's big
concept points ("the IPv4 architecture is fine, we just need to fix a few
details") turns out to be IPv6's Achilles' heel. (Although some of us
predicted that at the time IPv6 was adopted....)

Now if basic IPv6 (and basic IPv6 applications) were reworked so that it's
IMPOSSIBLE to tell, from looking at the packet, what kind of service it's
going to - e.g. by using random TCP ports for applications servers, and having
the ICP include a service name (the field for which is encrypted so
middle-boxes can't read it) to do the rendezvous - that would be the kind of
thing that might interest a few more people.

And while we're at it, you probably want to make it impossible for the ISP to
even tell if it's a TCP SYN, otherwise they'll probably just filter them *all*
out incoming...

    > Continuing work on IPv4 only creates the illusion that it is a viable
    > protocol for application developers to rely on for future income.

Continuing work on IPv6 only creates the illusion that it is a viable protocol
for the network as a whole to rely on for the provision of ubiquitous datagram
substrate service.

In anything even vaguely like its current form, that goal is completely
unreachable for IPv6, and the continued chanting of "IPv6 is the future" only
prevents work on *feasible* upgrades that will allow the continued provision
of ubiquitous datagram substrate service.

	Noel

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 13 Jan 2004 12:37:18 -0600

On 1/12/2004 9:03 PM, Noel Chiappa wrote:

> IPv6's only hope of some modest level of deployment is, as the latter
> part of your message points out, as the substrate for some hot
> application(s). Somehow I doubt anything the IETF does or does not do
> is going to have any affect on whether or not that happens.

Yup, it needs a killer app or feature. Bigger address space was that
feature, but one made moot by NATs.

There are other features (security, etc), but they are end-user oriented
and don't really hold promise to ISPs or the equipment manufacturers (the
simple cost-of-goods factor means that the vendor community has negative
motivation to offer IPv6 in low-end gear). There has to be some kind of
effort to get past these hurdles -- development of a routing service that
makes multi-homing simpler for everybody at a magnitude higher scale, or
convincing vendors that IPv6 in the cheapest gear is in their best
interests, and so forth.

Since the engineers in the IETF tend to hold these kind of marketing
efforts in relatively low regard, the likelihood of any of this changing
is close to nil.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Tony Hain" <[email-masked]>
Date: Tue, 13 Jan 2004 10:45:30 -0800

Noel Chiappa wrote:
> ...

This sounds more like a point to raise with a regulatory agency than a
technical discussion. Your ISP would make you a prime candidate for a
tunnel-broker that will get you past the blockage:
http://www.freenet6.net/

> whether or not that happens.

You may be right, but that only argues that the IETF is irrelevant to the
development of applications. If that is true, why do we have an applications
area full of working groups? My question was really why they are not
recognizing that at this point IPv4 is a dead end, and stopping any work
that assumes an IPv4 substrate? Yes apps are supposed to be agnostic, but we
all know that apps insist on hooks down the stack, because the app developer
believes he can do a better job of managing the network than the OS.
 
> predicted that at the time IPv6 was adopted....)

As you may recall, I really don't care about the bit pattern in the header,
as long as it has enough bits. I have been the pragmatist focused on getting
whatever is implemented actually deployed (we could have had TUBA deployed
long ago, as some of us had the routing infrastructure up before the IPv6
decision). IETF internal conflict has had more to do with any delays than
the market could have. 

> thing that might interest a few more people.

As I recall, several people including Peter Ford were arguing that the
transport layer should be reworked at the same time, because making the
stack and application changes would be a big enough job that people would
only want to do it once. There was a vast outcry in the IETF to make the
minimum number of changes and keep the architecture exactly the same. Now we
find continuing efforts to rework the architecture because various groups
didn't get the 'minor' tweak they wanted. 

> even tell if it's a TCP SYN, otherwise they'll probably just filter them

See tunneling comment above.

> of ubiquitous datagram substrate service.

Clearly work on IPv6 is not preventing work on changing the fundamental
Internet architecture (see multi6). Turning your earlier argument around,
there is nothing the IETF does or does not do that precludes the IETF from
doing something else. What the IETF can influence is public perception (read
that average developer on the street) by where it places a clear focus. From
the outside, the IETF is continuing to work on IPv4, therefore the IETF
doesn't believe it is a dead end. The average developer is not going to take
the time to learn something different unless it is clear the market is going
away from what he already knows. We can debate all day about how much
influence the IETF has on the market, but given the current abundance of
vendor participants, there is a clear path from standards focus to products
available to the marketplace. 

Like it or not, we are at the end of the IPV4 road, and IPv6 is the only
viable alternative. Anything that comes out of rearchitecting the Internet
will be at least another 10 year process. I realize you would like things to
occur much faster than that, but even IPv4 took over 10 years to make it out
of the lab and into mainstream use (this is easy to overlook if you were so
close to the action that you missed the forest for all those pesky trees).
The market will adopt any technology that seems to solve the perceived
problem cheaper than the alternatives. In a world of apps where consumer
clients connect to publicly operated servers, IPv4 w/NAT is the cheapest
approach. In a world with arbitrary app interactions, the cost of managing
code development and NAT configurations/updates will easily exceed the cost
of changing protocols. At the same time, anyone trying to make a sweeping
one-shot change is just asking for pain. There are tools to make the change
reasonably straight forward for the average network manager. Can the same be
said for a completely new architecture?

Tony

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Jan 2004 11:24:36 -0800

Eric A. Hall wrote:

> There are other features (security, etc),

One man's bread (security by defeating outside 'calls' to your address) 
is another man's poison (failed apps. because you defeated outside 
'calls' to your address).

Other than conserving addresses, NAT "features" are basically poison 
resold as bread.

Joe

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 13 Jan 2004 13:52:25 -0600

> Other than conserving addresses, NAT "features" are basically poison 

Heck, I don't even like the conservation feature.

Misguided allocation policies created a false demand. We would have been
better off to run out of addresses than to let gateways 'rescue' us from
our false shortage.

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jan 2004 17:07:31 -0500

--On Monday, 12 January, 2004 22:03 -0500 Noel Chiappa 
<[email-masked]> wrote:

>...
> IPv6 simply isn't going to get deployed "as a replacement for
> IPv4". It's just not enough better to make it worth switching
> - and you can flame all day about how NAT's are preventing
> deployment of new applications, but I can't run an SMTP or
> HTTP server in my house because my provider blocks incoming
> SMTP connections unless I pay for business service, and I
> personally find that a lot more problematic than the
> limitations of NAT.
>
> IPv6's only hope of some modest level of deployment is, as the
> latter part of your message points out, as the substrate for
> some hot application(s). Somehow I doubt anything the IETF
> does or does not do is going to have any affect on whether or
> not that happens.

Noel, I'm slightly more optimistic along at least two other 
dimensions...

(1) As others have pointed out, the knowledge/skill level of a 
typical ISP seems to be on a rapid downslope with no end in 
sight.  There are lots of ways in which that is not surprising, 
but the only realistic solution for someone who needs high 
reliability in that environment is multihoming, and there seems 
to be no hope for multihoming of small-scale networks with IPv4. 
The bad news, of course, is that the IPv6 multihoming ideas that 
don't cause immediate routing catastrophes seem to be about as 
ready/ mature/ deployed today as they were a decade ago, which 
is to say... Not. And that _is_ an IETF problem.

(2) The "no servers unless you pay business rates", and its 
close relative, "you don't get to run VPNs, or use your own 
email address rather than ours" nonsense you and many others are 
experiencing is sort of an old story.  In a competitive market, 
it is also a pretty simple matter of economics:

	* You don't "want" the server and address capability
	enough to pay for it, because you consider it
	excessively more expensive than the cheap "client"
	service.  I go ahead and pay for it, both because I have
	a higher perception of need and because it is still lots
	cheaper, and offers better performance most days, than a
	dedicated DS0 from any plausible ISP I've been able to
	find.
	
	* The difference between those "business rates" and
	whatever you are paying are mostly determined by a "what
	they can get away with" mentality -- we know what the
	marginal operational costs are.   If those prices stay
	high, it is either because there is no alternate
	provider, or because there is (illegal) price-fixing
	going on, or because no one sees a business opportunity
	by operating a business service at a lower margin.  Many
	of us can remember when the solution to "no viable
	Internet dialup service" was "go form a consortium with
	a few friends"... perhaps it is time to do something
	similar with DSL.  Or maybe we would rather whine than
	do something, perhaps because what we have been fed is "good 
	enough".

And, of course, if enough of those sorts of  things happen, it 
starts to put pressure on the address space, and that means IPv6 
unless someone comes up with another viable alternative _really_ 
soon.

No, that doesn't make me a lot more optimistic.  But some...

      john

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Jan 2004 15:04:37 -0800

John C Klensin wrote:
> 
> Noel, I'm slightly more optimistic along at least two other dimensions...
> 
...
> 
> (2) The "no servers unless you pay business rates", and its close 
> relative, "you don't get to run VPNs, or use your own email address 
> rather than ours" nonsense you and many others are experiencing is sort 
> of an old story.  In a competitive market, it is also a pretty simple 
> matter of economics:
> 

That's a specious argument; you declare it nonsense that ISPs charge 
business rates, but you admit that's what they do, and in fact endorse 
it by paying those rates.

>     by operating a business service at a lower margin. 

Or because it will cut into their "business" business, which is more to 
the point. The telcos are maximizing a pair of profits - business and 
consumer, not just consumer. The difference in rates is their attempt to 
create two markets, with the simplest amount of effort on their end.

These arguments could be used to justify any status-quo. Some economics 
environments evolve because of a shared perception, whether that 
perception is complicit or not.

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Tue, 13 Jan 2004 16:08:54 -0700 (MST)

> From: John C Klensin <[email-masked]>

> sight. ...

> 	by operating a business service at a lower margin.  

The second segment seems to ignore the implications of the first segment.
The marginal cost difference between "business" and "residental" is
zilch only if you have the same people running things and interacting
with customers.  Front line tech-support droids that are dumber than
the Windows boxes of residential customer cost a lot less than humans.
If your front line support people know have a clue about the LSRR IP
option, then either your rates are higher than $30/month or you have
customers like us who do most of our own support (and cost our employers
or ourselves a lot more than $30/month for that support).

> 	                                                    Many
> 	a few friends"... 

There are some surviving ISPs that were started and still run that way
least in geographical areas I know about.  Their prices seem to be
higher than the organizations in that race to maximum stupidity.

It is not a coincidence that they have very few internal spam problems.
They are never blacklisted, not even by the second tier spam blacklists,
even when they rent straight modem dial-up ports.  (Third tier DNS
blacklists are kooky 32-bit random number generators.)

> 	                  perhaps it is time to do something
> 	similar with DSL.  

I know people who have done that sort of thing with DSL and 802.11.
However, I fear that idea is generally killed for now by the fact
that IP bandwidth pricing is set by those outfits racing for ultimate
stupidity.  They see IP bandwidth as a loss-leader.

> 	                   Or maybe we would rather whine than

Until people like the individual complaining here that his cable-modem
is listed as a dynamic address are willing to pay for the costs of
real IP service, including the costs of doing more against your spamming
customers than asking blacklists to list your own addresses, there's
not much hope.

We could accept the fact that people who are not willing pay more than
$10-30/month are not interested in the Internet and stop listen to
their whining.  Detroit laughs as people who expect to get Mercedes
for Chevrolet prices.  Why can't we laugh at people who expect to get
real IP service for $10-30/month, or least stop taking their demands
literally?

If cable-modem IP is good enough for you, then you're not interested
in multihoming or even running your own VoIP system.  You might be
happy to have your phones connected to the email and web browser
demark/appliance maintained by your telco/cableco, but you're not really
interested in the Internet.  You lack the interest to be allowed to
run your own servers for anything.

Vernon Schryver    [email-masked]

---

From: John C Klensin <[email-masked]>
Date: Wed, 14 Jan 2004 08:36:28 -0500

--On Tuesday, 13 January, 2004 16:08 -0700 Vernon Schryver 
<[email-masked]> wrote:

>> (1) As others have pointed out, the knowledge/skill level of
>> a  typical ISP seems to be on a rapid downslope with no end
>> in  sight. ...
>
>> ...
> The second segment seems to ignore the implications of the
> first segment. The marginal cost difference between "business"
> and "residental" is zilch only if you have the same people
> running things and interacting with customers.  Front line
> tech-support droids that are dumber than the Windows boxes of
> residential customer cost a lot less than humans. If your
> front line support people know have a clue about the LSRR IP
> option, then either your rates are higher than $30/month or
> you have customers like us who do most of our own support (and
> cost our employers or ourselves a lot more than $30/month for
> that support).

A little hyperbole aside, this is, of course, correct.   My 
observation was based on the observation --however anecdotal-- 
that many or most vendors of "business" DSL or cable modem 
services, at least in the US, provide almost exactly the same 
level of service/ support as they do for their "residential" 
customers.  The only things one gets are more than one address 
(and relatively static ones if the ISP's normal practice is 
dynamic) and unblocking of a number of functions/activities. 
That level of support does not include "a clue about the LSRR IP 
option"; it often barely includes understanding that, if one can 
log into a CPE router and ping the local network but not the 
ISP's first-hop router, it might suggest a problem with the link 
or that remote router.  The level of support you are 
hypothesizing --either from the vendor or so that internal 
experts can get to non-droids quickly-- seems to come, with some 
ISPs, with yet another increment in price to a "business 
premium" or "managed" service; with others, one has to give up 
DSL or Cable and go to, e.g., fractional T1s.

While, despite some other comments, I still consider this 
nonsense -- the marginal prices for the "business service" are 
all out of proportion to marginal costs and my biases are such 
that I don't like the idea of incurring even small extra costs 
to cripple one service to force people to another, more 
expensive one-- they certainly constitute sound business 
practices for the ISPs, at least for as long as they can get 
away with it.

> There are some surviving ISPs that were started and still run
> that way least in geographical areas I know about.  Their
> prices seem to be higher than the organizations in that race
> to maximum stupidity.
>
> It is not a coincidence that they have very few internal spam
> problems. They are never blacklisted, not even by the second
> tier spam blacklists, even when they rent straight modem
> dial-up ports.  (Third tier DNS blacklists are kooky 32-bit
> random number generators.)

None of this, including the higher prices, are surprising. 
Racing to maximum stupidity and, e.g., eliminating all but the 
pretense of technical support, does lower costs and may improve 
volume.

> I know people who have done that sort of thing with DSL and
> 802.11. However, I fear that idea is generally killed for now
> by the fact that IP bandwidth pricing is set by those outfits
> racing for ultimate stupidity.  They see IP bandwidth as a
> loss-leader.

yes.  But then several of them turn around and incur extra costs 
(however small) to filter outgoing and incoming protocols 
sufficiently to force users into "pure client" mode, prevent 
their mail servers from being used as outbound relays unless the 
addresses are theirs or one buys an extra-cost "service", and so 
on.  Those things are not the mark of stupidity, they are the 
mark of a deliberate and considered business practice, however 
ugly we might find it.

> Until people like the individual complaining here that his
> cable-modem is listed as a dynamic address are willing to pay
> for the costs of real IP service, including the costs of doing
> more against your spamming customers than asking blacklists to
> list your own addresses, there's not much hope.

I think we agree... and that was part of my point.

> We could accept the fact that people who are not willing pay
> more than $10-30/month are not interested in the Internet and
> stop listen to their whining.  Detroit laughs as people who
> expect to get Mercedes for Chevrolet prices.  Why can't we
> laugh at people who expect to get real IP service for
> $10-30/month, or least stop taking their demands literally?
>
> If cable-modem IP is good enough for you, then you're not
> interested in multihoming or even running your own VoIP
> system.  You might be happy to have your phones connected to
> the email and web browser demark/appliance maintained by your
> telco/cableco, but you're not really interested in the
> Internet.  You lack the interest to be allowed to run your own
> servers for anything.

Exactly.  Sadly, but...

     john

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Jan 2004 07:36:46 +0900

John,

JCK> but the only realistic solution for someone who needs high
JCK> reliability in that environment is multihoming, and there seems 
JCK> to be no hope for multihoming of small-scale networks with IPv4.

There is not much of a solution, today, for either IPv4 _or_ IPv6.

However there are nearly 10 different proposals under consideration in
the IETF, to deal with multihoming.  Few are restricted to IPv4.

In other words, when there is a serious solution to multihoming -- ie,
being able to preserve a connection when using more than one IP
Address -- it will likely work for IPv4.

Most of these proposals are quite new.  No more than a year old and
many less than 6 months.

This does not speak well for anything happening immediately, of
course.  However quite a number of the proposals do not require any
significant infrastructure change.  This bodes well for rapid
deployment, once they make it through the standards process.

On the other hand, getting the IETF to produce standards track
specifications out of this large pack of candidates could take another
10 years...

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Jan 2004 12:39:39 -0500

Dave,

Just to pick a small nit or three...

--On Wednesday, 28 January, 2004 07:36 +0900 Dave Crocker 
<[email-masked]> wrote:

> JCK> reliability in that environment is multihoming, and there
> seems  JCK> to be no hope for multihoming of small-scale
> networks with IPv4.
>
> However there are nearly 10 different proposals under
> consideration in the IETF, to deal with multihoming.  Few are
> restricted to IPv4.

or to IPv6, which I assume is what you intended.

> In other words, when there is a serious solution to
> multihoming -- ie, being able to preserve a connection when
> using more than one IP Address -- it will likely work for IPv4.

Actually, that definition changes the problem into a much harder 
one, and one that I think is unnecessary for the problem I was 
discussing --unnecessary 99% of the time, if not always.  The 
reality is that there is very little that we do on the Internet 
today that require connection persistence when a link goes bad 
(or when "using more than one IP address").  If a connection 
goes down, email retries, file transfer connections are 
reconnected and the file (or the balance of the file if 
checkpointing is in use) is transferred again, URLs are 
refreshed, telnet and tunnel connections are recreated over 
other paths, and so on.  It might be claimed that our 
applications, and our human work habits, are designed to work at 
least moderately well when running over a TCP that is vunerable 
to dropped physical connections.

Would it be good to have a TCP, or TCP-equivalent, that did not 
have that vunerability, i.e., "could preserve a connection when 
using more than one address"?  Sure, if the cost was not too 
high on normal operations and we could actually get it.  But the 
goal has proven elusive for the last 30-odd years -- at least in 
the absence of running with full IP Mobility machinery all of 
the time, which involves its own issues -- and, frankly, I'm not 
holding my breath.

By contrast, the problem that I find of greatest concern is the 
one in which, if I'm communicating with you, and one or the 
other of us has multiple connections available, and the 
connection path between us (using one address each) disappears 
or goes bad, we can efficiently switch to a different 
combination... even if all open TCP connections drop and have to 
be reestablished in the interim. For _that_ problem, we had a 
reasonably effective IPv4 solution (at least for those who could 
afford it) for many years -- all one needed was multiple 
interfaces on the relevant equipment (the hosts early on and the 
router later) with, of course, a different connection and 
address on each interface.  But, when we imposed CIDR, and the 
address-allocation restrictions that went with it, it became 
impossible for someone to get the PI space that is required to 
operate a LAN behind such an arrangement (at least without 
having a NAT associated with the relevant router) unless one was 
running a _very_ large network.

Now, I'll stipulate this is a routing problem as much, or more, 
than it is an address-availability problem.  And I'll also agree 
that there appears to be little evidence that IPv6 is 
significantly more routing-friendly than IPv4 and hence, that 
any real routing-based solutions that help the one will help the 
other.  But,

	(i) if any of the options turn out to require an
	approach similar to the one that continue to work for
	big enterprises with PI space in IPv4, then we are going
	to need (lots) more address space.  And
	
	(ii) If any of the "multiple addresses per host" or
	"tricks with prefixes" approaches are actually workable
	and can be adequately defined and implemented at scale
	--and there is some evidence that variations of them can
	be, at least for leaf networks-- then they really do
	depend on structure and facilities that appear to me to
	are available in IPv6 and not in IPv4.

So, for the problem I was referring to (but perhaps not for your 
much more general formulation), I stand by my comment and 
analysis.

> Most of these proposals are quite new.  No more than a year
> old and many less than 6 months.
>
> course.  However quite a number of the proposals do not
> require any significant infrastructure change.  This bodes
> well for rapid deployment, once they make it through the
> standards process.
>
> specifications out of this large pack of candidates could take
> another 10 years...

Yes.  And it may speak to the IETF's sense of priorities that 
the efforts to which you refer are predominantly going into the 
much more complex and long-term problem, rather than the one 
that is presumably easier to solve and higher leverage.

     john

---

From: Pete Resnick <[email-masked]>
Date: Wed, 28 Jan 2004 13:09:35 -0600

On 1/28/04 at 12:39 PM -0500, John C Klensin wrote:

>The reality is that there is very little that we do on the Internet 
>today that require connection persistence when a link goes bad (or 
>when "using more than one IP address").  If a connection goes down, 
>email retries, file transfer connections are reconnected and the 
>file (or the balance of the file if checkpointing is in use) is 
>transferred again, URLs are refreshed, telnet and tunnel connections 
>are recreated over other paths, and so on.  It might be claimed that 
>our applications, and our human work habits, are designed to work at 
>least moderately well when running over a TCP that is vunerable to 
>dropped physical connections.
>
>Would it be good to have a TCP, or TCP-equivalent, that did not have 
>that vunerability, i.e., "could preserve a connection when using 
>more than one address"?  Sure, if the cost was not too high on 
>normal operations and we could actually get it.  But the goal has 
>proven elusive for the last 30-odd years -- at least in the absence 
>of running with full IP Mobility machinery all of the time, which 
>involves its own issues -- and, frankly, I'm not holding my breath.

I am rather ambivalent about this issue (it seems like the obvious 
thing to do, but also seems quite painful to accomplish), but I do 
think there is something missing in this response: "The cost" to 
which you refer needs to be weighed against the cost of *not* doing 
so, and that cost seems to have been mounting all along and shows no 
sign of slowing down. The fact is that we have had to engineer all 
sorts of application-layer solutions to this single problem and will 
continue to do so for new application-layer protocols into the 
future. Worse yet, some of those solutions continue to include 
ridiculously high-cost solutions such as having to retransmit entire 
files, and my guess is such costs (bandwidth and otherwise) will 
continue in the future. I also think that the argument ignores the 
possibility that if we do address the "connection persistence" 
problem, we will be able to do many things at the application layer 
that we have always avoided doing because of the cost of having to 
engineer around it. From the view up here in the nosebleed section, 
it seems like it is worth at least the attempt to get a solution.
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated

---

From: Dean Anderson <[email-masked]>
Date: Wed, 28 Jan 2004 15:04:36 -0500 (EST)

> There is not much of a solution, today, for either IPv4 _or_ IPv6.

There is a good solution, and it works for IPv4. But I can't tell you what
it is, since I am making money with that knowledge, and I don't want to 
tell my competitors my business.

But I will give you a hint:  Over-consolidation doesn't scale well.  It
just creates exhorbitant executive salaries, crazy-stupid business plans
funded by VC, and opportunities for undercooked books.  The dinosaurs are
dying, and they don't know why.  To continue the metaphor, I'm small and
furry, and I don't really know why they are dying either. But I know I'm
not in the same boat.

		--Dean

---

From: John C Klensin <[email-masked]>
Date: Wed, 28 Jan 2004 15:05:17 -0500

Pete,

I think the _attempt_ and _effort_ to get a solution to the 
persistent connection problem is entirely worthwhile and did not 
mean to suggest otherwise.  I think that ignoring or delaying an 
easier, and still important, problem while we work the 
persistent connection one borders on irresponsible.  And that 
distinction is the only one I was attempting to make.

We may still disagree, of course.

    john

--On Wednesday, 28 January, 2004 13:09 -0600 Pete Resnick 
<[email-masked]> wrote:

>> The reality is that there is very little that we do on the
>> Internet  today that require connection persistence when a
>> link goes bad (or  when "using more than one IP address").
>> If a connection goes down,  email retries, file transfer
>> connections are reconnected and the  file (or the balance of
>> the file if checkpointing is in use) is  transferred again,
>> URLs are refreshed, telnet and tunnel connections  are
>> recreated over other paths, and so on.  It might be claimed
>> that  our applications, and our human work habits, are
>> designed to work at  least moderately well when running over
>> a TCP that is vunerable to  dropped physical connections.
>>
>> Would it be good to have a TCP, or TCP-equivalent, that did
>> not have  that vunerability, i.e., "could preserve a
>> connection when using  more than one address"?  Sure, if the
>> cost was not too high on  normal operations and we could
>> actually get it.  But the goal has  proven elusive for the
>> last 30-odd years -- at least in the absence  of running with
>> full IP Mobility machinery all of the time, which  involves
>> its own issues -- and, frankly, I'm not holding my breath.
>
> I am rather ambivalent about this issue (it seems like the
> obvious thing to do, but also seems quite painful to
> accomplish), but I do think there is something missing in this
> response: "The cost" to which you refer needs to be weighed
> against the cost of *not* doing so, and that cost seems to
> have been mounting all along and shows no sign of slowing
> down. The fact is that we have had to engineer all sorts of
> application-layer solutions to this single problem and will
> ridiculously high-cost solutions such as having to retransmit
> entire files, and my guess is such costs (bandwidth and
> otherwise) will continue in the future. I also think that the
> argument ignores the possibility that if we do address the
> "connection persistence" problem, we will be able to do many
> things at the application layer that we have always avoided
> doing because of the cost of having to engineer around it.
> From the view up here in the nosebleed section, it seems like
> it is worth at least the attempt to get a solution.

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Wed, 28 Jan 2004 14:51:41 -0600

Dave:

Comments in-line below..

>There is not much of a solution, today, for either IPv4 _or_ IPv6.

This is just plain silly .. there is a perfectly good solution for
multi-homing.. its called SCTP.. been around since Oct 2000 as
an RFC and the SS7 over IP folks are using it now. Yes it works for
both IPv4 and IPv6 and it will even setup an association (connection
for you TCPites) that includes BOTH IPv4 and IPv6.

It fails over and keeps on working when an interface goes down...

>the IETF, to deal with multihoming.  Few are restricted to IPv4.

I have always wondered why we are spinning our wheels in
this multi-homing group when the solution already exists.
Lode Coene has brought this up in the Multi-6 working
group but he seems to be ignored.

You don't need to do anything.. its already there.. just use it.

R

-- 
Randall R. Stewart
815-477-2127 (office)
815-342-5222 (cell phone)

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Wed, 28 Jan 2004 15:00:51 -0600

>> using more than one IP Address -- it will likely work for IPv4. 

Yes.. SCTP solves the problem for V4 and V6 (missed that bit last time).

>>
>
> Actually, that definition changes the problem into a much harder one, 
> and one that I think is unnecessary for the problem I was discussing 
> --unnecessary 99% of the time, if not always.  The reality is that 
> there is very little that we do on the Internet today that require 
> connection persistence when a link goes bad (or when "using more than 
> one IP address").  If a connection goes down, email retries, file 
> transfer connections are reconnected and the file (or the balance of 
> the file if checkpointing is in use) is transferred again, URLs are 
> refreshed, telnet and tunnel connections are recreated over other 
> paths, and so on.  It might be claimed that our applications, and our 
> human work habits, are designed to work at least moderately well when 
> running over a TCP that is vunerable to dropped physical connections. 

Yes.. it is true we have trained folks to hit the reload button... there 
is no
doubt about that :->

> that vunerability, i.e., "could preserve a connection when using more 
> than one address"?  Sure, if the cost was not too high on normal 
> operations and we could actually get it.  But the goal has proven 
> elusive for the last 30-odd years 

John, please go look at RFC2960... it does this.. it is TCP equivalent.. 
and yes the
cost is not to high.

It is available in Linux 2.6, all of the BSD via KAME, Solaris (via 
package I am told), HP
and you can purchase it for many other platforms...

You can find information on this at:

http://www.sctp.org/implementations.html

It always seems to me like typical engineers... we solve the problem
one place and then rush out and try to solve it in yet another place... when
all we have to do is use what is already defined... I guess it is the 
ole NIH syndrom... :-<

> -- at least in the absence of running with full IP Mobility machinery 
> all of the time, which involves its own issues -- and, frankly, I'm 
> not holding my breath.
>
> By contrast, the problem that I find of greatest concern is the one in 
> which, if I'm communicating with you, and one or the other of us has 
> multiple connections available, and the connection path between us 
> (using one address each) disappears or goes bad, we can efficiently 
> switch to a different combination... even if all open TCP connections 
> drop and have to be reestablished in the interim. For _that_ problem, 
> we had a reasonably effective IPv4 solution (at least for those who 
> could afford it) for many years -- all one needed was multiple 
> router later) with, of course, a different connection and address on 
> each interface.  But, when we imposed CIDR, and the address-allocation 
> restrictions that went with it, it became impossible for someone to 
> get the PI space that is required to operate a LAN behind such an 
> arrangement (at least without having a NAT associated with the 
> relevant router) unless one was running a _very_ large network. 

The web server that I mention above has two addresses that are on the 
Big-I. And if
you connect via SCTP to the apache engine .. guess what.. you will use 
both of them..
and no I don't need a large block of lan allocation.. SCTP takes care of 
all of
this for me...

Will it work (multi-homed) behind a nat.. nope.. you reduce down to
singly homed .. but that is the price you pay for NAT..

I suppose one could define a inter-nat protocol so one could support 
multi-homing but
that is a swamp I would not care to walk down...

R

>
>
> Now, I'll stipulate this is a routing problem as much, or more, than 
> it is an address-availability problem.  And I'll also agree that there 
> appears to be little evidence that IPv6 is significantly more 
> routing-friendly than IPv4 and hence, that any real routing-based 
> solutions that help the one will help the other.  But,
>
> So, for the problem I was referring to (but perhaps not for your much 
> more general formulation), I stand by my comment and analysis.
>
> Yes.  And it may speak to the IETF's sense of priorities that the 
> efforts to which you refer are predominantly going into the much more 
> complex and long-term problem, rather than the one that is presumably 
> easier to solve and higher leverage.
>
>     john
>
>
>
>
>

---

From: "Jeffrey I. Schiller" <[email-masked]>
Date: Wed, 28 Jan 2004 16:03:06 -0500

Applications have to deal with more then just losing a
connection. They have to deal with the loss of state that occurs when
you lose a connection. In general you really don't know which
transactions finished and which ones didn't, so you have to re-sync
your state in some way. 

I believe that no matter what we do to provide connection persistence,
connections will still break. Whether they break because of a total
loss of path for a long period of time, or more likely path is lost
for longer then the human timeout (i.e, the human starts typing ^C
[unix] or clicking CANCEL [gui] or whatever).

As long as connections break, applications will have to deal with
getting back to work when a connection can be made again. Given I
don't believe you will ever get "perfect" connection persistence (even
if you can do better then TCP does today) so therefore applications
will always have to have that re-syncing baggage.

		 -Jeff

---

From: John Leslie <[email-masked]>
Date: Wed, 28 Jan 2004 16:20:23 -0500

John C Klensin <[email-masked]> wrote:
> one,

   Preserving a connection when using more that one IP address is
not _necessarily_ a much harder problem -- especially if we
stipulate that tunneling is a legitimate middleware operation.

> today that require connection persistence when a link goes bad... 

   But we certainly _should_ be doing things that would greatly
benefit from connection persistence when a link goes bad.

> It might be claimed that our applications, and our human work
> habits, are designed to work at least moderately well when running
> over a TCP that is vunerable to dropped physical connections.

   Alternatively, one might claim our work habits have "evolved"
to work moderately well...

> goal has proven elusive for the last 30-odd years...

   Might we do well to consider _why_ this is so?

> be reestablished in the interim.

   If I understand, John is looking for applications-level link
redundancy, which strikes me as unlikely to be easy to deploy.

> For _that_ problem, we had a reasonably effective IPv4 solution
> (at least for those who could afford it) for many years -- all
> one needed was multiple interfaces on the relevant equipment
> (the hosts early on and the router later) with, of course, a
> different connection and address on each interface. 

   Aren't we now talking what John said "changes the problem into
a much harder one" -- namely preserving connection when using
more than one IP address?

> But, when we imposed CIDR, and the address-allocation restrictions
> that went with it, it became impossible for someone to get the
> PI space that is required to operate a LAN behind such an

   A /20 is _not_ "very large" -- just impractical to justify for
small-scale projects. (Thus, the allocation policies prevented
much of the small-scale experimentation which normally comes in
the early stages of design.)

> than it is an address-availability problem. 

   I'm not sure I agree. It's true that address-availability
policies were driven by routing problems. One _could_ consider
the route-filtering policies to be "a routing problem", but this
doesn't strike me as useful.

> And I'll also agree that there appears to be little evidence
> that IPv6 is significantly more routing-friendly than IPv4

   Agreed.

> and hence, that any real routing-based solutions that help the
> one will help the other.  But,

   This gives the impression of overstating your case. Indeed, there
_will_ be solutions which require lots more IPv4 space; and there
will be solutions which depend on structure of IPv6. But these will
have to compete with other solutions which need neither.

> much more general formulation), I stand by my comment and 

   I won't attempt to restate your analysis. But I think your
analysis is too narrow. You quite ignore the tricks which many
smaller ISPs can perform -- especially when they cooperate.

   We have a genuine problem in that we'd like something immediately
scalable -- and only larger ISPs can immediately reach large numbers
of users -- and larger ISPs impose arbitrary and capricious limits
on what their customers can do.

   Nonetheless, larger ISPs will have so many problems deploying
IPv6 that I have no confidence they _can_ make it native to their
operations within five years. OTOH, if we need to tunnel it over
IPv4, the situation really isn't different from any other tunneling
over IPv4...

>> old and many less than 6 months.

   I'd love to investigate them...

>> well for rapid deployment, once they make it through the

   I rather suspect many of them could be deployed _before_ they
make it through the standards process. ;^)
 
>> specifications out of this large pack of candidates could take

   Exactly!

> that is presumably easier to solve and higher leverage.

   There is a great tendency to broaden the scope of a project as
you add more people to it; and a very rational tendency to extend
the time-frame being considered as the time-to-approach-consensus
appears to grow. Lamenting this won't change it.

   Where we go seriously wrong is requiring short-timeframe efforts
to coordinate their work with long-timeframe efforts.

--
John Leslie <[email-masked]>

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 28 Jan 2004 23:08:43 +0100

> Yes.. SCTP solves the problem for V4 and V6 (missed that bit last 

That remains to be seen. The list of issues that SCTP has or at least 
seems to have is long. To name a few:

- increased overhead compared to TCP
- requires significant changes from applications
- no backward compatibility of any kind
- source address selection problem isn't addressed fully, if at all

I can't be sure right now, but I also suspect SCTP could very well be 
vulnerable to some of the threats identified lately in the multi6 wg.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 28 Jan 2004 23:28:09 +0100

On 28-jan-04, at 18:39, John C Klensin wrote:

> email retries, file transfer connections are reconnected and the file 
> (or the balance of the file if checkpointing is in use) is transferred 
> again, URLs are refreshed, telnet and tunnel connections are recreated 
> over other paths, and so on.  It might be claimed that our 
> applications, and our human work habits, are designed to work at least 
> moderately well when running over a TCP that is vunerable to dropped 
> physical connections.

This assumes that when address A fails, address B keeps working. This 
is only true when routing is symmetric or the multiaddressed endpoint 
is able to detect the failure. And applications need to retry with 
other addresses. They typically don't do this very well if at all in 
IPv4, and only moderately well in IPv6.

> operations and we could actually get it.

There are several proof of concept multiaddress TCPs.

[sorry for the long quote:]

> relevant router) unless one was running a _very_ large network.

??? Why would you need PI space to be able to give hosts more than one 
address and use those successfully?

And if you had the PI space, why would you bother? Contrary to some 
reports multihoming using independent address space and links to more 
than one ISP works fairly well: failover times are almost always 
shorter than TCP or user timeouts.

> 	to need (lots) more address space.  And

More than what?

>> However quite a number of the proposals do not

>> specifications out of this large pack of candidates could take

Looking at the rate at which the IETF is coming up with ways to 
automatically determine IPv6 DNS resolver addresses I can hardly 
disagree.

But most of the multi6 proposals have large parts in common with other 
proposals. It seems to me that all we have to do is combine the best 
parts. How hard can that be. (Famous last words.)

> easier to solve and higher leverage.

Which would be?

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Wed, 28 Jan 2004 16:47:44 -0600

>> Yes.. SCTP solves the problem for V4 and V6 (missed that bit last time). 
Let me go through these one by one...

> - increased overhead compared to TCP 

Ok lets see. SCTP takes on average 4 more bytes per data packet then
TCP. However, if the TCP implementation enables timestamps then
that is not true and TCP takes more overhead by about 4 bytes...

Unless you are discussing another type of overhead...

> - requires significant changes from applications 

Ok let see.. for Mozilla we converted two lines of code

sd = socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP);
became --------------> sd = socket(AF_INET, SOCK_STREAM, IPPROTO_SCTP);

and

setsockopt(sd, IPPROTO_TCP, TCP_NODELAY, &on_off, sizeof(on_off));
became ---------------> setsockopt(sd, IPPROTO_SCTP, SCTP_NODELAY, 
&on_off, sizeof(on_off));

Now to take advantage of the stream feature you would need to do
more.. but for pure multi-homing one or two lines of change does
not seem that big of deal to me ...

> - no backward compatibility of any kind 

I am not sure what you mean by backward compatable? You definetly
can't have TCP and SCTP talk.. they are after all different protocols...
But if an application needs the redundancy move to it.. its there today
with about 2 lines of coding change...

> - source address selection problem isn't addressed fully, if at all 

I don't think I understand this issue either.. We have fully
addressed source address selection in the KAME implementation. It
is not a difficult problem.. it does require some code.. but any good
implementation must address this issue... And since the site scope
went away in IPv6 (at least for now) its easier to do then it was).

> vulnerable to some of the threats identified lately in the multi6 wg. 

I would have to look at your threats... if you add the dynamic address 
feature.. sure
thats why the document has not progressed.. but that will change when the
Purpose Built Keys instantation in SCTP happens.. hopefully soon.. just 
another
draft to write :-D

I think you may want to have a little closer look at SCTP.. you might
want to get the KAME BSD implementation and play with it for
a bit.. I think you would be amazed at how simple it is to
convert an application and you end up with multi-homing for
free with those 2 lines of code :->

R

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 29 Jan 2004 12:29:26 +0100

> that is not true and TCP takes more overhead by about 4 bytes...

Hm... timestamp option takes 12 bytes, what gives?

> Unless you are discussing another type of overhead...

Apart from the extra bytes (which isn't the main issue) there are the 
CRC32 checksum, which is considerably more expensive to compute than 
the TCP/UDP/IP checksum and the fact that SCTP has much more 
complexity. I would be surprised if the "fast path" for SCTP wouldn't 
turn out to be more expensive than that of TCP, especially without 
disabling all the features that make SCTP more interesting than TCP in 
the first place. And of course those heartbeats I love to hate. (Which 
don't provide any benefits when there are only two paths anyway.)

> Ok let see.. for Mozilla we converted two lines of code

Ok, we can discuss whether the changes are significant but the fact 
that changes are required at all is the main problem.

> &on_off, sizeof(on_off));

So is this enough to gain all multihoming benefits? Even if all other 
magic is hidden inside the protocol, the application must at least try 
all possible destination addresses. (But this is true for several other 
multi6 proposals as well.)

> not seem that big of deal to me ...

Doing WWW-style HTTP (lots of concurrent sessions) with SCTP would be 
nonsense, this is where the feature where multiple streams are 
multiplexed over a single session would be extremely useful.

> with about 2 lines of coding change...

But then you'll have to run services on both TCP and SCTP pretty much 
forever. Not a good thing.

> I don't think I understand this issue either..

Host has addresses a and b from ISPs A and B. Host sends a packet to 
address x which is routed over ISP A. However, the host selects source 
address b which is filtered by A, so no connectivity. For extra credit: 
solve same when this happens in the middle of a session, and in cases 
where connectivity consists of two one way paths. (a -> x and y -> b 
work, but not x -> a or b -> y)

> We have fully
> addressed source address selection in the KAME implementation.

In what way?

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Thu, 29 Jan 2004 06:43:39 -0600

> Hm... timestamp option takes 12 bytes, what gives? 

Ahh.. I never did the direct comparison at the RFCs... just what
I saw in ethereal..

My hosts are BSD based and always send MSS of 1448 bytes. SCTP
uses 1452.

Looking at the RFC's SCTP is 8 bytes more...

Don't think 8 bytes is really a killer.. especially when 4 of those
bytes can be used for the application (PPId)....

And especially since:

1) SCTP is not as subject to the wrap issue so Timestamps are not needed
2) SCTP is not as subject to data and control injection by blind attackers

and these come for free as well..

> don't provide any benefits when there are only two paths anyway.)

As to CRC32c .. yes it is more expensive.. but it affords a lot more
protection then the TCP checksum.  And just like the TCP checksum
there are already chipsets out there that do this in hardware... I
know intel has such a chipset and I am sure you will see it evolve
to other hardware makers as well.

As to the other features... if the application is going to
have to perform them then you have not removed the
cost.. you have just pushed it to a higher layer...

> that changes are required at all is the main problem. 

You don't get new features without change of some sort...

> all possible destination addresses. (But this is true for several 
> other multi6 proposals as well.) 

Yes you get a retransmissions to alternate addresses with this small 
change. And
you don't have to try all the addresses.. just an alternate until you
find one that works...

> multiplexed over a single session would be extremely useful. 

Yes.. if you wanted to do HTTP over SCTP and use features such
as streams you WOULD WANT to extend HTML so that you could
have the client "get pages" on particular streams... Then you could
have parrallel transfers over one association and not have to require
multiple TCP connections... this is optimal...

Jana was doing some investagtion into this at the University of 
Deleware's Protocol
Engineering Lab.. don't know if he will be writting a paper on it.. I 
imagine so...

> forever. Not a good thing.

Just like we are stuck with IPv4 "forever" as well...  Its pretty much
a given.. once a new technology comes along it is a slow gradual 
transition not
a mass move. IPv6 as been out since 1993 and we are not only 11 years into
the migration.. it will all happen eventually.. but I see many years 
left for
IPv4... and only a slow migration to V6...

The same, I think, can be said for SCTP.. and it is very early in this 
process if
you use IPv6 as a gage :-D

> address b which is filtered by A, so no connectivity. For extra 
> credit: solve same when this happens in the middle of a session, and 
> in cases where connectivity consists of two one way paths. (a -> x and 
> y -> b work, but not x -> a or b -> y) 

We have solved this in kame with our alternate routing code. Each time
an address fails more than a threshold (default of 2 rtx's if I remember 
right) we
ask the routing system for an alternate route... This makes it so that after
any two failed transmissions to x routed through B we would then ask for
a new route and get one over A... And yes it works in the the middle
of a session when the ISP makes a change during the conversation.. the
thresholds are not picky about when it happens ...

I might add that TCP could also take advantage of this code and gain
some reliability as well.

Oh, and note, in your scenario above.. if it happens in the middle of
a TCP connection the connection will fail.. or just plain not start
up if it is being filtered and that is the way the routes point.

For our KAME stack with the patch it will NOT fail in the middle...

>> addressed source address selection in the KAME implementation.

See above, you would be best to get the KAME stack, and the patch from
www.sctp.org
(further details are in the alternate routing discussion on the sctp.org 
web site)
and then play with it..

R

---

From: John C Klensin <[email-masked]>
Date: Thu, 29 Jan 2004 08:46:13 -0500

--On Thursday, 29 January, 2004 14:34 +0900 Dave Crocker 
<[email-masked]> wrote:

>...
> JCK> Yes.  And it may speak to the IETF's sense of priorities
> that JCK> the efforts to which you refer are predominantly
> going into the  JCK> much more complex and long-term problem,
> rather than the one  JCK> that is presumably easier to solve
> and higher leverage.
>
> I think a simple version of what you want is already available
> and has been for many years, namely multiple A records in the
> DNS.
>
> (OK.  Let's see you respond to _that_ and claim that the
> response only has picks at nits... Hah!)

Of course, multiple A records works, is out there, and have 
worked for years.  But they worked better before we introduced 
routers (i.e., when the hosts with multiple A records really had 
interfaces on different networks).  Today, it effectively 
implies having multiple addresses on an interface and multiple 
"local" address prefixes running around on the same physical LAN 
segment.  IPv4 was not designed to work well in that environment 
and, with at least some implementations that are arguably still 
conforming, it has some unfortunate side effects, such as not 
really knowing which addresses are local to you and which 
aren't, at least without setting up local routing tables that 
are well beyond the capabilities of the typical user.  The claim 
has been made that IPv6 _is_ designed to work in that 
environment, for whatever that claim may be worth.

Perhaps more important, as Noel points out, it doesn't scale 
very well, at least in terms of the routing fabric.   And, as I 
tried to point out, address preservation policies have had 
trickle-down effects that make it impractical for small 
enterprises.

Nits?  Probably not.

   best,
    john

---

From: Daniel Senie <[email-masked]>
Date: Thu, 29 Jan 2004 09:50:32 -0500

At 07:43 AM 1/29/2004, Randall R. Stewart (home) wrote:
>>CRC32 checksum, which is considerably more expensive to compute than the 
>>TCP/UDP/IP checksum and the fact that SCTP has much more complexity. I 
>>would be surprised if the "fast path" for SCTP wouldn't turn out to be 
>>more expensive than that of TCP, especially without disabling all the 
>>features that make SCTP more interesting than TCP in the first place. And 
>>of course those heartbeats I love to hate. (Which don't provide any 
>>benefits when there are only two paths anyway.)
>
>>Ok, we can discuss whether the changes are significant but the fact that 
>>changes are required at all is the main problem.
>
>

I really don't see how this fits with routing policy. You appear to make 
the assumption that a multi-homed site wants to use the multiple paths as 
equals.

>Jana was doing some investagtion into this at the University of Deleware's 
>Protocol
>the migration.. it will all happen eventually.. but I see many years left for

Ah, so you have indeed made routing policy decisions and placed them into 
the end systems of the originator of the session. This is most interesting. 
A site which has multiple connections with varying cost has to either live 
with the routing policy you've encoded into the stack or force dropped 
packets in some fashion to try to fool your stack.

I'm afraid the multihoming solution you are promoting does not match up 
entirely with the full scope of the multihoming problem. Though I'm sure 
some folks will be relieved to not need to formulate routing policies, it 
does seem a lot of folks really want to be able to control their routing. 
Today, there can be cost differences among circuits (think about circuits 
that are priced based on usage), preferred circuits, and so forth.

>(further details are in the alternate routing discussion on the sctp.org 

The other failing of the dual-address problem is the need for dual 
addresses. I guess we'll have to ask ARIN and the other RIRs to adjust 
their rules to allow justification of parallel allocations from multiple 
upstreams to a single company, where they'd only have needed a single block 
of a given size before. There's no free lunch, at least if there's any 
interest in using SCTP as a multihoming solution atop IPv4.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 30 Jan 2004 08:57:28 +0100

[I'm not going to get into _all_ the details as this probably isn't the 
right place for it.]

> You don't get new features without change of some sort...

Sure. But wouldn't it be better to keep the number of places where 
changes are necessary to an absolute minimum? I still have to use IPv4 
for 90% of my applications despite the fact that all the boxes I 
regularly use run IPv6 because each and everyone application that uses 
the socket API must be changed to support IPv6. Sure, it's only a few 
lines, but progress in this area is extremely slow.

That's why I think it makes more sense to backport the SCTP multihoming 
features to TCP so all TCP apps can use them without having to be 
changed, or even better: contain the changes in a separate shim layer 
so that all transport protocols can become address agile without having 
to be changed.

As far as I can tell, most of the multi6 wg participants are thinking 
along similar lines.

> have the client "get pages" on particular streams...

Why extend the markup language?

HTTP has supported sessions that stay alive so they can be used for 
more than just one operation for many years now. The trouble is that 
you get head of line blockage if you fetch all the elements that make 
up a WWW page in serial fashion. Being able to multiplex different 
streams (within an SCTP session) that all fetch elements simultenously 
would fix this. This of course requires changes to both HTTP clients 
and servers, but it should otherwise be transparent to the user.

> the migration.. it will all happen eventually..

Actually the oldest RFC with IPv6 specs that I know of is from december 
1995, so it's "only" 8 years. And IPv6 implementations from before this 
century (give or take) are too immature to be usable. Even today there 
are some significant hurdles that must be cleared up before any serious 
migration can start.

Anyway, with IPv6 the address format is fundamentally incompatible with 
that of IPv4 so there is no choice but running them concurrently during 
the transition period. A new multihoming-capable transport protocol 
doesn't have to be incompatible with existing TCP so this situation is 
avoidable here.

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Fri, 30 Jan 2004 06:11:56 -0600

Daniel Senie wrote:

>
> I really don't see how this fits with routing policy. You appear to 
> make the assumption that a multi-homed site wants to use the multiple 
> paths as equals.
>

>>> Doing WWW-style HTTP (lots of concurrent sessions) with SCTP would 
>>> be nonsense, this is where the feature where multiple streams are 
>>> But then you'll have to run services on both TCP and SCTP pretty 
>>> much forever. Not a good thing.
>>
>>
>> The same, I think, can be said for SCTP.. and it is very early in 
>> this process if
>>> address x which is routed over ISP A. However, the host selects 
>>> source address b which is filtered by A, so no connectivity. For 
>>> extra credit: solve same when this happens in the middle of a 
>>> session, and in cases where connectivity consists of two one way 
>>> paths. (a -> x and y -> b work, but not x -> a or b -> y)
>>
>>
>>
>> an address fails more than a threshold (default of 2 rtx's if I 
>> remember right) we
> Ah, so you have indeed made routing policy decisions and placed them 
> into the end systems of the originator of the session. This is most 
> interesting. A site which has multiple connections with varying cost 
> has to either live with the routing policy you've encoded into the 
> stack or force dropped packets in some fashion to try to fool your stack. 

No.. it is not the stack that makes the policy.. it is what is in the
routing tables...

Note that all SCTP (or TCP for that matter) does is ask for an alternate.
If the administrator does not have an alternate available.. or deems it
costs to much then it would not be at that level of the table. Also the
admin controls which is priority by order.. .i.e. if I add

route add default 10.1.1.1
and then
route add default 10.2.1.1

Then you will get the 10.1.1.1 first... the signifigance of this is
that you use the alternate route ONLY when you are having
difficulty getting to through to the peer... i.e. T3-rtx's...

So if you were really concerned about cost.. the admin would
need to put the cheapest route in first... in reality my two
routes in my multi-homed box do not cost any more.. they
are equal cost.. I have them since I want to stay connected and
I want it simple.. i.e. I add two default routes and SCTP just
works...

>
>
> I'm afraid the multihoming solution you are promoting does not match 
> up entirely with the full scope of the multihoming problem. Though I'm 
> sure some folks will be relieved to not need to formulate routing 
> policies, it does seem a lot of folks really want to be able to 
> control their routing. Today, there can be cost differences among 
> circuits (think about circuits that are priced based on usage), 
> preferred circuits, and so forth.

I think you need to look at how most of us purchase broadband.. I buy
two DSL's into my house... I don't really care which one is used.. they cost
no more if I let one sit idle then if I send/recv ton's of data. This is the
reality of the DSL for 49.95 a month... there is not cost difference.. only
when I start running T1's and such to my house does cost become an
issue... and I just don't see the masses doing this.. they will wait
for the DSL and cable modem to be in the neighborhood...

>> (further details are in the alternate routing discussion on the 
>> sctp.org web site)
>> and then play with it..
>
>
> their rules to allow justification of parallel allocations from 
> multiple upstreams to a single company, where they'd only have needed 
> a single block of a given size before. There's no free lunch, at least 
> if there's any interest in using SCTP as a multihoming solution atop 
> IPv4. 

I don't see why I would want two addresses to the same companies.. I buy
two DSL's in my location because I know that any single ISP tends to 
fail.. thus
I get two seperate non-related IP addresses... like I have now.. go do
a dig on
stewart.chicago.il.us

The two V4 addresses you see are NOT related... one is SBC/Ameritech the
other is speakeasy..

This requires no changes in any ARIN or RIRs rules...

R

---

From: Dave Crocker <[email-masked]>
Date: Fri, 30 Jan 2004 10:33:11 +0900

John,

JJCK> Of course, multiple A records works, is out there, and have
JCK> worked for years.  But they worked better before we introduced 
JCK> routers (i.e., when the hosts with multiple A records really had 
JCK> interfaces on different networks).  Today, it effectively 
JCK> implies having multiple addresses on an interface and multiple 
JCK> "local" address prefixes running around on the same physical LAN 
JCK> segment.

Multiple addresses is multiple addresses. (I quite the fact that that
is grammatical, in this particular case...) Reliability is affected by
whether they are through different interfaces and through different
network paths, but the degree of host mechanism is really the same.

The fact that the indirection of routed sub-networks is involved is of
course significant, but it is merely the cost of scaling.  Better, it
really is only (did I say "only"?) an administrative cost.  Still, the
fact that a little more administration gives a lot more reliability
strikes me as a fair trade.

JCK> IPv4 was not designed to work well in that environment 
JCK> and, with at least some implementations that are arguably still 
JCK> conforming, ...  The claim
JCK> has been made that IPv6 _is_ designed to work in that 
JCK> environment, for whatever that claim may be worth.

I've heard that claim too and have yet to hear it substantiated,
although I've asked.

JCK> Perhaps more important, as Noel points out, it doesn't scale 
JCK> very well, at least in terms of the routing fabric.

Let's see.  I get a CIDR network address from one provider.  I get
another from another.  I can have my BGP announce both of them all the
time or one of them at a time.  I guess I do not see the horrible
scaling problem.

d/

---

From: Dave Crocker <[email-masked]>
Date: Fri, 30 Jan 2004 10:33:11 +0900

John,

d/

---

From: Daniel Senie <[email-masked]>
Date: Fri, 30 Jan 2004 11:22:37 -0500

We're clearly looking at this issue from VERY different viewpoints. See 
comments inline.

At 07:11 AM 1/30/2004, Randall R. Stewart (home) wrote:
>>the end systems of the originator of the session. This is most 
>>interesting. A site which has multiple connections with varying cost has 
>>to either live with the routing policy you've encoded into the stack or 
>>force dropped packets in some fashion to try to fool your stack.
>

Wrong origination of routing information. I see from comments below you're 
looking at the "I'm an end user and want to have multiple paths to the 
Internet to get more bandwidth or redundancy" aspect. I run a web hosting 
company. We want to be able to have multiple upstream providers and 
multihome to guard against outages at any one provider. We pay for 
bandwidth based on usage, and so may well have several connections, but 
designate one or more to be used only if other connectivity as a last resort.

As the content provider who's spending lots of money on which of MY 
circuits I want traffic to arrive on, I don't want you deciding on your two 
DSL lines to choose one of my sets of addresses or the other. If you stream 
video over a link I don't want used normally, my bills would skyrocket.

This is why I said the decisions in YOUR stack are deciding routing policy 
for MY circuits. In the IPv4/BGP world configuration of this stuff isn't 
clean or simple, but it is possible to encourage the use of some links over 
others. I don't see how I would be able to implement that type of control 
with your simplistic SCTP stack tests, other than to play games with 
discarding or delaying probe packets from your computer on some paths.

>admin controls which is priority by order.. .i.e. if I add

Wrong table. You can control which interfaces YOU use to reach the world. I 
don't see how I can control the weightings and preferences your computer 
will use to reach MY multihomed systems.

>I want it simple.. i.e. I add two default routes and SCTP just

I'm concerned about traffic inbound to my site, and the cost of the various 
pipes I use for that.

>I think you need to look at how most of us purchase broadband..

This is where I got clued in to the fact that we're not talking about the 
same problem.

>  I buy

I don't expect the masses to multihome at all. I do expect most content 
providers to be multihomed, and to want to manage that multihoming. Today, 
it's possible to multhome if you have PI space, or if you get a couple of 
cooperating network providers to punch sufficient holes in their aggregates 
and toss routes to one another (to deal with folks who filter the more 
specifics). A great deal of effort has been expended by many folk to 
provide ways to specify routing policy and path preferences.

If I can't make these types of policy decisions, there's no reason for me 
to implement the type of multihoming you're promoting.

>>upstreams to a single company, where they'd only have needed a single 
>>block of a given size before. There's no free lunch, at least if there's 
>>any interest in using SCTP as a multihoming solution atop IPv4.
>

If I've got a farm of 100 servers, I would need each of them to have an 
address on each provider. Will those providers allow me to obtain more 
address space? Under present rules, I have to justify the space I ask for 
from each upstream. Each of them wants to know how I'm utilizing space from 
the other.

>This requires no changes in any ARIN or RIRs rules...

It sure seems like it would if you're asking for a bigger chunk of space 
from each provider. Will I be able to get a /23 from each of 3 providers 
without running afoul of ARIN rules? I don't know for sure, but suspect it 
would be an issue.

Since you're talking about residential circuits, and I'm talking about fat 
pipes in data centers, it seems the examples each of us had in mind don't 
make sense for the others' arguments.

Dan

---

From: Michael Thomas <[email-masked]>
Date: Fri, 30 Jan 2004 09:09:57 -0800 (PST)

> discarding or delaying probe packets from your computer on some paths.

   Now wait a minute. It's your bandwidth... shouldn't 
   you be _shaping_ his traffic if there's some reason
   he shouldn't be using one path or another? And thus
   provide negative feedback so that he has incentive
   to use the path that you want him to use? After all,
   the end consumer is likely clueless about your
   network and is going to do whatever they can to
   get what they're after -- bandwidth. Sounds to
   me that if he's able to cause your bills go
   through the ceiling _you're_ doing something
   wrong.

		Mike

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Fri, 30 Jan 2004 12:41:21 -0600

Dave:

The scaling problem shows up when you tell the provider of ISP-1
to announce a route for the other address... this effectively
becomes a "host route" being advertised....

But of course the whole point is that we don't need this.. at least
not with SCTP...and even your proposal which is cloned from add-ip
should not require these "host routes"

R

---

From: "Seok J. Koh" <[email-masked]>
Date: Sat, 31 Jan 2004 09:24:28 +0900

> doesn't have to be incompatible with existing TCP so this situation is 

It does not seem reasonable to compare the IPv6/IPv4 deployment and 
SCTP/TCP deployment. SCTP and TCP are basically the end-to-end protocols 
so that hosts/terminals have only to have them, whereas IPv4 and IPv6 are 
subject to network support or change.

IMHO, we don't need to worry about inter-operablity or compatibilty between
SCTP and TCP. I mean SCTP will be used by the new emerging applications and
services including mobile internet stuffs, for example, between mobile clients
and contents servers of the M-ISP.

Seok J. Koh

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 30 Jan 2004 19:41:16 -0500 (EST)

>    through the ceiling _you're_ doing something

Yes, I agree. Afterall, the purpose for a content provider to be
multihomed is so that they can provide more redundancy to their customers.
The content provider only knows about the parameters (capacity, etc) of
its incoming links, but has NO clue about the dynamics/quality of the
customer's path to those links. Any given user may have a different
experience to each of the content provider's incoming links. Thus, the
customer is in the best position to decide which destination is best at
any point in time. So, as Mike said, if the content provider has a
preference, it must encourage/discourage using traffic shapers, etc.

~armando

0--                                              --0
| Armando L. Caro Jr.  |  Protocol Engineering Lab |
| www.armandocaro.net  |    University of Delaware |
0--                                              --0

---

From: "Jeroen Massar" <[email-masked]>
Date: Sat, 31 Jan 2004 01:55:55 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Seok J. Koh wrote:

> SCTP/TCP deployment. SCTP and TCP are basically the 
> end-to-end protocols so that hosts/terminals have only to have them, whereas IPv4 
> and IPv6 are subject to network support or change.

I see a possibility of speeding up SCTP deployment by using semi-NAT,
what about PT, Protocol-Translation, this could be accomplished using
a BIA (Bump In the API) approach.

Scenario: Clients connects to server.
Client's TCP/SCTP stack sees a connect() using TCP, it changes the
protocol field to SCTP and just tries to connect(), if it fails it
uses the normal TCP protocol.

Server starts to listen on TCP. It checks to see if the same port
for SCTP is already in use, if not, it also binds to SCTP.

Thus any client will also try to use sctp and any servercode will
automatically bind to SCTP, voila free multihoming code.
Putting this in the defacto Linux kernel will instantaniously have
a major deployment as generally people who keep up with new kernels
also enable the new toys in those kernels.

Or did someone already draft or even implement this? :)

Greets,
 Jeroen

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 30 Jan 2004 20:18:01 -0500 (EST)

> Or did someone already draft or even implement this? :)

I have thought about this idea many times, but didn't really have feel of
feasible it is.

~armando

0--                                              --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 30 Jan 2004 20:14:47 -0500 (EST)

> so that all transport protocols can become address agile without having

This _kind_ of a solution has already been proposed by Joe Touch and Ted
Faber in their ICNP 97 paper, "Dynamic Host Routing for Production Use of
Developmental Networks". It works, but on of the problems is that you hide
path information from the transport. TCP maintains congestion control
information about a single destination, with the assumption that the path
will not change during the connection. If it does occasionally, then it's
fine. However, with multihoming, the change may be a common occurance
throughtout the lifetime of a connection depending on the application and
the use of the multiple paths (failover, concurrent multipath transfer,
etc). So TCP (or whatever transport) should not be blind to the fact that
data is potentially going over a different path. Otherwise, the congestion
control parameters/algorithms will not work properly.

Also, since the transport is maintaining per path information, it is in
the best decision to decide which path to use. Yes, a shim layer could
cooperate with the transport to get this info, but then you are changing
TCP anyway.

> As far as I can tell, most of the multi6 wg participants are thinking

I must admit that I haven't following the multi6 too closely yet, but I
plan to catch up and follow. But from what I have seen, it seems that SCTP
is ignored on the sidelines. _Maybe_ SCTP doesn't solve _all_ the
problems, but I think SCTP should be evaluated more closely before
developing yet another protocol. ...because I believe that no matter how
much we don't want to touch TCP or IP, something has to be changed to get
all this functionality. In my opinion, the transport is the easiest to
change, since they live on the endpoints and operating systems do
eventually get upgraded.

> Why extend the markup language?

I think that was a typo...

> and servers, but it should otherwise be transparent to the user.

This is essentially what Jana did. I don't really know the details of his
changes, but he exploited SCTP's multistreaming to avoid head of line
blocking. Sourabh Ladha did similar work with FTP.

Sourabh Ladha, Paul D. Amer,   Improving Multiple File Transfers Using
SCTP Multistreaming , IPCCC 2004, Phoenix AZ.
http://www.eecis.udel.edu/~ladha/2003.MultistreamedFTP.pdf

> doesn't have to be incompatible with existing TCP so this situation is

No, it doesn't have to, but as I said above... something has to
change. Without change, you're limited to the improvements you can make.

~armando

0--                                              --0

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Fri, 30 Jan 2004 20:29:20 -0500 (EST)

> shorter than TCP or user timeouts.

I assume you mean failovers from first hop links failing, because
end-to-end failures cannot be detected faster than a timeout.

~armando

0--                                              --0

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 30 Jan 2004 22:06:24 -0600

Armondo - nice discussion.

Just a couple of notes on one of my favorite topics...

Spencer

----- Original Message ----- 
From: "Armando L. Caro Jr." <[email-masked]>
To: "Iljitsch van Beijnum" <[email-masked]>
Cc: "Randall R. Stewart (home)" <[email-masked]>;
"[email-masked]' Discussion" <[email-masked]>
Sent: Friday, January 30, 2004 7:14 PM
Subject: Re: Death of the Internet - details at 11

> control parameters/algorithms will not work properly.

Yeah, you might think that. I did, when we proposed TRIGTRAN (how
could finding out that a link failed be WRONG?).

The problem was that when we were discussing notifications like
PATH_CHANGE (or whatever we ended up calling it), the TCP has a
decision to make when it sees PATH_CHANGE - "do I

- slow start, recognizing that I now haven't got the remotest clue
what the path capacity is, or

- maintain my congestion window and try to actually LOSE something
before I decide that I need to adjust?"

TCPs have been ignoring things like ICMP Unreachable for decades, so
the second idea isn't totally bogus - they respond to actual loss, not
to the potential for loss.

Once you decide that your path change is no different than cross
traffic starting and stopping, which we adjust to after losing
something, transports ignoring path changes makes a lot of sense. If
you are changing paths frequently (and round-robin would be the
limiting case), slow-starting every time you change paths is
excessive - and, if you're going to respond to PATH_CHANGE, what else
would you do? You could do congestion avoidance during the first round
trip, or actually lose something and do congestion avoidance on the
second round trip - not a big difference, conceptually.

[deleted down to]

> I think that was a typo...

Commonly-deployed browsers don't pipeline HTTP requests because they
don't get any advice on (1) size of objects or (2) recommended
retrieval order. Extending HTML to provide one or the other would make
at least as much sense as extending HTTP.

> > and servers, but it should otherwise be transparent to the user.

As above - fetching all elements simultaneously makes perfect sense to
transport guys, but can give pretty flaky-looking pages during
retrieval. If you think HTTP/1.1 with two connections makes sense, you
might start retrieving the smallest elements on one connection and the
largest on the other, and meet in the middle, but since the browser
has no idea how to do this, it's pretty difficult to resist the
temptation to wait for a connection to become idle before sending
another request.

And this is why HTTP over cellular links has horrible throughput - if
you stop-and-wait request 300-byte objects with one-second RTTs, your
throughputp looks a LOT like 300 bytes per second...

---

From: James Seng <[email-masked]>
Date: Sat, 31 Jan 2004 13:19:24 +0800

Pricing and architecture of the Internet: Historical perspectives from 
telecommunications and transportation, Andrew Odlyzko

http://www.dtc.umn.edu/~odlyzko/doc/pricing.architecture.pdf

would give you very new perspective to all these so-called "threats".

-James Seng

Spencer Dawkins wrote:

>>This _kind_ of a solution has already been proposed by Joe Touch and
> 
> Ted
> 
>>Faber in their ICNP 97 paper, "Dynamic Host Routing for Production
> 
> Use of
> 
>>Developmental Networks". It works, but on of the problems is that
> 
> you hide
> 
>>path information from the transport. TCP maintains congestion
> 
> control
> 
>>information about a single destination, with the assumption that the
> 
> path
> 
>>will not change during the connection. If it does occasionally, then
> 
> it's
> 
>>fine. However, with multihoming, the change may be a common
> 
> occurance
> 
>>throughtout the lifetime of a connection depending on the
> 
> application and
> 
>>the use of the multiple paths (failover, concurrent multipath
> 
> transfer,
> 
>>etc). So TCP (or whatever transport) should not be blind to the fact
> 
> that
> 
>>data is potentially going over a different path. Otherwise, the
> 
> congestion
> 
>>>HTTP has supported sessions that stay alive so they can be used
> 
> for
> 
>>>more than just one operation for many years now. The trouble is
> 
> that
> 
>>>you get head of line blockage if you fetch all the elements that
> 
> make
> 
>>>streams (within an SCTP session) that all fetch elements
> 
> simultenously
> 
>>>would fix this. This of course requires changes to both HTTP
> 
> clients
>

---

From: "Armando L. Caro Jr." <[email-masked]>
Date: Sat, 31 Jan 2004 13:06:47 -0500 (EST)

> second round trip - not a big difference, conceptually.

I understand your point, and I have heard this argument before when I
brought up the same issue in various Mobile IP meetings/talks. But I don't
agree. How is this behavior different from simply beginning a new TCP
connection with a large cwnd?

Also, if multihoming is supported at the transport so that the transport
is aware of the different paths (cwnd, ssthresh, etc), then there is a
potential of doing concurrent multipath transfer (aka load balancing, load
sharing, etc) at the transport layer.  Otherwise, CMT cannot be done
well... at least not any better than an application load balancing across
multiple connections. Jana Iyengar is doing work in this area. I cc'd him
in case he wants to jump in here.

~armando

0--                                              --0

---

From: arifumi <[email-masked]>
Date: Sun, 1 Feb 2004 04:10:31 +0900

X-Mailer: Sylpheed version 0.9.8claws (GTK+ 1.2.10; i386--netbsdelf)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

(Resend)
Hi Mike,

On Fri, 30 Jan 2004 09:09:57 -0800 (PST)

>    through the ceiling _you're_ doing something

I guess Daniel's problem is not that simply solved.
Just the shaping isn't enough, because the backup path should
be fully used only when the main path has some "trouble".
Moreover, "trouble" isn't also such a simple thing like
a direct link failure, such as fibre cut, but it includes
yet upper link(ISP)'s failure.
This is not even a problem of "OK" or "Not OK".
I mean, those cases are very likely to happen that you can
communicate with only a few of ASes through the main path.

So, though you may say "we can stop the shaping if we have
troubles in the main path," it is not such easy.
You have to change the bandwidth of backup path's shaping
depending on the number of routes passed from main path's
upstream ISP.

I'm not familiar with BGP routing techniques that much,
but I guess these policies can be implemented, right ?

Though I believe end-to-end multihoming is the way to go,
I don't have a good answer for this problem.

-- 
/arifumi
[email-masked]

---

From: arifumi <[email-masked]>
Date: Sat, 31 Jan 2004 19:46:41 +0900

Hi Mike,

-- 
/arifumi
[email-masked]