Subject: [RRG] Are we solving the wrong problem?

From: "Mark Handley" <[email-masked]>
Date: Tue, 19 Feb 2008 05:35:09 +0000

I've been mulling this around for a long time now, and I think we may
be trying to solve a number of problems at the wrong layer.

The routing problems we're trying to solve seem to stem primarily from
an inability to perform aggregation of routing information.  There are
a number of causes for this, including:

 - historical allocation policies (the swamp).
 - inappropriate information hiding in BGP.
 - multihoming.
 - traffic engineering.

The first is not a serious problem, though may become one when we run
out of IPv4 addresses.  The second is one that RRG might take a
serious look at.  I've made one attempt at this with HLP, but other
solutions are certainly possible too, even without replacing BGP.  But
the current concerns mostly seem to stem from multihoming and traffic
engineering.

I believe that if we are ever to make routing scale a great deal
better, we should not be attempting to solve these last two problems
primarily within the routing system.  Clearly if you present routing
people with a problem, we'll come up with routing solutions, but if
you take a broader view, we can probably do much much better (Frank
Kelly deserves credit for the original idea I'm going to suggest). I'm
slowly writing a more detailed document discussing what I think the
solution space should be, but I'll try and give the general idea with
the simple example of a site that is dual homed to two ISPs.

Current practice attempts to hide this dual homing behind a single IP
address for each host.  This then requires a long prefix to be
advertised via both ISPs, with appropriate AS prepending to balance
traffic.  If either edge-link goes down, on average half the Internet
gets to process the routing update.  Worse, this doesn't do a great
job of load balancing, so prefix splitting is sometimes performed to
better balance load, resulting in even more stress to the global
routing system.  In short, attempts at improving local robustness
create global stresses, and potentially global fragility, which is the
problem we're all concerned with.

So, what happens if we stop trying to hide the multihoming.  Take a
server at this multi-homed site and give it two IP addresses, one from
each provider's aggregated prefix.  Now we modify TCP to use both
addresses *simultaneously* - this isn't the same as SCTP, which
switches between the two.  The client sets up a connection to one
address, but in the handshake learns about the other address too.  Now
it runs two congestion control loops, one with each of the server's IP
addresses.  Packets are shared between the two addresses by the two
congestion control loops - if one congestion-controlled path goes
twice as fast as the other, twice as many packets go that way.

OK, so what is the emergent behaviour?  The traffic self-load balances
across the two links.  If one link becomes congested, the remaining
traffic moves to the other link automatically.  This is quite unlike
conventional congestion control, which merely spreads the traffic out
in time - this actually moves the traffic away from the congested path
towards the uncongested path.  Traffic engineering in this sort of
scenario just falls out for free without needing to involve routing at
all.  And more advanced traffic engineering is possible using local
rate-limiting on one path to move traffic away from that link towards
the other.  Again, this falls out without stressing routing.

Now, there's quite a bit more to it than this (for example, it's great
for mobile devices that want to use multiple radios simultaneously),
but there are also still quite a lot of unanswered questions.  For
example, how much does this solve backbone traffic engineering
problems?  The theory says it might.  I'm working on a document that
discusses these issues in more depth.  But I think the general idea
should be clear - with backwards-compatible changes to the transport
layer and using multiple aggregatable IP addresses for each
multi-homed system, we ought to be able to remove some of the main
drivers of routing stress from the Internet.  That would then leave us
to tackle the real routing issues in the routing protocols.

I hope this makes some sort of sense,

Mark

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Brian Dickson <[email-masked]>
Date: Mon, 18 Feb 2008 22:17:18 -0800

> be trying to solve a number of problems at the wrong layer.
[snip]
> But
> primarily within the routing system.
Me too, for what it's worth. There are many different directions from 
which to tackle the multihoming
problem, and the more we collectively attack it, the better the chances 
are we'll find a solution
that is good enough, scales well, and can be incrementally deployed.
>  Take a
I think one difficulty is when you extended that for connections between 
two boxes,
with respectively N and M addresses. What do you do? N x M connections, 
or something else?

There are important issues to consider... packet loss can be caused by 
congestion, which responds
to back-off, or by errors, which does not, and in the extreme cases can 
drastically affect TCP.
(Think about error rates where 0.1% of packets have CRC errors, or 1%, 
or 30%.)

Edge links are often treated as duplex, but are really pairs of simplex 
links, with asymmetric properties,
including possibly speed (DSL is a good example), rate limits (cable), 
congestion, errors, etc.

Perhaps if the methods used for pushing bits over modems with multiple 
frequencies at the same time,
like telebit did with its trellis stuff, were used on the N x M 
"channels" in each direction, adaptively,
the desired (described) effect could be achieved.

However, a few other considerations are:
* Duration vs benefit - how long are flows compared to the time needed 
to reach "optimal" performance?
If the average flow is not long enough, and the behavior isn't common to 
all flows on a per-link basis,
this system may perform considerably below expected levels, or levels of 
other schemes.
* Congestion versus unreachable interfaces - if state is hidden by 
aggregation, start-up phase may be adversely
impacted. On the other hand, what is the cost of not hiding that state, 
and keeping/advertising that state information?

And there can be cases where the *client* is multihomed, and the server 
is not... and so on...

One philosophical aspect to the whole issue can be summarized by the phrase:
Routing sucks.
Advertisements of routes attract traffic, in other words.

So, TE issues should generally be controlled by the entity that controls 
routing advertisements.
This is something to consider when contemplating "solutions" which 
places TE into the mutual
control of both parties, rather than unilateral control of the announcer 
- it has the potential to upset
business models for how cost is attributed and/or controlled.
> I hope this makes some sort of sense,
Yes, it is clear to me.

I hope it is also clear that we are all attacking the problem space, not 
each other. :-)

Brian Dickson

---

From: "Mark Handley" <[email-masked]>
Date: Tue, 19 Feb 2008 06:39:53 +0000

> with respectively N and M addresses. What do you do? N x M connections,

Yes, I gave a deliberately simple example,  as email isn't the best
place to reason design complexities.

Of course you want to support multi-homed clients and servers, so you
have the NxM issue, plus both N and M will be dynamic if both ends are
mobile (which I expect they increasingly will be).  The fluid flow
models would say use all NxM, because that gives you the best traffic
engineering.  My gut feeling is that that's probably unnecessary,
though it will require research to know for sure.

As a rule of thumb, I think you need to separate TE for inbound
multihoming from that for outbound links.  The former is a global
problem if you involve routing, whereas the latter is always purely
local, so easier to solve.

One option is to use only one of your active source addresses at a
time in packets you send, so essentially you have N subflows one way
and M the other.  If you do this, the outbound traffic engineering is
a local routing problem and the inbound is a congestion management
problem.

Another option is to simply pair up addresses from the two sides, and
use source-address-based routing locally to reach the right exit
router from the sending site.  Then you don't get the full
flexibility, but likely its enough to do a reasonable job of TE at
both sides without involving routing.

Overall, I think we're in some sort of agreement.  Anyway, my point
isn't that we know the _best _ way to do this, but that there is a
whole space of techniques that should be able to lessen the stresses
on global routing.  Then we can get back to solving routing problems
with routing protocols and traffic management problems with traffic
management protocols.

Cheers,
Mark

---

From: Eliot Lear <[email-masked]>
Date: Tue, 19 Feb 2008 08:16:16 +0100

Mark,

> twice as fast as the other, twice as many packets go that way.

You're certainly challenging the orthodox notion that the routing system 
provides you the best path to get from one point to another.  I don't 
see anything wrong with that.  But I also don't see anything wrong with 
improving the routing system to provide the same function.

> In short, attempts at improving local robustness
This comment goes to the nature of state.  I wonder if you believe it 
would hold true if we could change that nature by creating two buckets, 
multilateral and bilateral states, and dumping most state in the 2nd 
bucket?  Could we do so and provide optimal routing?  What mechanisms 
would be needed?  Would they conflict with the approach you take above?

Eliot

---

From: Geoff Huston <[email-masked]>
Date: Tue, 19 Feb 2008 21:08:39 +1100

Eliot Lear wrote:
> provides you the best path to get from one point to another.

I can't see that Eliot - it seems close to Shim6 in some ways, although 
here Mark is advocating a "shim' at the session / transport level of the 
protocol stack rather than at the IP level. I have my concerns at the 
robustness and effeciency of a session level approach and see more merit 
at the IP level, but thats a personal perspective. But thats orthogonal 
to the properties of a routing system.

(By the way I don't necessarily believe that the routing system 
necessarily provides the "best" path. A routing system is a distributed 
computation environment whose aim is to provide a consistent outcome for 
all forwarding elements.)

>  I don't 

I just re-read section 7 of RFC 3221 (Architectural approaches to a 
scalable Exterior Routing Protocol, dating from 2001). I honestly don;t 
think we've poushed much beyond the state described then. Personally I 
think Mark is right to question the scaleability of "monolithic" nature 
of inter-domain routing when a single distributed computation is applied 
across the cross product of topology and policy, and whether you attempt 
to delay some of the computation, or partition it I'm not sure that the 
total information processing load has altered thereby.

Geoff

---

From: Eliot Lear <[email-masked]>
Date: Tue, 19 Feb 2008 11:21:27 +0100

Geoff,

I think we're quibbling, but...

>>
>> You're certainly challenging the orthodox notion that the routing 
>> system provides you the best path to get from one point to another.
>
> I can't see that Eliot - it seems close to Shim6 in some ways, 
> although here Mark is advocating a "shim' at the session / transport 
> level of the protocol stack rather than at the IP level. I have my 
> concerns at the robustness and effeciency of a session level approach 
> and see more merit at the IP level, but thats a personal perspective. 
> But thats orthogonal to the properties of a routing system.

Mark is attempting to solve multihoming and TE further up the stack.  
The problem is currently handled at layer 3 by the routing system, and 
many proposed solutions in this group focus on that layer.

> necessarily provides the "best" path. A routing system is a 
> distributed computation environment whose aim is to provide a 
> consistent outcome for all forwarding elements.)

I didn't define the term.  In fact, as you know very well, "best", is in 
the eye of the beholder, but obviously part of the definition should 
include consistency.  A routing system that produces results that are 
optimal or fair to no one would not be deployed.

>
>>  I don't see anything wrong with that.  But I also don't see anything 
>> wrong with improving the routing system to provide the same function.
>
> scalable Exterior Routing Protocol, dating from 2001). I honestly 
> don;t think we've poushed much beyond the state described then. 
> Personally I think Mark is right to question the scaleability of 
> "monolithic" nature of inter-domain routing when a single distributed 
> computation is applied across the cross product of topology and 
> policy, and whether you attempt to delay some of the computation, or 
> partition it I'm not sure that the total information processing load 
> has altered thereby.

What I attempted to question was whether we could break the "monolithic" 
nature of inter-domain routing into smaller byte-sized chunks and then 
partition the problem such that some portions aren't quite so 
"monolithic".  But again, I don't see anything wrong with pursuing both 
avenues...

Eliot

---

From: Scott Brim <[email-masked]>
Date: Tue, 19 Feb 2008 06:41:46 -0500

Mark, a few thoughts ...

First there's the problem of discovery of multiple ways to address a
destination.  Do I assume correctly you would build on DNS?  TE is
more than load balancing.  Sometimes policy rears its head, and
sometimes policy is specific to traffic type, not just IP layer info.
This needs to be factored in somehow, and DNS can't support it.  We
could also do an initial discovery handshake between endpoints once
they reach each other at any address.  That would also allow policy to
influence negotiation, and possibly policy intermediaries (a la
NUTSS).  Of course, it would introduce delay :-p.

> twice as fast as the other, twice as many packets go that way.

What about (1) synchronization of oscillations and (2) packets
arriving very out of order, since the source will continually be
testing every allowed path.

What about multicast? :-)

Scott

---

From: "Mark Handley" <[email-masked]>
Date: Tue, 19 Feb 2008 14:55:52 +0000

> to the properties of a routing system.

Well, it's not really orthogonal, because they have very different
emergent behaviour, so leave different amounts of work for the routing
system to do.

The specific reason for layer 4 is that you need to stripe data across
multiple *congestion controlled* paths for the traffic to
self-traffic-engineer. Only then do the bytes to be transfered
actually move away from the congested paths to the uncongested paths
in a stable manner.  I don't believe you can get this unless the
congestion control part of the stack is aware of the different
paths/addresses and load-balancing across them.

Cheers,
Mark

---

From: "Mark Handley" <[email-masked]>
Date: Tue, 19 Feb 2008 15:08:24 +0000

> NUTSS).  Of course, it would introduce delay :-p.

You could use DNS, but I'm not convinced this is the best way because
it doesn't fit well with using the same mechanisms to support
mobility.  Better I think to simply add the extra addresses as part of
the TCP sync handshake.

I agree policy is very important.  These mechanisms actually enable a
whole load of local policy tools that are not currently possible by
shaping one path to move traffic to other paths.

> testing every allowed path.

Synchronization should be no worse than we get with current TCP flows.
 RED is good for these issues.

The TCP connection will have to put packets back in order, but since
TCP is aware of the subflows, it shouldn't be making the wrong
deductions (as it currently does with reordering).  This does of
course require TCP protocol changes, but they're incrementally
deployable and can be negotiated on a per-connection basis.

> What about multicast? :-)

That's one of the things I don't have a clue about :-)

- Mark

---

From: David Williamson <[email-masked]>
Date: Tue, 19 Feb 2008 08:28:04 -0800

> twice as fast as the other, twice as many packets go that way.

Interesting idea, but not all critical traffic is TCP.  My site, for
example, depends on UDP based applications.  My customers (generally
Fortune 100 companies) aren't really interested in routing scale
problems, but just want the service I offer to work.  That requires me,
in the current world, to multihome a whole pile of PI allocations.
While my announcements are pretty efficient, I'm clearly one of the
problem sites you try to address, yet your solution is inapplicable for
the vast majority of traffic.

A simple approach might be to try to stripe UDP traffic across all
available paths.  I hope it is obvious why this is a bad idea with
real-time traffic.

For me, at least, your solution is only a half-solution.  Interesting,
but I think I agree with Geoff that this needs to get fixed in the
network layer. 

-David

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 08:48:05 -0800

Earlier, David Williamson wrote:
% Interesting idea, but not all critical traffic is TCP.  My site,
% for example, depends on UDP based applications.  My customers
% (generally Fortune 100 companies) aren't really interested in routing
% scale problems, but just want the service I offer to work.  That
% requires me, in the current world, to multihome a whole pile of
% PI allocations.

It would be helpful if one could write up an I-D describing
one's operational practice and its objectives.  That would
help inform everyone of one's desires.

% While my announcements are pretty efficient, I'm clearly one
% of the problem sites you try to address, yet your solution is
% inapplicable for the vast majority of traffic.

There are multiple proposals on the table here.  That one
proposal might not be suitable for one's situation does not
imply that all solutions are not suitable for one's situation.

% A simple approach might be to try to stripe UDP traffic across
% all available paths.  I hope it is obvious why this is a bad idea
% with real-time traffic.
%
% For me, at least, your solution is only a half-solution.  Interesting,
% but I think I agree with Geoff that this needs to get fixed in the
% network layer.

Again, one non-network-layer proposal not being suitable
to some requirements does not imply that all non-network-layer
proposals will not be suitable to requirements.

Please let us be careful not to over generalise about possible
solutions; each deserves individual evaluation and consideration.

Cheers,

Ran Atkinson
[email-masked]

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 09:37:11 -0800

Earlier, Scott Brim wrote:
% First there's the problem of discovery of multiple ways to address
% a destination.  Do I assume correctly you would build on DNS?
% TE is more than load balancing.  Sometimes policy rears its head, and
% sometimes policy is specific to traffic type, not just IP layer info.
% This needs to be factored in somehow, and DNS can't support it.  We
% could also do an initial discovery handshake between endpoints once
% they reach each other at any address.  That would also allow policy to
% influence negotiation, and possibly policy intermediaries (a la
% NUTSS).  Of course, it would introduce delay :-p.

It would be helpful if a problem statement (or several of them)
could be written up as I-Ds.  That would permit a more complete
explanation of different scenarios and would facilitate thoughtful
analysis by folks in the RG.

For example, I found the recent draft from BT folks very helpful
in understanding their perspectives and concerns.

Ran

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 19 Feb 2008 18:38:40 +0000

Hi, Mark.

I think you have come to a similar conclusion to the one described in 
the mail that I sent to the RAM list back in December 2006 (attached).

Since then I have been struggling with the issue of how to deal with 
sub-flows where the traffic is carried over multiple paths through the 
network.

You are contemplating a shim6/(real) sctp (i.e. not the 'interim' 
solution which choses 1 of n paths) solution using multiple addresses.

I was thinking about an IP layer solution which could identify the 
sub-flows in a useful but low cost and non-revealing way without much 
involvement from the sending host, but would not rquire multiple 
addesses at the transport level.

My best thought on this so far was for each router through which the 
traffic passes to take a suitable field for the packet (part of the flow 
label field in IPv6 would seem a good idea), hash the value with some 
suitable fixed but randomized value that is characteristic of the router 
and write it back into the packet.  When the packet arrives at the 
destination this field would contain the signature of the path that the 
packet had taken.  Assuming deterministic hash functions and a fixed 
initial value related to the exit path from the source domain that is 
also passed along to the destination, the 'ack' could return the values 
which would enable the source to determine which paths were working best.

My feeling is that this would not require a large number of bits (maybe 
8 for each part, giving up to 256 exit paths/routes through the 
network)and there is (at least for IPv6 ) I believe an incremental 
deployment solution.

The solution is still very much a work in progress.

Regards,
Elwyn

> I hope this makes some sort of sense,

Subject: Traffic engineering and the network model
From: Elwyn Davies <[email-masked]>
To: [email-masked]
Date: Thu, 21 Dec 2006 17:29:00 +0000

A while back
Pekka Nikander wrote:
> Noel Chiappa wrote:
>> When I look at what the problem seems to be ..., it seems to be 
>> growth in the size of the routing table (with dynamics of routing 
>> table entries coming in second). The two main drivers of growth seem 
>> to be i) multi-homing, and ii) general entropy and lack of 
>> aggregation. Is this correct?
>
> I've been told that traffic engineering is another major factor.  
> However, I have to admit not understanding what people mean with it.  
> (A good reference would help.)
Networks use traffic engineering to maximise the utility (in the 
economic sense) of their network assets.  To do this they have to limit 
the accepted portion of the offered traffic and direct the accepted 
traffic towards its intended destination subject to a number of 
constraints, especially:
- Capacity and availability of plant (routers, links etc)
- Policies (business, regulatory, political, etc.)
- Contractual (customer SLAs, transit agreements, etc)

It was clear from the IAB Routing and Addressing workshop, and also from 
other discussions which I have had, that traffic engineering is very 
important to network operators and some of the ways in which it is done 
utilize the capabilities of the routing system (deliberate deaggregation 
of prefixes in particular) so that it contributes significantly to the 
growth of the core routing tables.

Traffic engineering only becomes 'interesting' if there are multiple, 
comparably effective paths between many pairs of points in the network:  
traffic engineering on a simple network where there is exactly one path 
between a pair of (end-)points degenerates into limiting the accepted 
traffic to the capacity of bottleneck link - the classical model for 
which TCP is optimized!

The core network today is increasingly 'meshy' for economic and 
robustness reasons, and the desirable but inexorable growth of traffic 
means that link capacity is increasingly well- or over-used.  
Accordingly traffic engineering is an essential tool for network owners 
and operators.

Observation 1: Multihoming is a form of traffic engineering.

On the architecture discussion list there was some limited amount of 
thought about the model we use for the network in the core.  I would 
like to take this a bit further and see how the network model fits with 
the classical and current reality, and how it interacts with 
multihoming/traffic engineering.

Application View:
=================
Seen from the point of view of an application running on a node, the 
network *internet* layer provides, at its most abstract:
- a path for packets from 'here-to-anywhere' which is not necessarily 
reliable
- unconstrained capacity for delivery of packets

There are no assumptions about how the network implements the path and 
there are no assumptions about constraints on the size of the path.  In 
particular
- there is no assumption about uniqueness of path (in time or topology)
- there is no specific guarantee of ordered delivery

Aside: At this level, the distinction between a 'user' application 
running on an end-point and the 'routing application' running at 
intermediate points is that the user application expects symmetric 
'anywhere-to-here' connectivity whereas the routing  application doesn't.

Interestingly, the transport layer may make additional assumptions about 
and requirements on the internet layer that are not specifically 
provided by the classical model of the internet layer:
- the transport layer (e.g., TCP) may assume that the network is of 
limited capacity
- the transport layer may make relatively strong assumptions about 
ordered delivery

The constraints applied by the transport layer reflect the classical 
model of Internet routing where there was a unique best path which had a 
bottleneck segment that might change relatively slowly with time, but 
would generally be stable during the lifetime of most application flows.

The development of the modern Internet with multihoming and traffic 
engineering across multiple equally capable paths appears to be well 
matched to the basic assumptions (no assumption of unique path, 
unconstrained capacity) on the internet layer but struggles to meet the 
constraints applied by the transport layer.

Routing View
============
The routing system in the classical model:
- assumes there is a single best path from here-to-anywhere at any given 
time
- does not worry about capacity constraints

Adapting the routing system to the newer model and the real world has 
required a number of hacks such as the TE extensions (constraining the 
single best path due to capacity limits) and ECMP allowing some 
utilization of multiple parallel paths. Overall this has not been very 
satisfactory and it is certainly not an architecturally pure solution.  
Moreover the distribution of traffic across multiple paths by ECMP 
typically relies on hashing the whole of the destination and maybe the 
source addresses and other info in the packet to ensure that the 
transport layer constraints on ordering are met.  In many cases this 
splitting is very fine grained and relies on using more information than 
is required for routing.  This may limit our ability to do information 
hiding in the routing system.

The deliberate disaggregation of prefixes is a response to the need to 
force the routing system to distribute traffic over multiple paths 
leading to the same destination where the specific tools are inadequate.

So what...
==========
Observation 2: The routing system today is not a very good match for the 
reality of the network model today.  This is partly caused by a mismatch 
between the assumptions at the internet layer and the transport layer, 
and partly by the development of meshy networks.

In today's meshy network, it is not just multihomed end sites that see 
multiple useful routes from here-to-anywhere.  Many routers in the core 
network could, if the routing system allowed it, also see multiple 
useful routes.  I would therefore claim that traffic engineering and 
multihoming contain a common problem that does not just manifest itself 
at the edge of the network.

Observation 3: A good solution to the routing (scalability) problem 
needs to embrace the availability of (and need to use) multiple routes 
between both edge and interior points in the network.

The id/locator split solution which we have been looking at attempts to 
hide this situation at one particular locus in the network (somewhere 
between the end host and the first provider edge), rather than solving 
it generally.  It would presumably leave the traffic engineering problem 
in the core using the existing routing based techniques.

In essence, unlike today's network, where the internet layer uses 
reasonably uniform routing techniques throughout and crafts traffic 
engineering/multihoming solutions from the same tools everywhere, the 
network would be partitioned, with the edge and core using different 
techniques to achieve the required traffic engineering.

Observation 4: Adopting multiple different solutions to the 
multihoming/traffic enginering problems at different places in the 
network is likely to lead to interactions between the solutions.

One area of interaction that I can see is the need to extract 
information from inner headers in core routers to correctly execute ECMP 
and other classification schemes if a form of encapsulation is used.  
This will increase the processing and memory bandwidth burden on core 
routers, particularly during the transition.

Observation 5: Disguising a problem rather than solving it will likely 
lead to needing a more complex solution in the future.

There is a significant risk that network growth and increases in core 
connectivity will lead to the routing table size problem reasserting 
itself due to the use of complex traffic engineering.  The problem would 
thus be postponed rather than solved, and the result might be the need 
to combine solutions rather than using a common one.

Conclusion:
===========
Whilst it is highly likely that the id/loc split is a good idea, we 
shouldn't assume that it is a panacea for the multiple path problem.  A 
uniform routing solution which manages the existence of multiple paths 
may well constrain the growth of the routing tables better than multiple 
partial solutions and provide a solution to the multihoming aspects of 
the core and edge problems.

In the longer term we need to look at the assumptions of the network 
model we are using and determine if we can modify them to make the 
multiple path problem more tractable.

---

From: Dow Street <[email-masked]>
Date: Tue, 19 Feb 2008 11:09:29 -0800

Elwyn,

I have also been thinking about this type of path hash - a  
(compressed) record of the router interfaces the packet has  
traversed.  If the receiver saw the hash value change over the course  
of a session, it would be a hint that the path changed:

- restart congestion control / RTT measurements?
- off-path attacker spoofing packets?
- persistent oscillation of hash values -> load-balanced links  
somewhere (watch for packet re-ordering)

It may make sense to have the hash done per router *interface*.  If  
we're making host changes, something like this is probably worth  
considering as a part of the information exchange between network and  
endpoint.

R,
Dow

On Feb 19, 2008, at 10:38 AM, Elwyn Davies wrote:

> Hi, Mark.
>
> I think you have come to a similar conclusion to the one described  
> in the mail that I sent to the RAM list back in December 2006  
> (attached).
>
> Since then I have been struggling with the issue of how to deal  
> with sub-flows where the traffic is carried over multiple paths  
> through the network.
>
> sub-flows in a useful but low cost and non-revealing way without  
> much involvement from the sending host, but would not rquire  
> multiple addesses at the transport level.
>
> My best thought on this so far was for each router through which  
> the traffic passes to take a suitable field for the packet (part of  
> the flow label field in IPv6 would seem a good idea), hash the  
> value with some suitable fixed but randomized value that is  
> characteristic of the router and write it back into the packet.   
> When the packet arrives at the destination this field would contain  
> the signature of the path that the packet had taken.  Assuming  
> deterministic hash functions and a fixed initial value related to  
> the exit path from the source domain that is also passed along to  
> the destination, the 'ack' could return the values which would  
> enable the source to determine which paths were working best.
>
> My feeling is that this would not require a large number of bits  
> (maybe 8 for each part, giving up to 256 exit paths/routes through  
> the network)and there is (at least for IPv6 ) I believe an  
> incremental deployment solution.
>
> Date: December 21, 2006 9:29:00 AM PST
> To: [email-masked]
>>> table entries coming in second). The two main drivers of growth  
>>> seem to be i) multi-homing, and ii) general entropy and lack of  
>> However, I have to admit not understanding what people mean with  
>> it.  (A good reference would help.)
> economic sense) of their network assets.  To do this they have to  
> limit the accepted portion of the offered traffic and direct the  
> accepted traffic towards its intended destination subject to a  
> number of constraints, especially:
> It was clear from the IAB Routing and Addressing workshop, and also  
> from other discussions which I have had, that traffic engineering  
> is very important to network operators and some of the ways in  
> which it is done utilize the capabilities of the routing system  
> (deliberate deaggregation of prefixes in particular) so that it  
> contributes significantly to the growth of the core routing tables.
>
> Traffic engineering only becomes 'interesting' if there are  
> multiple, comparably effective paths between many pairs of points  
> in the network:  traffic engineering on a simple network where  
> there is exactly one path between a pair of (end-)points  
> degenerates into limiting the accepted traffic to the capacity of  
> bottleneck link - the classical model for which TCP is optimized!
>
> robustness reasons, and the desirable but inexorable growth of  
> traffic means that link capacity is increasingly well- or over- 
> used.  Accordingly traffic engineering is an essential tool for  
> network owners and operators.
>
> On the architecture discussion list there was some limited amount  
> of thought about the model we use for the network in the core.  I  
> would like to take this a bit further and see how the network model  
> fits with the classical and current reality, and how it interacts  
> with multihoming/traffic engineering.
>
> Application View:
> =================
> Seen from the point of view of an application running on a node,  
> the network *internet* layer provides, at its most abstract:
> - a path for packets from 'here-to-anywhere' which is not  
> necessarily reliable
> There are no assumptions about how the network implements the path  
> and there are no assumptions about constraints on the size of the  
> path.  In particular
> Interestingly, the transport layer may make additional assumptions  
> about and requirements on the internet layer that are not  
> specifically provided by the classical model of the internet layer:
> The constraints applied by the transport layer reflect the  
> classical model of Internet routing where there was a unique best  
> path which had a bottleneck segment that might change relatively  
> slowly with time, but would generally be stable during the lifetime  
> of most application flows.
>
> engineering across multiple equally capable paths appears to be  
> well matched to the basic assumptions (no assumption of unique  
> path, unconstrained capacity) on the internet layer but struggles  
> to meet the constraints applied by the transport layer.
>
> Routing View
> ============
> - assumes there is a single best path from here-to-anywhere at any  
> given time
> Adapting the routing system to the newer model and the real world  
> has required a number of hacks such as the TE extensions  
> (constraining the single best path due to capacity limits) and ECMP  
> allowing some utilization of multiple parallel paths. Overall this  
> has not been very satisfactory and it is certainly not an  
> architecturally pure solution.  Moreover the distribution of  
> traffic across multiple paths by ECMP typically relies on hashing  
> the whole of the destination and maybe the source addresses and  
> other info in the packet to ensure that the transport layer  
> constraints on ordering are met.  In many cases this splitting is  
> very fine grained and relies on using more information than is  
> required for routing.  This may limit our ability to do information  
> The deliberate disaggregation of prefixes is a response to the need  
> to force the routing system to distribute traffic over multiple  
> paths leading to the same destination where the specific tools are  
> inadequate.
>
> So what...
> ==========
> Observation 2: The routing system today is not a very good match  
> for the reality of the network model today.  This is partly caused  
> by a mismatch between the assumptions at the internet layer and the  
> transport layer, and partly by the development of meshy networks.
>
> In today's meshy network, it is not just multihomed end sites that  
> see multiple useful routes from here-to-anywhere.  Many routers in  
> the core network could, if the routing system allowed it, also see  
> multiple useful routes.  I would therefore claim that traffic  
> engineering and multihoming contain a common problem that does not  
> just manifest itself at the edge of the network.
>
> needs to embrace the availability of (and need to use) multiple  
> routes between both edge and interior points in the network.
>
> The id/locator split solution which we have been looking at  
> attempts to hide this situation at one particular locus in the  
> network (somewhere between the end host and the first provider  
> edge), rather than solving it generally.  It would presumably leave  
> the traffic engineering problem in the core using the existing  
> routing based techniques.
>
> engineering/multihoming solutions from the same tools everywhere,  
> the network would be partitioned, with the edge and core using  
> different techniques to achieve the required traffic engineering.
>
> information from inner headers in core routers to correctly execute  
> ECMP and other classification schemes if a form of encapsulation is  
> used.  This will increase the processing and memory bandwidth  
> burden on core routers, particularly during the transition.
>
> Observation 5: Disguising a problem rather than solving it will  
> likely lead to needing a more complex solution in the future.
>
> There is a significant risk that network growth and increases in  
> core connectivity will lead to the routing table size problem  
> reasserting itself due to the use of complex traffic engineering.   
> The problem would thus be postponed rather than solved, and the  
> result might be the need to combine solutions rather than using a  
> common one.
>
>
> Conclusion:
> ===========
> shouldn't assume that it is a panacea for the multiple path  
> problem.  A uniform routing solution which manages the existence of  
> multiple paths may well constrain the growth of the routing tables  
> better than multiple partial solutions and provide a solution to  
> the multihoming aspects of the core and edge problems.
>
> In the longer term we need to look at the assumptions of the  
> network model we are using and determine if we can modify them to  
> make the multiple path problem more tractable.
>
>
>
>

---

From: "Mark Handley" <[email-masked]>
Date: Wed, 20 Feb 2008 05:01:54 +0000

> available paths.  I hope it is obvious why this is a bad idea with

For UDP, clearly a lot depends on the application.  In general, L4
multihoming/mobility should work well for UDP too, so long as the
flows are congestion controlled, and so long as the delay of the
slowest path is acceptable.

For non-congestion controlled traffic, there's no control loop to
react to congestion and cause the apps to move the data to the other
paths.  You can probably get crude balancing, but there's nothing to
reduce the overload on the congested paths.  Likely any crude
congestion control would be sufficient to get benefits though if the
flows are long-lived.

For real-time traffic, the real question is one of delay, not
reordering.  Such apps need a playout buffer anyway to remove jitter,
and this should remedy reordering too (if it doesn't already do this,
various wireless links will badly affect your apps).  For streaming
media, there's no real problem, but for interactive real-time apps,
the effects of the jitter buffer are effectively to increase the delay
to that of the slowest path.  The likely solution would be to have a
delay threshold, and only use paths that are less than that threshold.
 However, there are a lot of unanswered questions about the stability
of such solutions.  Only research will tell what the best solutions
here are.

Cheers,
Mark

---

From: "Victor Grishchenko" <[email-masked]>
Date: Wed, 20 Feb 2008 14:29:26 +0500

Yes, I completely agree. I considered this approach in the TAMARA concept as
"vectorized TCP".
There is a strong reason to follow this path: as the internet is still a
(broken) end-to-end network, only the end points may have exhaustive
information on relative performance of different paths *for free*. More
precisely, by end-points I mean the TCP code at the end-point hosts,
because, obviously, it already does the calculations and keeps the data. Any
entities in the middle may just guess.
Finally, if end-point TCP stacks do some congestion control decisions why do
not they do some routing decisions?
This vectorized TCP / real SCTP approach complements prefix-bunch
architectures well.

-- 
Victor

---

From: Pekka Nikander <[email-masked]>
Date: Thu, 21 Feb 2008 13:04:39 +0200

Mark,

In broad terms, I agree with your assessment and architectural proposal.

One specific comment re the following:

> paths/addresses and load-balancing across them.

A few years ago I waved my hands and claimed that one should take a  
fresh look at congestion control, basically dividing it into a path- 
specific and session-specific parts.  The former might best be  
located at the current layer 3, somewhere where shim6 and hip live  
today, while the other probably belongs closer to the application,  
e.g. "upper part" of layer 4.

Pekka Nikander, "ACCCoRD: Architectural Considerations on the  
placement of Congestion Control Routines and Data -- Extended  
Abstract," in Proceedings of High Speed Networking Workshop,  
Budabest, Hungary, May 17-18, 2004, Budabest University of Technology  
and Economics.
http://www.tml.tkk.fi/~pnr/publications/hsn2004.pdf

Unfortunately, I haven't really done any real work in the area.  So,  
I may be grossly mistaken.  And there definitely are some interesting  
"cross-layers" issues involved.

--Pekka Nikander

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 3 Mar 2008 22:30:58 +0100

On 19 feb 2008, at 19:38, Elwyn Davies wrote:

> sub-flows where the traffic is carried over multiple paths through  
> the network.

> traffic passes to take a suitable field for the packet (part of the  
> flow label field in IPv6 would seem a good idea), hash the value  
> with some suitable fixed but randomized value that is characteristic  
> of the router and write it back into the packet.  When the packet  
> arrives at the destination this field would contain the signature of  
> the path that the packet had taken.  Assuming deterministic hash  
> functions and a fixed initial value related to the exit path from  
> the source domain that is also passed along to the destination, the  
> 'ack' could return the values which would enable the source to  
> determine which paths were working best.

By some strange coincidence I've been giving this problem some thought  
as well recently. But I was looking to attack the problem from the  
other direction: let a host write a value in the packet header that  
routers can use in their path selection process. For IPv6, an obvious  
candidate for this would be the flow label, another would be the  
diffserv bits. Even if DSCPs are used for other applications, it's  
likely that a few can be found that hosts can use to tell routers that  
the packet in question should take a different path than the default.

The sender would then correlate the ACKs that come back with the  
packets that were sent and the paths that they were sent over and  
maintain congestion state separately for each path.

---

From: Jari Arkko <[email-masked]>
Date: Tue, 04 Mar 2008 13:13:35 +0200

Mark,

These are interesting thoughts and worthwhile to be looked at in more
detail. Do you have a proposal?

You are basically going into the direction that is similar to Shim6 in
the sense that multihoming is exposed, not hidden. But also different
from Shim6 in the sense that it operated at layer 3, and from what I
understand your goal was to actually develop new transport layers that
would be capable of taking the advantage of such exposed multihoming.

I suspect that load balancing of a single connection over multiple paths
is a tough problem. How much research has there been on that, and do we
have results that would already be usable for something like that? I.e.,
is this solvable in the near future?

In any case, what I find interesting in this space is the different
design tradeoffs. A routing system that hides multihoming and provider
independence from the endpoints is easy for the endpoints and edge
networks. I.e., you do not have to change hosts in any way, every
network has a single prefix, renumbering is not necessary,
providers/network owners are in control of what kind of multihoming and
TE is going on, etc.

But it also makes the routing system more expensive, because it has to
maintain a lot of information. Many of the RRG people are searching for
a better organization of this information so that its maintenance would
be cheaper -- but you are actually looking at removing some of this
information. I guess the main question is, can we substantially reduce
the costs of the routing system while keeping the same amount of
information and functionality in it? I'm not sure I know the answer yet.
Another drawback of the hiding approach is that it might be ultimately
less capable, if you consider things like hosts being able to react on
transport layer timescales to congestion and their own communication
demands.

Jari

---

From: Scott Brim <[email-masked]>
Date: Tue, 04 Mar 2008 09:12:14 -0800

> information and functionality in it? I'm not sure I know the answer yet.

The problem, as usual, is all the policy stuff, e.g. the various kinds 
of traffic engineering.  I do not see that adding capability to 
endpoints will reduce what you have to do at the border router 
significantly.  In Mark's approach the host uses multiple addresses 
intelligently but the border router still has to advertise reachability 
for them, manage what traffic takes which path, and so on.

> transport layer timescales to congestion and their own communication

Could you be more specific?  We have Mark's approach, with load 
balancing according to experienced throughput.  The only thing routing 
could do to disrupt that would be to dramatically change routing 
frequently.  Or are you thinking of "vertical coupling" approaches where 
something up at session layer can ask the network for changes in QoS, 
routing or virtual topology?  Or ... ?

Scott

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 4 Mar 2008 18:33:47 +0100

El 04/03/2008, a las 18:12, Scott Brim escribió:

> The problem, as usual, is all the policy stuff, e.g. the various  
> kinds of traffic engineering.  I do not see that adding capability  
> to endpoints will reduce what you have to do at the border router  
> intelligently but the border router still has to advertise  
> reachability for them, manage what traffic takes which path, and so  
> on.

well, the nice thing about this approach is that the routers do have a  
knob for doing TE, which is basically increment the congestion  
perceived by the endpoint. If they do that, the endpoint will diverge  
its traffic to alternative paths and we actually achieve what the  
router seems to desire, i.e. moving the traffic towarda a different  
link, right?

regards, marcelo

> balancing according to experienced throughput.  The only thing  
> routing could do to disrupt that would be to dramatically change  
> routing frequently.  Or are you thinking of "vertical coupling"  
> approaches where something up at session layer can ask the network  
> for changes in QoS, routing or virtual topology?  Or ... ?
>
> Scott
>
> --

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 4 Mar 2008 18:27:23 +0100

Hi Jari,

> would be capable of taking the advantage of such exposed multihoming.

FWIW, imho whether the actual path selection is made on the transport  
layer using different addresses or doing it at the ip layer, is an  
implementation issue, but the main concepts apply in both cases. I  
mean, the idea is to actually split the load in multiple paths, so the  
usage of the network is more optimal and congestion is automatically  
avoided

> is this solvable in the near future?

there is a lot of research already been done on this area actually and  
the results are very interesting.
Most of this work was initiated by Kelly and it seems that the  
theoretical part is well advanced so we could actually try to use some  
of these results.

Some basic papers on this (selected by Damon cc)

     *  "Stability of end-to-end algorithms for joint routing and rate  
control", Kelly and Voice.

     * "Fluid models of integrated traffic and multipath routing",  
Key, Massoulie

     * "Optimal flow control and routing in multi-path networks",  
Wang, Palaniswami, Low

     * "Optimal Congestion Control with Multipath Routing Using TCP- 
FAST and a Variant of RIP", Mallada, Paganini

     * "Multi-path TCP: a joint congestion control and routing scheme  
to exploit path diversity in the internet", Han, Shakkottai, Hollot,  
Srikant, Towsley

     * "Congestion control with adaptive multipath routing based on  
optimization", Paganini

> providers/network owners are in control of what kind of multihoming  

the interesting part here is that TE can be performed in a different  
way, but seems that both parties (the ISP and the users)still have  
knobs to do this

regards, marcelo

---

From: "Tony Li" <[email-masked]>
Date: Tue, 4 Mar 2008 18:12:38 -0800

|But it also makes the routing system more expensive, because it has to
|maintain a lot of information.

Jari,

Could you please say more?  It would seem to me that with Mark's proposal, a
site would only advertise its PA prefixes, which would then be subject to
subsequent aggregation.  This would seem to be about as optimal as one could
get, given our current paradigms for information propagation.

Regards,
Tony

---

From: "William Herrin" <[email-masked]>
Date: Tue, 4 Mar 2008 23:24:49 -0500

Mark,

If you're feeling ambitious, change "two" to "many" and modify TCP so
that during the SYN+SYN/ACK, both sides can exchange cryptographically
signed connection IDs. The presence of these two IDs with a correct
signature identifies subsequent packets associated with the
connection, regardless of the source and destination addresses.

Any such "TCP" packets can also contain an "address update" option
which offers the other side a fresh list of addresses at which it can
be reached as well as hints about priority and load balancing.

Among other nifty things, this would allow you to completely dispense
with the source port. If you wanted to, you could also change the
destination port to a destination service name, since it need only
appear in the original SYN packet.

I suppose if you want to retain backwards compatibility, you would at
least include the source and destination port numbers in the SYN
packet and drop them later if the options returned in the SYN/ACK
agreed to use your new TCP.

If you're feeling *really* ambitious, exchange the hostname during the
connection setup as well so that if all the connection's IP addresses
change while the connection is idle, the next transmitter can  find
the other side by refreshing the address list with a DNS lookup.
Require the DNS server to allow it's clients to update their
hostname's list of IP addresses.

Alternately, you could dispense with the cryptography and address
update option and instead use a forward DNS lookup to refresh the
connection's permitted address list any time you receive a packet from
a source that isn't in the original list.

Next, just to be clever, throw in an "on hold" TCP option which allows
a mobile station that knows it will be unavailable for a while to
instruct the folks with which it is currently communicating not to
attempt communications on that connection for up to some number of
seconds because it will be unavailable.

If you do all of these and then succeed in deprecating old-style
source port/dest port connections, this has some interesting
implications for route aggregation. You'll find it becomes possible to
create a fully automatic system to resize and reassign IP addresses at
need which in turn allows the Internet as a whole to automatically
achieve something near the optimum achievable aggregation through the
kind of "emergent behavior" you talk about.

Say, that's pretty neat. I had considered most of these ideas before,
but it didn't occur to me to use a TCP protocol addition as the
starting point so that deployment could be incremental and fully
backwards compatible. If you don't write something up, maybe I will.

Regards,
Bill Herrin

-- 
William D. Herrin                  [email-masked]  [email-masked]
3005 Crane Dr.                        Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Pekka Nikander <[email-masked]>
Date: Wed, 5 Mar 2008 07:39:36 +0200

Hi William,

> connection, regardless of the source and destination addresses.

Interesting.  This sounds pretty much like HIP base exchange...

> be reached as well as hints about priority and load balancing.

... and this like HIP mobility & multi-homing.

> appear in the original SYN packet.

And this somewhat (but not quite) like id delegation in HIP.

> agreed to use your new TCP.

And this like the hack (so far unspecified) to carry HIP I1 as TCP  
options.

> hostname's list of IP addresses.

If you replace DNS with a rendezvous server, your hit again at HIP :-)

<snip>

> kind of "emergent behavior" you talk about.

That was realised in the HIP community some 6-7 years ago, and I  
guess in the Nimrod and some other communities even earlier.

> backwards compatible. If you don't write something up, maybe I will.

The biggest differences between your thinking and HIP seems to be  
that HIP is implemented below TCP, so that it works also for UDP.  I  
think it matches better with the IP semantics, as a single IP address  
is (today) typically associated with a number of co-located TCP and  
UDP end-points.

Then there is also work specific to TCP, e.g Christian Huitema's eTCP  
(or whatever it was called).

I'm glad to see that more people come to the roughly same  
conclusions, independently.  :-)

--Pekka Nikander

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 5 Mar 2008 16:42:09 +0100

Hi Pekka,

El 05/03/2008, a las 6:39, Pekka Nikander escribió:

...
> think it matches better with the IP semantics, as a single IP  
> address is (today) typically associated with a number of co-located  
> TCP and UDP end-points.
>
> Then there is also work specific to TCP, e.g Christian Huitema's  
> eTCP (or whatever it was called).
>

right, but i think that what is missing in all these proposals is to  
understand the interaction with congestion control, which may be  
critical if they get deployed.
Moreover, understanding how these approaches can help to avoid  
congested paths, could actually be a solution to one of the cuases of  
the routing scalability problem, since many to the TE tricks and  
espcially the updates with more specific that change frequently seem  
to be related with trying to move traffic from one path to a different  
one, probably avoiding congested parts of the network. If there is a  
mechanism that does it automatically for the hosts, then, maybe the  
number of more specific announced and especially the highly volatile  
ones, will substantially decrease. But, i guess there are many  
hypothesis in this sentence, to actually make a strong case at this  
point

regards, marcelo

---

From: "Mark Handley" <[email-masked]>
Date: Wed, 5 Mar 2008 15:58:47 +0000

>  critical if they get deployed.

Absolutely.  The stable load-balancing properties come out of using
multiple links simultaneously in such a way that congestion on one
path causes more bytes to be sent the other way.  But crucially it
must not move all the traffic the other way, or the system is unstable
and can oscillate backwards and forwards between the multiple paths.
This is the reason why I think you can solve the problem at L4 without
changing the apps (or at the application layer, but only if you change
the apps), but I don't think you get the same properties if you solve
the problem below layer 4.

 - Mark

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 06 Mar 2008 13:34:39 +1300

> the problem below layer 4.

You certainly won't; but layer 4 already has to deal with unannounced
changes in layer 3 today, doesn't it? One question is whether a
hypothetical new pattern of layer 3 behaviour would require another
round of layer 4 algorithm R&D.

I have difficulty seeing how we can generically solve the problems
before us at layer 4, given that layer 4 is *not* the waist of the
hourglass, despite the arguments Jonathan makes in
draft-rosenberg-internet-waist-hourglass-00.txt. Making layer 4
aware of layer 3 behaviour, on the other hand, seems very
desirable to me. (To give a caricatural example, it would be
absurd if every time SCTP changed addresses, a layer 3 man-and-encap
scheme mapped them back to the same locators.)

    Brian

---

From: Scott Brim <[email-masked]>
Date: Wed, 5 Mar 2008 08:23:59 -0800

> Interesting.  This sounds pretty much like HIP base exchange...

I had exactly the same thought, smiling as I read.

This is interesting stuff but mostly orthogonal to routing (alas).  It
allows endpoints to manage their congestion and do simple TE.  It
doesn't allow more sophisticated policy, avoid renumbering, etc. (see
the RRG design goals).

---

From: "Tony Li" <[email-masked]>
Date: Wed, 5 Mar 2008 17:04:50 -0800

|This is interesting stuff but mostly orthogonal to routing (alas).  It
|allows endpoints to manage their congestion and do simple TE.  It
|doesn't allow more sophisticated policy, avoid renumbering, etc. (see
|the RRG design goals).

Doesn't it obviate the need to advertise PI prefixes in the first place?

Tony

---

From: Pekka Nikander <[email-masked]>
Date: Thu, 6 Mar 2008 07:06:15 +0200

Mark,

>>  right, but i think that what is missing in all these proposals  
>> [like HIP, SHIM6, etc] is to understand the interaction with  
>> congestion control, which may be critical if they get deployed.
>
> must not move all the traffic the other way, or the system is  
> unstable and can oscillate backwards and forwards between the  
> multiple paths.

So far I fully agree, even don't I cannot claim really understanding  
the issue.  [Kelly & Voice is very next on my reading list; Bob B.  
commended it too, in addition to Marcelo.]

> This is the reason why I think you can solve the problem at L4  
> without changing the apps (or at the application layer, but only if  
> you change the apps), but I don't think you get the same properties  
> if you solve the problem below layer 4.

Here I do not agree, due to a) signalling efficiency, b) mobility, c)  
path sharing within the network, and I guess a couple of other things  
I just don't remember this early moment.

I still think that we should reconsider the sublayering or location  
of functionality within layers 3/4, and rethink where to put  
congestion control.  For congestion control, we should apparently  
separate flow-related functions (at transport) and path-related  
functions (at end-to-end), where path-related functions are shared by  
all flows going over the same path.  [Consider also John Day's view  
in his new book, where he basically claims that logically layers 3  
and 4 form just one single layer, and some of the layering we've done  
is just architecturally wrong due to us not understanding this.]

--Pekka

---

From: Pekka Nikander <[email-masked]>
Date: Thu, 6 Mar 2008 07:13:59 +0200

> Doesn't it obviate the need to advertise PI prefixes in the first  

I think so, in the end of the day.  It makes re-numbering a non- 
issue, as even active connections will trivially survive.  But the  
transition will be long..... (15 years?)

Hence, if we want to move there (which I think we do), we need some  
interim solutions and a roadmap.

Already a few years back in the Amsterdam SHIM6 interim meeting we  
decided to make SHIM6 and HIP header-compatible in the hope that that  
would make such a roadmap easier to explore.  And now I see all this  
work by Christian V. et al on Six/One to look at the same problem  
from a different angle, moving the host functionality into the  
network (as I tried to outline in the now expired "generix" draft:  
http://tools.ietf.org/html/draft-nikander-ram-generix-proxying-00).

--Pekka

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 6 Mar 2008 09:39:54 +0100

Hi Brian,

El 06/03/2008, a las 1:34, Brian E Carpenter escribió:

>> unstable
>> without
> changes in layer 3 today, doesn't it?

right, but i think the point is that it seems clear that layer 4 knows  
more about the path status than layer 3, simply because is closer to  
the app. So, TCP knows about whether the path is congested or not, and  
if it working in a much more accurate way than layer 3.

So, from my perspective, i think that each layer should perform the  
corresponding role. So i agree with you that the right place to select  
addresses (and to make the id locator split) is at layer 3, but i  
think that the best place to figure out when a different path is  
needed (either to divert the whole flow or to spill additional bytes  
that don't seem to fit in the current path) is at the tcp layer. The  
results seems to imply that if you do this, you can route around  
congestion and improve the overall utilization of the netowrk

> One question is whether a
> draft-rosenberg-internet-waist-hourglass-00.txt.

i agree with you here.
For UDP, the problem will need to be solved below, since it doesn't  
seem to provide much benefit doing it at the udp layer (the other  
option is doing it above UDP, but people are already doing that)
But for TCP, which is basically the congestion control mechanism that  
we have available, it seems that a much tighter realtion between id/ 
loc and tcp woudl indeed would be beneficial

Regards, marcelo

> Making layer 4

---

From: "Tony Li" <[email-masked]>
Date: Thu, 6 Mar 2008 00:46:33 -0800

|
|I think so, in the end of the day.  It makes re-numbering a non-
|issue, as even active connections will trivially survive.  But the
|transition will be long..... (15 years?)
|
|Hence, if we want to move there (which I think we do), we need some
|interim solutions and a roadmap.
|

As always with host changes, I'm not convinced that it takes 15 years to
make significant forward progress.  The (in)security of the net has done us
a favor in that we can now update a significant portion of the hosts within
one month of having a patch, and certainly with a new OS every 3-5 years.

Test: 15 years ago I was programming on a Sun clone (SunOS 4.x) using an NCD
Xterminal.  None of that hardware has been seen for years and Solaris is now
on version 10.  Personally, I've gone through 4 PCs, several versions of
Windows, numerous versions of FreeBSD and more laptops than I can think of.

All it takes to make rapid forward progress is concerted effort, and all
that takes is rough consensus...

Tony

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 6 Mar 2008 09:48:08 +0100

Hi Scott,

El 05/03/2008, a las 17:23, Scott Brim escribió:

> This is interesting stuff but mostly orthogonal to routing (alas).

I am not yet sure about that (i am not sure that is not the case  
neither though)

It would be possible to think that TCP marks the packets saying that  
it wants a different path, cause the currnet one is congested, and  
then we would need a much richer interaction with routing.

 From my perspective, this is not routing but it is not transport  
neither, it is the interaction between them and there is interesting  
and promising work to be done there imho

>  It
> allows endpoints to manage their congestion and do simple TE.

I disagree with the word simple :-)
I think this provides really interesting TE because is quite dynamic  
TE response to congestion. this is what people try to achieve today by  
injecting more specifics that are highly volatile. More specifics for  
TE represrent about 30% of current routing table (Huston, Zhang et  
al). Moreover, i would think thjat the represent the most volatile  
ones, since they are likely to change because of the conditions, so  
the are likely to account for an imporntat part of the updates. Maybe  
doing this type of mechanism would allow automatic rerouting to avoid  
congestion and will avoid the more specifics related to that (of  
course, the more specifics related to a different type of TE needs  
wouldn't go away)

>  It

agree that some other forms of policy routing are not so clear, but  
note that the ISPs can give congestions hints to make the flows to be  
rerouted, so it would be indeed possible to achive some forms of  
policy manipulation with this

renumbering would need more work, and doesn't seem to me to be a key  
feature of this approach to put in some way :-)

regards, marcelo

---

From: marcelo bagnulo braun <[email-masked]>
Date: Thu, 6 Mar 2008 09:51:19 +0100

> Doesn't it obviate the need to advertise PI prefixes in the first  

right but that implies that the site is using PA and then all the  
renumbering pains appear, right? am i missing something?

We could think that ones of the identifiers used are a ULA and then  
avoid internal connectivity problems through rebumbering events (since  
this type of approaches can very easily work with some addresses that  
are not working) abnd also avoid part of the renumbering pain since  
the only string exposed to the apps is the ULA, so, it could be less  
pain, but still it seems that the PA addresses are ewxposed to the  
hosts, at least in the form of locators.

Regards, marcelo

---

From: "William Herrin" <[email-masked]>
Date: Thu, 6 Mar 2008 09:03:56 -0500

>  that takes is rough consensus...

Hi Tony,

It depends on the character of the change. It didn't take 15 years for
TCP window scaling to be deployed; it was out there in only a couple
years. On the other hand, I'd be surprised if IPv6 was deployed to a
usefully ubiquitous level even 20 years after we started.

The former was small, strictly backwards compatible and required no
configuration changes. So it got picked up in the tech refresh cycle
and made it out quickly.

The latter failed to meet those any of those three criteria and is
still struggling for it.

It's not a question of consensus or effort. It's a question of hassle
factor. The higher the hassle factor, the slower it gets deployed.

Regards,
Bill Herrin

---

From: Scott Brim <[email-masked]>
Date: Thu, 6 Mar 2008 16:32:04 -0500

Excerpts from Tony Li at 17:04:50 -0800 on Wed  5 Mar 2008:
> |This is interesting stuff but mostly orthogonal to routing (alas).
> |It allows endpoints to manage their congestion and do simple TE.
> |It doesn't allow more sophisticated policy, avoid renumbering,
> |etc. (see the RRG design goals).
> 

It does reduce the number of routes in the core.  Another nice thing
is that it's incrementally deployable (different hosts can have
different levels of inteligence).  However, host behavior is based on
knowledge of their IP addresses, and hosts have to be reconfigured
when PA provider connections change, so you still have renumbering
issues.  Also routes are no better than they are today.

---

From: Jari Arkko <[email-masked]>
Date: Thu, 06 Mar 2008 13:39:58 -0500

Scott,

> reachability for them, manage what traffic takes which path, and so on.

Right. But what the border router has to do (e.g., keep two sets of
prefixes) is different from what has to happen in the core of the
Internet. In Mark's approach you would not have to push the multihoming
problem and the individual organization's prefixes to the core; it would
work with just provider aggregated addresses. So that is the cost saving
in his approach (and the costs are elsewhere...)

> frequently.  Or are you thinking of "vertical coupling" approaches
> where something up at session layer can ask the network for changes in
> QoS, routing or virtual topology?  Or ... ?

I did not claim that the routing system would disrupt Mark's approach.
My point was that a host based approach might theoretically have more
information to base its decisions on than a router based design. And
hence be better.

But admittedly I was handwaving this. My rationale was that a host that
has partial topology information (set of prefixes), congestion
information, and application demand information could potentially do a
better job than the routing system which traditionally has only the
topology information and might have some amount of information about
congestion. And hosts may have information beyond a single AS, such as
when my laptop is connected to two different providers.

But of course, the devil is in the details, so its not clear that host
approach would win even in a theoretical comparison. First of all, the
routing system may have mechanisms to at least locally react much better
to congestion/load sharing than the hosts. And obviously, the routing
system knows more about the topology than hosts.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Thu, 06 Mar 2008 13:42:19 -0500

Tony,

> get, given our current paradigms for information propagation.
Right. That is what Mark's scheme would do. My comment about the routing
system becoming more expensive was about the other type of solution
where you let the routers handle multihoming.

Sorry for being unclear.

Jari

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 8 Mar 2008 11:08:02 +0100

> usefully ubiquitous level even 20 years after we started.

You are mixing two questions: the availability and use of code. IPv6  
has been available in Windows XP, Mac OS X 10.2 and other OSes for  
more than half a decade now. (Granted, those initial implementations  
in those two OSes weren't ready for prime time, but still.) However,  
just having code in hosts doesn't mean the protocol gets executed.  
Many organizations are reluctant to deploy new "stuff" without a very  
good reason to do so: if it doesn't buy them anything, why risk  
problems? And that's for simple stuff such as ECN, when actual work is  
involved, such as with IPv6, this gets orders of magnitude harder.

A counter argument against the OS update argument: yes, the main OSes  
are often upgraded. But these days, very many devices speak IP and not  
all of them are often updated or even realistically updatable.  
Reflashing a five year old base station or modem, even if the software  
is available, isn't likely to happen in most consumer networks.

But I do agree that it wouldn't take 15 years, especially if there is  
a modicum of backward compatibility. Even if my print server or 3 year  
old cell phone can't do the new cool stuff, if my brand new wifi iPod  
and my regular computers can and the old stuff can still do what it  
did before, I'm happy. And that iPod and the computers can be updated  
quite easily within a few weeks. (I'll run updates unless I'm  
traveling or I'm close to a deadline.) For enterprise installations  
this will be longer but even they will adopt new software after a year  
or two for 98% of their stuff.