Subject: [RRG] Why delaying initial packets matters

From: Robin Whittle <[email-masked]>
Date: Fri, 08 Feb 2008 14:00:10 +1100

In "Re: [RRG] Re: Aggregation Implies Provider Dependence (LISP-ALT)
& Ivip dependencies too", Brian Carpenter wrote:

> However, user expectations are of a noticeable delay between
> dialling a phone number and hearing the ringing tone. So e2e
> lookup solutions to both number portability and roaming are
> tolerable. If we're prepared to tolerate such a delay in initial
> identifier-to-locator lookup, why are we having this
> conversation? (And note that Mobile IP effectively *does* use
> such a solution.)

A scheme such as ALT will also delay some of the DNS (identifier to
locator) lookups, since some DNS servers will be on EID space which
the ITR has no mapping for.

I think there at at least two reasons we should be concerned about
the delay problems inherent in a map-encap scheme which depends on
any kind of global query server network (LISP-CONS, LISP-ALT or Bill
Herrin's TRRP):

1 - Great reluctance to introduce or impose any new architecture
    which further delays the establishment of communications.

2 - Concern that if the new kind of address space involves
    extra delays of any measurable kind - especially when they
    are perceptible by end-users - that this will be a serious
    and perhaps fatal barrier to the widespread adoption of the
    new kind of address space.

So even if we decided these delays should be acceptable for all
future end-users we also need to consider how end-users will
actually feel about these delays.

Consider how much money end-users spend on web servers, with fast
CPUs, high peak data rates to the Net etc.  This is so the browsing
user finds the website nice and fast.  Part of it is also that the
end-user feels good about their web server.

This may seem a trivial, non-technical, matter - just as purchasing
flashy new prestige automobiles is not really a technical matter,
since they all have to travel at the same speed on the roads.

The way end-users feel about their network is important.  Since we
need the new architecture to be widely - ideally ubiquitously -
adopted, we need to ensure that the new kind of address space feels
good to future end-users.  These will be a much wider range of
organisations and individuals than currently run multihomed networks.

Consider a business end-user choosing between two types of address
space - such as for their web server, or for their desktop computer
users in the office.

One type is ordinary PI space as we have it today, directly handled
by a BGP advertised prefix.  The other is from the map-encap scheme
and is known to involve somewhat slower responsiveness due to the
new architecture being based on ALT etc.

It will be difficult to convince end-users to pay money (including
less money) for the slower type of address space.

This also applies indirectly if the end-user is not seeking their
own address space as such, but is purchasing services on a host
which uses one type of address space or the other.

I assume we want hosting companies to use the new map-encap type of
address space.

If there is any significant (however defined) measurable loss of
responsiveness due to initial packet delays in the new type of
address space, the hosting companies would avoid it like the plague.
 They would promote their servers as running on genuine PI address
space, not that second-rate, cheap, often a little slower to
start-up, type of LISP-ALT address space.

LISP-NERD, APT and Ivip avoid these delays.  Initial packets are
either tunnelled directly by a full database ITR (NERD and with
ITRDs in Ivip, or the Default Mapper in APT) or they are tunnelled
very rapidly by a caching ITR (Ivip), since the caching ITR can get
a response from a local Query Server within a few milliseconds or
tens of milliseconds.  (Ivip also allows the initial packets to be
passed on to where they will be handled by a full database ITRD and
tunnelled without delay - the caching ITRs handle all the traffic as
soon as they have the mapping data.)

For us to choose a new map-encap architecture which relies on a
global query server network, with its inherent initial packet
delays, administrative and security difficulties, and its greater
vulnerability to packet loss I think we we would need to be
convinced that the benefits of such a scheme are absolutely
essential, and that therefore we have to put up with
the new type of address space being slower for end-users.

As far as I can see, the sole benefit of ALT, CONS or TRRP is that
at some stage in the future when the mapping database might grow to
immense proportions, these architectures will still be feasible -
while any alternative architecture (which involves multiple copies
of the database being pushed and stored to multiple devices around
the Net) would become technically or economically prohibitive.

The alternatives (NERD, APT and Ivip) all involve sending the
database to multiple full database query servers and/or full
database ITRs all around the Net.

Are there any major arguments for ALT (or CONS or any other global
query server system) other than that it can scale to indefinitely
large database sizes, while any architecture involving local query
servers (supposedly) could not handle such a large database?

Later, I will post an argument about why it is, and always will be,
practical and desirable to push even the largest imaginable mapping
database to a system of local query servers and/or ITRs.

  - Robin         http://www.firstpr.com.au/ip/ivip/

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Kevin Loch <[email-masked]>
Date: Thu, 07 Feb 2008 22:56:36 -0500

> I assume we want hosting companies to use the new map-encap type of

There are several reasons hosting companies should not and would
not use such type of addresses.

1. There are very few hosting companies compared to the number
of end users.  Currently something like 4 orders of magnitude.
2. The vast majority of DNS zones are served by DNS servers
at hosting companies.    This is especially important if new
address architecture uses existing DNS infrastructure.
3. It is unlikely that any encap or remapping system will
be as reliable as traditional routing.

> start-up, type of LISP-ALT address space.

Any perceived deficiency will make it a non-starter for hosting
companies.

5. Most hosting companies today have direct allocations from RIR's.

- Kevin

---

From: David Conrad <[email-masked]>
Date: Thu, 7 Feb 2008 21:31:25 -0800

Robin,

> the ITR has no mapping for.

Not necessarily.  The mapping service (e.g., DNS) servers can exist in  
locator space.  In fact, this makes many things much easier (e.g.,  
avoiding circular dependency).  The downside is that the mapping  
service itself obviously cannot take advantage of the locator/ 
identifier split.

>    new kind of address space.

I believe Brian's point was that extra delays of measurable kind have  
_not_, even when perceptible by end users, been fatal in the past.   
People adjust.  Software developers adjust.  Etc. Why, I remember a  
day when if I did something on my computer that required Internet  
connectivity, I'd go get a soda instead of listening to the screech  
and moan of my modem as it brought up my SLIP line... (and yes, I did  
walk through hip deep snow to school both ways).

And to be clear, the delays you are concerned with are at  
communication initiation with a "new" endpoint only. A time when a few  
hundred more milliseconds of latency would likely be lost in the noise  
of DNS lookups, connection handshake, crypto handshake, etc.

> less money) for the slower type of address space.

You are aware that there are still companies around that sell dialup,  
partly because they're significantly cheaper than DSL and cable, right?

> LISP-NERD, APT and Ivip avoid these delays.

At the cost of scalability and slower update cycles.  TANSTAAFL.

> large database sizes,

A pull system can potentially provide for much higher rates of change  
than a push system.  As I believe has been mentioned in the past, a  
pull system (if augmented by forwarding) could even be used for  
mobility.

> while any architecture involving local query

In my mind, the question isn't so much about the size of the database  
(DRAM is cheap, disk is cheaper), but rather propagating updates.  It  
seems to me that pull systems are merely replicating some of the  
problems we face in the existing routing system since all updates must  
be propagated globally, regardless of whether they are of any interest  
to the source network, on the (given potential scaling targets  
essentially, statistically speaking, insignificant) chance that the  
source edge network _might_ want the update.

> database to a system of local query servers and/or ITRs.

I'll be quite interested in reading it.

Regards,
-drc

---

From: Robin Whittle <[email-masked]>
Date: Fri, 08 Feb 2008 17:43:28 +1100

Hi David,

You wrote, in part:

> The mapping service (e.g., DNS) servers can exist in locator
> space.  

Yes, but I think we want end-users who run their own DNS to be happy
putting their servers on their own map-encap space machines.

If there are too many gotchas, such as "It might be best not to run
DNS on map-encap addresses", then this puts up another barrier to
the widespread adoption we are relying on.

> In fact, this makes many things much easier (e.g.,
> avoiding circular dependency).  

Any aspect of the map-encap system which depends on some part of the
DNS will come with a requirement that all such DNS servers are on
ordinary addresses (RLOC in LISP).  This is unlikely to involve
end-users.

In Ivip, each end-user's mapping is changed via a system which is
secured with a username and password.  It doesn't rely on any
physical infrastructure of the end-user, and could be done manually,
or by some multihoming monitoring system run by another company.

Ivip would involve being able to download snapshots of various parts
of the mapping database, probably by HTTP.  The organisations which
provide these (Root Update Authorisation System) would be using PI
space for all their servers, including DNS.

> I believe Brian's point was that extra delays of measurable kind
> have _not_, even when perceptible by end users, been fatal in the
> past. 

I am not saying they would be "fatal".

People will have a choice whether to adopt the new address space or
not.  Any cause of extra delay, unreliability etc. will make them
very wary of adopting it.   This may be true even if in our wiser
judgement we consider the problem to be negligible.  As long as
there is a perceived deficiency, it will be much harder to have the
new space widely adopted.

> People adjust.  Software developers adjust.

They do when there is no choice.  Maybe for some or many end-users
there will be no choice - they can't get PI space for one reason or
another.  In that case, it doesn't matter a lot if the map-encap
space has problems.

Rapid adoption would be fostered by the new space having no such
problems.

> communication initiation with a "new" endpoint only. A time when
> a few hundred more milliseconds of latency would likely be lost
> in the noise of DNS lookups, connection handshake, crypto
> handshake, etc.

I am not convinced it would only be a total of a few hundred msec.
with ALT.

DNS lookup of B's IP address, A -> B and B -> A all could involve
delays while ITRs wait for a request (or initial packet) to traverse
this global query server system.  ALT also is likely to be less
reliable than a full database ITR or a caching ITR using a local (0
to a few hundred km) query server, due to the much longer path
length and therefore chance of packet loss.

The fact that extra delay beyond a few tens of milliseconds exists
is a reason for people to avoid it.  How strong a reason, I can't
say for sure - but waiting for things to respond on the Net is so
tiresome.  Anything which makes it worse will be thought of very dimly.

>> It will be difficult to convince end-users to pay money
>> (including less money) for the slower type of address space.
> 
> You are aware that there are still companies around that sell
> dialup, partly because they're significantly cheaper than DSL and
> cable, right?

Sure, but we don't want just the lower echelons of end-users to
adopt map-encap space.  We need it to be adopted widely, ultimately
ubiquitously.  Ideally this will happen because it is good and cheap
and flexible, not because it is second rate and cheap and a subset
of people adopt because they are cajoled into it and/or because they
aren't as fussy as other people.

> At the cost of scalability and slower update cycles.  TANSTAAFL.

APT and Ivip only need to get the mapping database updates to a
subset of the ITRs and to the query servers which the remainder of
the ITRs use.  NERD needs to get the updates to all ITRs.

I was hoping to write up more details of the fast database
distribution system which Ivip relies on, in the next few weeks, but
 work commitments will probably delay this.

>> Are there any major arguments for ALT (or CONS or any other
>> global query server system) other than that it can scale to
>> indefinitely large database sizes,
> 
> A pull system can potentially provide for much higher rates of
> change than a push system.  

Not unless there was some way of pushing updates to ITRs which might
need to know about it, or unless there was such a short caching time
that the query and response traffic would be excessive.

> As I believe has been mentioned in
> the past, a pull system (if augmented by forwarding) could even
> be used for mobility.

Do you mean by "forwarding" a way that the query server (in ALT, one
or more ETRs) remembers which ITRs asked it for mapping information
in some recent time period, and then sends updated mapping
information to those ITRs whenever the mapping is changed?

Ivip's Query Servers will do this, but this is a less onerous task
than with ALT, CONS or TRRP where the single (or one of a few) query
servers for the entire world has to do this, over much longer
distances to far more numerous ITRs.

>> while any architecture involving local query servers
>> (supposedly) could not handle such a large database?
> 
> In my mind, the question isn't so much about the size of the
> database (DRAM is cheap, disk is cheaper), but rather propagating
> updates.  It seems to me that pull systems are merely replicating

I think you mean:               push systems (e.g. NERD, APT & Ivip)

> some of the problems we face in the existing routing system since
> all updates must be propagated globally, regardless of whether
> they are of any interest to the source network, on the (given
> potential scaling targets essentially, statistically speaking,
> insignificant) chance that the source edge network _might_ want
> the update.

Yes.  With NERD, every ITR in the world gets to know about every
change in the mapping of every EID.

In APT, only the Default Mappers get all the updates.

In Ivip, only the full database ITRs and query servers get all the
updates.  Ivip is more flexible than APT regarding the location of
full database ITRs and Query Servers, with the rest of the work
being done by caching ITRs.

This greatly reduces the problem you refer to, but doesn't make it
go away entirely.

I don't think it is OK for any end-user to change their mapping
frequently (and they could change it once every few seconds with
Ivip) and have each change propagated around the world.  For one
thing, they could use it as a free broadcast channel . . .

I think mapping updates in any push system need to be either at a
low rate covered by some flat fee, or be charged for in some way, so
that those who make really frequent changes contribute in some way
to the cost of pushing the changes all round the world, and so will
think more carefully about making these changes.

>> Later, I will post an argument about why it is, and always will
>> be, practical and desirable to push even the largest imaginable
>> mapping database to a system of local query servers and/or
>> ITRs.
> 

I am waiting for some more LISP-ALT folks to resurface on the list.

  - Robin

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 08 Feb 2008 09:00:51 +0100

Robin,
>     which further delays the establishment of communications.

I completely disagree for two reasons. First, applications are used to 
have a small delay before the first packet flows, this delay has been 
introduced by the DNS a long time ago. I guess that some people were 
worried by the added delay at that time, but the benefits of the DNS are 
clearly more important than this additional second. Second, the delay is 
only encountered for the *first packet* that needs a mapping. With a 
router based solution such as LISP, many flows will benefit from the 
mapping learned from previous flows on the routers.

>     new kind of address space.

A new architecture will bring new benefits and some costs in terms of 
delays for a minor fraction of the packets (0.01% or fewer ?) and somme 
added overhead due to encapsulation. This added costs must be compared 
to the benefits of the new architecture.

Olivier

-- 
http://inl.info.ucl.ac.be , Universite catholique de Louvain, Belgium

---

From: Stig Venaas <[email-masked]>
Date: Fri, 08 Feb 2008 10:22:35 +0100

> to the benefits of the new architecture.

I'm not so worried about the delay itself, but rather how it might
affect transport protocols. The DNS delay is not much of an issue since
the application does not try to exchange any actual data until DNS is
completed.

One solution, which is a bit of a kludge, could be to have some kind of
interaction between DNS and the mapping system, where the necessary
mappings can be installed (based on DNS replies), before the DNS reply
is given to the application. I don't really like this though.

Another possibility could be to add some kind of signalling where hosts
more or less signal that they want to communicate with this other host
before they actual do the application parts like say TCP. E.g., you
could imagine that when a TCP application makes a connect() call to some
destination, the stack will signal to the network that it wishes to
communicate with this destination, and not send TCP SYN until it has got
an ACK back that it's okay.

I'm sure there could be other ways. Anyway, I believe such delays will
require changes in applications and/or host stacks.

Stig

---

From: <[email-masked]>
Date: Fri, 8 Feb 2008 17:06:19 +0200

Hllo Olivier,

"A new architecture will bring new benefits and some costs in terms of
delays for a minor fraction of the packets (0.01% or fewer ?) and somme
added overhead due to encapsulation. This added costs must be compared
to the benefits of the new architecture." 

But the key issue is to who gains the benefits and who pays for it. The
content providers (hosting companies) will not pay for a solution that
only offers drawbacks to them, but benefits would go to ISP. There is no
incentive for content houses to deploy such a solution. 

I think in the last IETF there was a good presentation about what makes
a successful protocol. Worth of considering here as well.

- Hannu

---

From: Tony Li <[email-masked]>
Date: Fri, 8 Feb 2008 09:48:44 -0800

> incentive for content houses to deploy such a solution.

The incentive for everyone is to deploy a scalable routing  
architecture that serves us for the long term.  Regardless of the  
proposal that RRG puts forward, it's thermodynamically impossible  
that it will have no cost.  There will be deployment costs,  
forwarding plane changes, and a mapping subsystem to support.

Having a functioning Internet for our great grandchildren is the  
benefit and without that, the hosting companies are simply out of  
business.

Tony

---

From: Christian Vogt <[email-masked]>
Date: Fri, 08 Feb 2008 20:52:04 +0200

Stig -

> is given to the application.  [...]

Right, I had been thinking about this also.  Few thoughts:

- Piggybacking mapping records to DNS replies would enable pre-loading
   of an edge network's local mapping caches -- reactively, but still
   before the transport protocol launches its session and starts waiting
   for timeouts.

- DNS-based mapping resolution does not imply per-address mappings; new
   DNS records may well describe mappings on an address-block basis.

- DNS-based mapping resolution could be used in conjunction with another
   mapping system, as an optimization.  One could leave it up to an edge
   network to decide whether or not to support the optimization by
   attaching mapping records to DNS replies generated by its DNS servers.

- It may be unwise to rely *exclusively* on DNS-based mapping resolution
   because the technique does not work that well when a host starts a new
   session without an initial DNS look-up.  In such a case, DNS-based
   mapping resolution would likely have to involve reverse DNS because
   the look-up key would not be an FQDN, but an identifier IP address.
   And this will be really costly in terms of delay.

Comments are welcome.

- Christian

---

From: <[email-masked]>
Date: Fri, 8 Feb 2008 19:49:53 -0000

________________________________

From: [email-masked] on behalf of Tony Li
Sent: Fri 08/02/2008 17:48
To: <[email-masked]>
Cc: [email-masked]
Subject: Re: [RRG] Why delaying initial packets matters

Tony

Putting an ISP hat on here, having a network for the great grandchildren is out of scope of any 5 year business plan.

There are other possible solutions to the problems of ISP longevity. For example, they may simply pass on costs and make the network so expensive that its growth becomes limited that way. Or they may decide not to support end to end transparent services, deploy NAT and get a few simple PA address for the public side.  Clearly they prefered NAT to the IPv6 grand designed solution to a threat that could destroy the Internet. Probably they will come up with something even more hideous, but that for them makes sound business sense. 

Louise

---

From: Christian Vogt <[email-masked]>
Date: Fri, 08 Feb 2008 21:52:10 +0200

David -

> pull system (if augmented by forwarding) could even be used for

Yes.  The advantage of a reactive (pull) mapping system over a proactive
(push) one is that the former can provide tailored responses, such as
responses that indicate which in a list of locators is currently
preferred.  It thus provides a tool for traffic engineering of incoming
packets.  This is an advantage, albeit a small one given that a locator
can be changed in-band within a round-trip after mapping resolution.

Regarding mobility support of a reactive mapping system:  It would work
for local mobility within an edge network.  But not for global mobility.

- Christian

---

From: David Meyer <[email-masked]>
Date: Fri, 8 Feb 2008 13:37:08 -0800

On Fri, Feb 08, 2008 at 09:48:44AM -0800, Tony Li wrote:
> The incentive for everyone is to deploy a scalable routing architecture 
> that serves us for the long term.  Regardless of the proposal that RRG 
> puts forward, it's thermodynamically impossible that it will have no 
> cost.  There will be deployment costs, forwarding plane changes, and a 
> mapping subsystem to support.
>
> Having a functioning Internet for our great grandchildren is the benefit 
> and without that, the hosting companies are simply out of business.

	You know all know, of course I agree with Tony; but
	actually I have a stronger version of the statement,
	which includes the observation that history will not
	treat us kindly if we loose the great (and really,
	democraticizing) thing that is the Internet. And what is
	worse, if, despite the considerable intellectual
	capability represented by our community, we couldn't come
	together over something as important as the Internet,
	well, that would be an epic disaster in the history of
	humanity. What is at stake is no less than that. 

	But then you know all of that.

	Dave

---

From: David Conrad <[email-masked]>
Date: Fri, 8 Feb 2008 17:13:29 -0800

Robin,

> putting their servers on their own map-encap space machines.

In my view, the name-to-address mapping service (i.e., the DNS) is  
(should be) different than the EID-to-LOC mapping service (which  
could, but does not have to, look something like, e.g., the DNS).

>> say for sure

My feeling is that you are significantly overrating how much the vast  
majority of Internet users would even notice, much less care.  But,  
similarly to you, I can't say for sure.

> ubiquitously.

Pragmatically speaking, not really.  What we _need_ is to keep the  
growth in routing information below the growth in the routing system's  
ability to handle that information for the foreseeable future.

What we _want_ is a nice clean architecture that allows us to reach  
what we need and do so ubiquitously.

If the anticipated growth in the Internet is at the edges, it may be  
something focused on the edge is sufficient.  I would agree this is  
not ideal, but I feel it is important to keep this in mind.

> information to those ITRs whenever the mapping is changed?

Imagine a system that looks like (but isn't) the DNS with an  
authoritative source for mappings, ITRs as caching servers and mapping  
entries having TTLs.  As a node transitions from one ITR to the next,  
the node notifies the old ITR to forward to the new ITR for a couple  
of TTLs (one TTL should be sufficient, but just in case) at the same  
time it updates its mapping at the authoritative source.  If the node  
moves on, same thing happens, with packets going to the original ITR  
being forwarded multiple times (or the node could keep track of the  
ITRs it traverses through within a couple of TTLs and update them all  
to forward to the ITR it is using).

> I think you mean:               push systems (e.g. NERD, APT & Ivip)

Oops, yes.

> think more carefully about making these changes.

Again, I'm very uncomfortable assuming a particular business model to  
overcome the effects of an architecture.  But maybe that's just me.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Fri, 8 Feb 2008 17:43:12 -0800

Stig,

> require changes in applications and/or host stacks.

I'm not necessarily disagreeing, but before looking at potential  
solutions, can you be a bit more specific in what you believe will  
require changes?  Let's say you have a system that imposes a (say) 500  
ms delay on the first packet of a flow (but no significant change in  
subsequent packets).  Would you see a requirement for changes in UDP- 
based applications? Or would the changes just need to be in TCP-based  
applications?

Thanks,
-drc

---

From: David Conrad <[email-masked]>
Date: Fri, 8 Feb 2008 17:53:58 -0800

Hi,

> incentive for content houses to deploy such a solution.

Indeed.  One benefit I can imagine to justify a change is to support  
multi-homing without having to know or do BGP-fu or figure out how to  
get (and pay for) your ISPs to accept/announce your prefix(es).  From  
the ISP perspective, you'd look like a normal single-homed customer.  
Obviously this would apply for either push or pull systems.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Fri, 8 Feb 2008 17:58:14 -0800

Christian,

> for local mobility within an edge network.  But not for global  

Could you expand on this?  Assuming a pull system something along the  
lines of what I described to Robin, why wouldn't it work for global  
mobility?

Thanks,
-drc

---

From: Marshall Eubanks <[email-masked]>
Date: Sat, 9 Feb 2008 09:26:36 -0500

On Feb 8, 2008, at 8:43 PM, David Conrad wrote:

> require changes?  Let's say you have a system that imposes a (say)  
> 500 ms delay on the first packet of a flow (but no significant  
> change in subsequent packets).  Would you see a requirement for  
> changes in UDP-based applications? Or would the changes just need  
> to be in TCP-based applications?

It wouldn't necessarily change the kernel stack, but a 500 msec delay  
on every stream start would sure make video channel changing slow,.  
There is a lot of work going into making channel changing fast, and  
that work would become moot in this case.

Of course, a lot would depend on what is meant by "flow" above. If  
only the first packet from a /24 to a end user is
delayed, that would change little at the application layer.

Regards
Marshall

---

From: David Conrad <[email-masked]>
Date: Sat, 9 Feb 2008 09:50:22 -0800

Marshall,

Thanks for the response.

On Feb 9, 2008, at 6:26 AM, Marshall Eubanks wrote:
> It wouldn't necessarily change the kernel stack, but a 500 msec  
> delay on every stream start would sure make video channel changing  
> slow,.

I'd be ecstatic if my Tivo could change channels faster than 3000  
msecs (or could do it without getting the wrong channel 15% of the  
time, sigh), but that's neither here nor there.

> that work would become moot in this case.

Right.

> delayed, that would change little at the application layer.

All channels would come from of the same address block?  What are the  
current delay assumptions?

Regards,
-drc

---

From: Marshall Eubanks <[email-masked]>
Date: Sun, 10 Feb 2008 15:38:17 -0500

Dear David;

On Feb 9, 2008, at 12:50 PM, David Conrad wrote:

>> There is a lot of work going into making channel changing fast,  
>> and that work would become moot in this case.
>
> Right.
>
> All channels would come from of the same address block?

I was trying to guess what was really meant by "flow" in this  
context. It seems to me that the
worse case is per "tcp flow" and the best case is "every packet from  
some address block to another address block
in some (longish) time." (By "TCP flow" you could substitute "RTP  
SSID and port" for UDP.)

In most cases, DNS is involved in the start of an interaction with a  
remote site, and that can also take
a substantial time... the first time. If what's at issue is something  
with similar characteristics to DNS,
I think that you have an existence proof that it could be accepted.

> What are the current delay assumptions?

Well, it depends. A lot of (all?) players have a multi-second time  
out, so any multi-second delays will cause
real problems. Some players use non reception of packets as a sign of  
blockage, so they may switch bandwidth or
from UDP to TCP if the first packets take too long to arrive.

More seriously, channel changing goals are typically a few 100 msec  
at most, including RTT. Any longer is
supposed to annoy users. Whether or not that is supportable is  
another question - but I think a large
section of the streaming / broadcast community would object strongly  
if that applies to every stream.

Regards
Marshall

---

From: Stig Venaas <[email-masked]>
Date: Mon, 11 Feb 2008 08:36:35 +0100

David Conrad wrote:
> subsequent packets).  Would you see a requirement for changes in
> UDP-based applications? Or would the changes just need to be in
> TCP-based applications?

Right, I agree we need to look into the possible issues first.

I can see at least 3 possible theoretical effects of delaying the first
packet.

1. First packet is delayed, packets are kept in order
2. First packet is delayed, subsequent packets may arrive first
3. All initial packets dropped

Which of these apply depends on how the application protocol or
transport protocol behaves, as well as the underlying system.

Looking at 1.

Assuming you have to wait 500ms for some mapping to be in place. If
you have a simple UDP based protocol with a retransmission time
well above 500s, then it may not be an issue at all, apart from
perhaps a user experiencing longer delay. If the retransmission
time is less than 500ms + RTT, then you will get retransmissions
as well. If the initial packets are buffered, then you may get
additional load on some server, having to process both the
initial and retransmitted packets.

Regarding case 2. I think it should be possible to avoid
reordering.  Once a mapping is in place, it might be that some
buffered packets are delivered after some others that are
forwarded. I'm not so sure reordering is much of a problem though.

The 3rd case is what I had in mind when I posted. I believe
many UDP based applications have timeouts and retransmit
only after a few seconds. In this case, dropping the initial
packet will result in users experiencing far too long delays
in my opinion. It's not necessarily good if applications
retransmit in less than a second either (too short timeouts
may in other cases result in servers receiving the same
request multiple times).

I also think there might be some other effects of this. E.g.
if you have two servers A and B where an application should
prefer A over B, it might be that when initial packets to A
are delayed/dropped, it will try B instead. Possibly marking
A as bad/dead and continue using B that is not preferred.

Well, this is still more or less speculation. I suppose what
should be done, is to study some common applications or
protocols and see what the effects would be. An obvious first
candidate would be DNS I think.

Stig

---

From: Lars Eggert <[email-masked]>
Date: Mon, 11 Feb 2008 10:53:38 +0200

> 3. All initial packets dropped
...
> candidate would be DNS I think.

Exactly right. I've said this a few times in the past; data on how  
these different behaviors affect applications and transport protocols  
are essential before we can determine what's acceptable.

Although UDP-based apps (e.g., RTP) are important, what will happen  
depends a lot on the individual application. Initially, it would be  
easier to investigate how TCP/SCTP/DCCP react when 1, 2 or 3 happens,  
because their behavior is more tightly specified.

For example, for TCP, I'd expect that in case 1, the initially sampled  
RTT would be too high, causing a slower slow-start, in case 2, there'd  
be a few 500msec timeouts during the first few roundtrips due to the  
reorderings appearing as losses, and for 3, a 3-second timeout will  
kick in before the SYN will be retransmitted. But that's just a guess  
- I haven't measured/simulated anything, either.

Lars

---

From: Eliot Lear <[email-masked]>
Date: Mon, 11 Feb 2008 10:55:07 +0100

Hi Stig,

What follows are a bunch of questions based on DROPS and not DELAYS.
> candidate would be DNS I think.

And Dave is just the guy for DNS ;-)  My own limited understanding of 
DNS is that bind has a 5 second initial default timeout (from 
/usr/include/resolv.h) and that Windows XP has an initial timeout of 1 
second, and then another 2 seconds.  But this isn't the beginning and 
end of it.  There are two potential complicating factors.  First, not 
all applications use the OS resolvers.  At one time I believe this was 
the case at least with Mozilla.

Keeping in mind that these queries are likely recursive, we shift our 
scrutiny to the name server.  Supposing a query is dropped, will the 
next query go to the same name server or to another listed name server?  
If it's to another name server could that query also be dropped?  Will 
behavior differ based on the number of zone cuts and whether the 
appropriate name servers are in the same proximity as the parent servers?

Can the application mitigate this issue by pre-caching queries?  Some 
already do something like this but for different reasons.  Dave's Tivo 
could do this when it receives EPG information, for instance.

Regards,

Eliot

---

From: Luigi Iannone <[email-masked]>
Date: Mon, 11 Feb 2008 18:41:30 +0100

Hi,

Le 11-févr.-08 à 09:53, Lars Eggert a écrit :

> these different behaviors affect applications and transport  
> protocols are essential before we can determine what's acceptable.
>
> easier to investigate how TCP/SCTP/DCCP react when 1, 2 or 3  
> happens, because their behavior is more tightly specified.
>

besides,
  we need to look at the size of the buffers needed to store packets  
and the overhead introduced on the control plane of routers.
This can be an issue if there are too much packets to manage.

Luigi

> For example, for TCP, I'd expect that in case 1, the initially  
> sampled RTT would be too high, causing a slower slow-start, in case  
> 2, there'd be a few 500msec timeouts during the first few  
> roundtrips due to the reorderings appearing as losses, and for 3, a  
> 3-second timeout will kick in before the SYN will be retransmitted.  
> But that's just a guess - I haven't measured/simulated anything,  
> either.
>
> Lars
>
> --

Luigi Iannone

[email-masked]

---

From: David Conrad <[email-masked]>
Date: Mon, 11 Feb 2008 09:51:59 -0800

Stig,

> transport protocol behaves, as well as the underlying system.

Since all of these are possible today, won't all application or  
transport protocols need to deal with the possibility of these  
happening already?

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Mon, 11 Feb 2008 10:26:48 -0800

Eliot,

On Feb 11, 2008, at 1:55 AM, Eliot Lear wrote:
>> An obvious first candidate would be DNS I think.

The DNS provides a useful model for a pull-based system, but I'm not  
sure it would be particularly useful to assume if something can be  
shown it can't work if you put the mapping into the DNS, that implies  
it can't work period.

> Keeping in mind that these queries are likely recursive, we shift  
> our scrutiny to the name server.  Supposing a query is dropped, will  
> the next query go to the same name server or to another listed name  
> server?  If it's to another name server could that query also be  
> dropped?  Will behavior differ based on the number of zone cuts and  
> whether the appropriate name servers are in the same proximity as  
> the parent servers?

It's amazing that the DNS works at all, isn't it?  And yet, with all  
that recursion and potential for timeout, it seems to work reasonably  
well.

However, as I've said on several occasions, while I might personally  
use the DNS as a model for a pull-based system (primarily because most  
folks think they understand what that implies), I don't think the DNS  
(as it exists today) is the right answer.  For one thing, the timeouts  
would likely need to be rejiggered since the existing timeouts are  
based on assumptions that were derived from when 56 Kbps was  
considered "high speed".

As far as I can tell, a pure pull-based system is likely as unworkable  
as a pure push-based system. As Robin has suggested, a hybrid system  
is probably going to be necessary, so there are yet more tradeoffs to  
consider.

Regards,
-drc

---

From: Lars Eggert <[email-masked]>
Date: Tue, 12 Feb 2008 10:54:37 +0200

> transport protocols need to deal with the possibility of these  

Sure, and the concern isn't that apps and transports can't deal with  
these events when they happen. However, dealing with these events  
usually has some negative performance impact.

The question is if any of the proposed layer-3 extensions increase the  
probability of these events happening to a degree where user-perceived  
performance is significantly impacted.

And the answer may well be "no, they don't", but I'd be good to back  
that statement up with data.

Lars

---

From: David Conrad <[email-masked]>
Date: Tue, 12 Feb 2008 14:08:45 -0800

Lars,

> usually has some negative performance impact.

Yes.  And if the negative performance impact is sufficiently great,  
software and/or protocols and/or expectations are modified.  Or at  
least they have been in the past.

> The question is if any of the proposed layer-3 extensions increase  
> the probability of these events happening to a degree where user- 
> perceived performance is significantly impacted.
>
> And the answer may well be "no, they don't",

I'm sure they do.  Once again and with feeling: TANSTAAFL.

I'm a bit confused.  I thought we're talking here about modifying a  
key component of the underlying Internet architecture for the long  
term.  Such modification will have both positive and negative  
impacts.  The question isn't whether or not there will be an impact,  
rather the question is (or should be) does the benefit outweigh the  
cost.

Conceptually, as far as I can tell, we have a tradeoff.  All things  
being equal and relative to each other:

1) push-based systems will

a) not significantly increase latency/packet loss
b) be less scalable
c) allow less dynamicity

2) pull-based systems will

a) increase latency/packet loss, at least for the first packet of a  
new 'flow'
b) be more scalable
c) permit more dynamicity

Does anyone disagree with these assumptions?

> but I'd be good to back that statement up with data.

I agree data should be collected and loudly applaud the efforts of  
Dino et al. in writing actual code (I personally don't trust  
mathematical models as long ago I saw one too many models  
demonstrating how ATM was a perfect answer, regardless of the  
question).  However, from a conceptual point of view, I don't see how  
we can get away from the tradeoffs I mention above.  Whether or not  
you find option (1) or option (2) acceptable will likely largely  
depend on the assumptions you make going in -- we can all come up with  
scenarios that are unacceptable for either option.

  From my perspective, looking at how the Internet has evolved over  
time, scalability and dynamic behavior have been areas in which we've  
been bitten time and time again. We continue to see increased  
deaggregation.  We continue to see increased growth.  We continue to  
see increased dynamic behavior.  We also see increased bandwidth,  
cheaper memory, faster processors, etc. I don't see these changes  
reversing or even slowing down over time.  Thus, it seems to me we  
should anticipate and optimize for these changes instead of optimizing  
for the way things have been.

However, with all that soapboxing done with, I freely admit I'm not  
half as bright as most of the people on this list and it is likely I'm  
missing something fundamental.  I'd be appreciative if someone could  
explain it to me (using small words)...

Regards,
-drc

---

From: Robin Whittle <[email-masked]>
Date: Wed, 13 Feb 2008 20:41:31 +1100

Short version:

   We have choices other than pure push or pure pull.

   A suitably flexible hybrid push-pull system can be deployed
   according to all the prevailing conditions and provide
   minimal packet delays, due to the use of query servers which
   are always local - a lot closer and faster responding than
   relying on a global query server network of a pure pull
   system such as CONS or ALT.

David Conrad wrote:

> Conceptually, as far as I can tell, we have a tradeoff.  All
> things being equal and relative to each other:
>
>    a) increase latency/packet loss, at least for the first
>       packet of a new 'flow'
>
>    b) be more scalable
>

I agree with these statements for pure push and pure pull except
that a fast enough push system will allow dynamicity (faster changes
of mapping information) in accordance with its speed.

Also, the only way a pure pull system can achieve high rates of
mapping change is to reduce the cache time, which directly increases
the frequency of requests and responses.  A pull system could be
modified with "notify", so that when the mapping changes within a
certain time (less than the caching time of the ITR) after a request
was responded to, the query server (ETR in LISP-ALT) sends a
notification to every such ITR that the mapping has changed.  This
would be more complex, harder to make reliable and secure, and would
have bottlenecks, but would alleviate the problem of all ITRs
having to send queries and get responses ever minute in order to
maintain one minute "dynamicity".

We have choices other than pure push or pure pull.

APT is a hybrid push-pull system with default mappers, which are
local query servers which also function as full database ITRs.  All
the other ITRs are caching ITRs and can get their mapping data very
quickly from these local query servers.  Even if the caching ITR
didn't send the traffic packet (as a request for mapping) and have
it encapsulated by the default mapper, but held the packet until the
mapping data arrived, there would still be very little delay in
initial packets.

Ivip is a more flexible hybrid push-pull system.

Hybrid push-pull map-encap schemes need not delay initial packets by
more than a few tens of milliseconds (typical, unless the request or
response packet is dropped, which occasionally it will be), since
one or more local query servers and perhaps full database ITRs are
not far away.

(The LISP-NERD ID uses "hybrid" in a different way - ITRs might have
full mapping for some parts of the mapped address space and use a
query server network to get mapping for EIDs outside that space.
This is not the sort of "hybrid" I am referring to, though it may
have some benefits.)

I think the primary attraction of pure pull is that there is only
one copy of the database - fully distributed to the ETRs (or some
other kind of query server) which are closely associated with the
end-users.  That way, if an end-user wants to have a vast number of
EIDs, and to change their mapping every few seconds, this imposes no
cost on the rest of the system.

The primary objection to pure push is that every ITR in the world
would need to get every update for every EID, requiring a lot of
traffic and storage for updates which are never in fact used.

Hybrid push-pull enables mapping data to be pushed to some point -
for full database ITRs and query servers - and for the rest of the
ITRs to be caching ITRs getting mapping data quickly from these
local query servers.

In order for a pure pull system to be shown to be optimal, I think
it would be necessary to show that every possible hybrid push-pull
system would be unreasonably expensive, impractical etc. considering
the benefits they provide.

I need to write up my ideas for Ivip's fast push mapping
distribution system to show it is practical and desirable,
considering the benefits (minimal packet delay and support for a
new, efficient, kind of mobility).

I will do this as soon as I can, along with rewriting the rest of
the material in a more approachable fashion and updating my approach
to PMTUD and fragmentation.

A hybrid system such as Ivip - with notification from the full
database query servers to all ITRs which recently queried some
mapping information which has changed - will be highly flexible.  It
could deployed according to local conditions, traffic patterns, the
technology of the day and according to the size and rate of change
of the mapping database.

I will argue in greater detail later that even when the largest
realistic estimates of number of EIDs/micronets eventuate, it will
always will be practical and desirable to push to some number of
full database query servers around the Net.

That number is likely to be in the tens to hundreds of thousands,
but even if it was only to a hundred query server sites, that would
still solve the initial packet delay problem, since ITRs would get
their mapping information from the nearest such site which is far
faster and more reliable than relying on a global query server
network such as ALT.

Also, I think it is far more modular, secure and manageable to
control the mapping information with a completely separate system
which has nothing necessarily to do with ETRs.  NERD, Ivip and I
think APT either allow or require this, while my impression of ALT
is that the ETRs are typically, or always, the authoritative source
of mapping information.

---

From: Stig Venaas <[email-masked]>
Date: Wed, 13 Feb 2008 10:57:38 +0100

>    system such as CONS or ALT.

The other thing I would say, is that we have a tradeoff. But we
may need to look into host/stack/application behaviour to know
how important the various factors are when doing that tradeoff.

The main thing IMO is to have scalable routing, and some
in-optimal application behaviour or host/stack/application
changes may be a small price to pay. However, we should look
into those to make a more informed decision, not ignore them
until later. I guess this is where we need input from transport
or applications people...

Stig

---

From: Lars Eggert <[email-masked]>
Date: Wed, 13 Feb 2008 13:50:03 +0200

On 2008-2-13, at 11:57, ext Stig Venaas wrote:
> changes may be a small price to pay.

Makes complete sense from a routing perspective, because someone else  
is feeling the pain and the routers are seeing the gain.

But there are maybe a few million routers in the world, and maybe even  
fewer for which scalability is a serious issue, compared to a few  
billion hosts. Deploying something new at the network layer that might  
require changes to host stacks or applications to maintain performance  
at current levels would be a non-starter.

Lars

---

From: Robin Whittle <[email-masked]>
Date: Wed, 13 Feb 2008 23:14:36 +1100

I agree entirely with Lars.

  - Robin

> But there are maybe a few million routers in the world, and maybe
> even fewer for which scalability is a serious issue, compared to
> a few billion hosts. Deploying something new at the network layer
> that might require changes to host stacks or applications to
> maintain performance at current levels would be a non-starter.

---

From: David Meyer <[email-masked]>
Date: Wed, 13 Feb 2008 08:11:24 -0800

On Tue, Feb 12, 2008 at 02:08:45PM -0800, David Conrad wrote:
> software and/or protocols and/or expectations are modified.  Or at least 
> they have been in the past.
>
> I'm a bit confused.  I thought we're talking here about modifying a key 
> component of the underlying Internet architecture for the long term.  
> Such modification will have both positive and negative impacts.  The 
> question isn't whether or not there will be an impact, rather the 
> question is (or should be) does the benefit outweigh the cost.
>
> a) increase latency/packet loss, at least for the first packet of a new 
> 'flow'
> b) be more scalable

	I for one don't, but might note that 

	(i).	b) and c) are related by the (rate*state) product, and
	
	(ii). (rate*state) interacts strongly with a)

	For example, a push based model would (in theory) inflate
	the amount of state that some systems must carry while
	(in theory) minimizing lookup latency, while with a
	pull model, you minimize state but (in theory) increase
	lookup latency. Update rate would seem to be an issue in
	both cases (only for different reasons).

	The point being that the mapping system proposals we've
	seen represent points in an (at least) 3-dimensional
	space (rate, state, latency); where you land in that
	space encodes the tradeoffs made in the mapping system
	design. 

> I agree data should be collected and loudly applaud the efforts of Dino 
> et al. in writing actual code (I personally don't trust mathematical 
> models as long ago I saw one too many models demonstrating how ATM was a 
> perfect answer, regardless of the question).  

	There is something to pragmatism, isn't there. 

> However, from a conceptual point of view, I don't see how we
> can get away from the tradeoffs I mention above.  

	"Well, not unless you can alter time, speed up the
	harvest, or teleport me off this rock."

	But really, the tradeoffs you outline are fundamental
	(i.e., there is a physics to all of this).

> Whether or not you find option (1) or option (2) acceptable
> will likely largely depend on the assumptions you make going in
> -- we can all come up with scenarios that are unacceptable for
> either option.

	Yep. The important point you are making is that there are
	tradeoffs made in each design, and those tradeoffs need
	to be evaluated the in light of actual data/experience
	(at least I think that is what you are saying). 

	And will one size fit all? Does one size need to fit all?
	Hard to say at this point.

> From my perspective, looking at how the Internet has evolved over time, 
> scalability and dynamic behavior have been areas in which we've been 
> bitten time and time again. We continue to see increased deaggregation.  
> We continue to see increased growth.  We continue to see increased 
> dynamic behavior.  We also see increased bandwidth, cheaper memory, 
> faster processors, etc. I don't see these changes reversing or even 
> slowing down over time.  Thus, it seems to me we should anticipate and 
> optimize for these changes instead of optimizing for the way things have 
> been.

	Excellent point, and I most strongly agree.

	Dave

---

From: Tony Li <[email-masked]>
Date: Wed, 13 Feb 2008 10:06:12 -0800

>> maintain performance at current levels would be a non-starter.

Oh good, so then we can cancel IPv6 too?

Tony

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 13 Feb 2008 10:15:17 -0800

> Makes complete sense from a routing perspective, because someone  
> else is feeling the pain and the routers are seeing the gain.
>
> even fewer for which scalability is a serious issue, compared to a  
> few billion hosts. Deploying something new at the network layer that  
> might require changes to host stacks or applications to maintain  
> performance at current levels would be a non-starter.

Lars, it's not an either or thing, and just because there are more  
hosts than routers or whatever metric you want to use, there *is*  
going to be pain. We can share it or we can put it all in one place.

My experience is, if you wanted to put it all in one place, you really  
can't. So we are going to share it.

Dino

---

From: [email-masked]
Date: Wed, 13 Feb 2008 13:56:59 EST

This non-starter arguments sounds like a repeated objection against what I  
proposed a while ago.
But, whoever were committed to openess, could have easily improved/modified  
my proposal  to become a pure router task without involving  any host.
 
Heiner

---

From: Lars Eggert <[email-masked]>
Date: Thu, 14 Feb 2008 11:20:55 +0200

> going to be pain. We can share it or we can put it all in one place.

I agree it's not an either-or decision. And it is actually possible to  
combing routing scalability changes with other changes at the  
internetworking layer, so that application and transport performance  
*increases* rather than decreases.

For example, an RRG proposal that attempted to offset lookup-induced  
delays/inefficiencies through enabling QuickStart/XCP/RCP or maybe  
some other to-be-defined explicit feedback scheme would be *extremely*  
interesting, IMO.

(Yes, making use of the explicit feedback will require changes to the  
hosts, too. But at least there'd be a strong incentive for the hosts  
to change.)

Lars

---

From: Scott Brim <[email-masked]>
Date: Thu, 14 Feb 2008 07:55:45 -0500

Excerpts from Lars Eggert on Thu, Feb 14, 2008 11:20:55AM +0200:
>> Lars, it's not an either or thing, and just because there are more hosts 
>> than routers or whatever metric you want to use, there *is* going to be 
>> pain. We can share it or we can put it all in one place.
>
> delays/inefficiencies through enabling QuickStart/XCP/RCP or maybe some 
> other to-be-defined explicit feedback scheme would be *extremely* 
> interesting, IMO.
>
> hosts, too. But at least there'd be a strong incentive for the hosts to 
> change.)

Have we seen any explicit example problems wrt transport performance
yet?

---

From: Lars Eggert <[email-masked]>
Date: Thu, 14 Feb 2008 17:29:22 +0200

> Have we seen any explicit example problems wrt transport performance

I at least haven't seen any reports on this, but I don't know if  
that's because people have looked and not found any issues, or if  
that's because nobody is looking.

(Again: I'm not claiming that there will be issues - I'm only saying  
that we need to verify whether there are or not. And I'm not concerned  
about any particular proposal either - we need to investigate the  
higher-layer impact for any proposal we decide to take forward.)

Lars

---

From: Scott Brim <[email-masked]>
Date: Thu, 14 Feb 2008 10:36:14 -0500

Excerpts from Lars Eggert on Thu, Feb 14, 2008 05:29:22PM +0200:
> I at least haven't seen any reports on this, but I don't know if that's 
> because people have looked and not found any issues, or if that's because 
> nobody is looking.
>
> (Again: I'm not claiming that there will be issues - I'm only saying that 
> we need to verify whether there are or not. And I'm not concerned about any 
> particular proposal either - we need to investigate the higher-layer impact 
> for any proposal we decide to take forward.)

Sounds great, and now is a good time to do it, but I'm not a transport
layer guy.  How can we verify whether there will or will not be
issues?  That sounds rather hard :-).  Could you put up a specific
possible problem for us to discuss?

---

From: [email-masked] (Noel Chiappa)
Date: Thu, 14 Feb 2008 13:20:23 -0500 (EST)

> From: Robin Whittle <[email-masked]>

    > A suitably flexible hybrid push-pull system can be deployed

Yes, but... A hybrid system is inevitably going to be more complex, and I
think we should avoid complexity unless absolutely unavoidable.

(This statement may come as a bit of a shock to some people, given some of
the complex systems I have been involved with in the past, such as:

  http://ana-3.lcs.mit.edu/~jnc/nimrod/nimrod_block_30_60.jpg

but I refer you to the end of my statement - "unless ... unavoidable".)

Complexity is bad for the whole usual bunch of reasons, but one big one in
this case (which isn't applicable in all cases) is that such a system will
need extra configuration, and we all know that extra configuration is often
problematic.

    > a lot closer and faster responding than relying on a global query
    > server network of a pure pull system such as CONS or ALT.

You can have systems which are 'pure pull' but which have the property that
queries are answered "a lot closer and faster", close to as fast as a push
system - just add intelligent caching. There's no reason on G-d's green earth
why a query for "google.com" shouldn't be answered at the nearest possible
node in the pseudo-hierarchy.

Best of all, with good caching, you get most of the speedup of a mixed
push/pull system with i) *no* configuration, and ii) almost no extra
mechanism (the caching I finally designed for CONS uses one counter in every
entry, and 1 bit in responses).

So I'd really like to see us try an optimized 'pull' system (one that
includes the optimization of forwarding un-mapped packets over an alternative
forwarding structure, i.e. no waiting for the binding to arrive, which could
worst-case be a complete round-trip, even with caching), and see if that
provides an acceptable level of service, before we get into the complexities
of hybrid push/pull systems.

    > the only way a pure pull system can achieve high rates of mapping
    > change is to reduce the cache time .. A pull system could be modified
    > with "notify", so that when the mapping changes within a certain time
    > .. after a request was responded to, the query server ..  sends a
    > notification to every such ITR that the mapping has changed.

There are other ways to do this. E.g. the ETR could notice that the packet is
for a destination EID which is no longer at that location, and return a "not
here now" error message to the source ITR. I haven't thought about it hard,
to find the best optimization; this is another place where there's a complex
tradeoff between performance, complexity and overhead.

	Noel

---

From: [email-masked] (Noel Chiappa)
Date: Thu, 14 Feb 2008 13:25:13 -0500 (EST)

> From: David Conrad <[email-masked]>

    > looking at how the Internet has evolved over time, scalability and
    > dynamic behavior have been areas in which we've been bitten time and
    > time again. .. We also see increased bandwidth, cheaper memory, faster
    > processors, etc. I don't see these changes reversing or even slowing
    > down over time. Thus, it seems to me we should anticipate and optimize
    > for these changes instead of optimizing for the way things have been.

Yes - optimizing for the current state of things has always screwed us (the
most notable being the change from variable-length addresses in IPv3 to the
fixed 32-bit addresses of IPv4). We really need to pick architectures that
have a lot of headroom and flexibility, and not pick systems which are
narrowly tailored to the current state of affairs.

We're jacking up the internetwork layer and putting a whole new layer in
there, and also adding another layer of binding. That's a Big Step. Let's do
something significant, not just the minimal patch.

	Noel

---

From: David Conrad <[email-masked]>
Date: Thu, 14 Feb 2008 14:02:40 -0500

Robin,

> system such as CONS or ALT.

I agree a hybrid system is likely the best choice (one could argue  
that even the DNS is a hybrid system). The devil is in the details, of  
course.  I look forward to reading your proposal.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Thu, 14 Feb 2008 17:40:10 -0500

Lars,

> performance at current levels would be a non-starter.

We already have to change host stacks and applications to support  
IPv6.  Given the minimal deployment of IPv6 to date, particularly  
within the application space, this doesn't seem like a non-starter to  
me.

And to be clear, most applications that exist today (that have been  
ported to IPv6) wouldn't need to change.  Even in the worst case pure  
pull-based system, the only applications that would need to change  
would be those that are very sensitive to delays in the initial flow  
RTT.  To date, there has been mention of one (channel changing) on  
this list.  I'm sure there are others and I'd be interested in  
understanding their constraints, but I imagine those applications are  
in a distinct minority.

Regards,
-drc

---

From: Robin Whittle <[email-masked]>
Date: Fri, 15 Feb 2008 13:53:58 +1100

Hi Noel,

You wrote, in part:

> Yes, but... A hybrid system is inevitably going to be more
> complex, and I think we should avoid complexity unless absolutely
> unavoidable.
> 
> Complexity is bad for the whole usual bunch of reasons, but one
> big one in this case (which isn't applicable in all cases) is
> that such a system will need extra configuration, and we all know
> that extra configuration is often problematic.

Yes, I am keeping an eye on that.

>> a lot closer and faster responding than relying on a global
>> query server network of a pure pull system such as CONS or ALT.
>> 
> 
> You can have systems which are 'pure pull' but which have the
> property that queries are answered "a lot closer and faster",
> close to as fast as a push system - just add intelligent caching.

However if you expect those local caches to accumulate a large body
of information - large enough to make reliance on the slow ALT
global query server system rare enough not to be much of a problem -
then you are going to have to cache mapping information for a long
time.  This has the severe cost that you either need to accept low
rates of mapping change by end-users, or you need to add complexity
in the form of a "notify" system when the mapping is changed.

Even with such a caching scheme with notification, the worst case
situation will occur frequently enough to be a blight on this new
kind of address space: where the mapping and initial packet delivery
depends on the slow and not necessarily reliable ALT network.

I would rather have a clean, ideally fast, push system to get the
mapping data out to lots of ITRs and full database query servers.

> There's no reason on G-d's green earth why a query for
> "google.com" shouldn't be answered at the nearest possible node
> in the pseudo-hierarchy.
> 
> Best of all, with good caching, you get most of the speedup of a
> mixed push/pull system with i) *no* configuration, and ii) almost
> no extra mechanism (the caching I finally designed for CONS uses
> one counter in every entry, and 1 bit in responses).
> 
> So I'd really like to see us try an optimized 'pull' system (one
> that includes the optimization of forwarding un-mapped packets
> over an alternative forwarding structure, i.e. no waiting for the
> binding to arrive, which could worst-case be a complete
> round-trip, even with caching), and see if that provides an
> acceptable level of service, before we get into the complexities 

I think this definitely should be pursued in parallel with hybrid
push-pull arrangements.

>> the only way a pure pull system can achieve high rates of
>> mapping change is to reduce the cache time .. A pull system
>> could be modified with "notify", so that when the mapping
>> changes within a certain time .. after a request was responded
>> to, the query server ..  sends a notification to every such ITR
>> that the mapping has changed.
> 
> There are other ways to do this. E.g. the ETR could notice that
> the packet is for a destination EID which is no longer at that
> location, and return a "not here now" error message to the source
> ITR. I haven't thought about it hard, to find the best
> optimization; this is another place where there's a complex 

But what if the ETR is not reachable?  This reliance on ETRs for
functions other than decapsulating packets and responding to
reachability and PMTU probing strikes me as problematic and overly
complex.  For one thing, you have to have inter-ETR communication.
For another you need a lot of *configuration* and authentication
stuff, since an ETR is likely to be owned by an ISP, but it would
need a way of proving to ITRs all around the Net (and the ALT
network etc.) that it is one of the authoritative sources of mapping
information for an EID which is assigned to an end-user.

With a push or hybrid push-pull system, the ETRs are not involved in
the mapping information at all.  So a whole bunch of ETRs can
disappear and the system still works fine, using other ETRs.

> We're jacking up the internetwork layer and putting a whole new
> layer in there, and also adding another layer of binding. That's a
> Big Step. Let's do something significant, not just the minimal
> patch.

Yes, so let's think boldly.

Extra complexity is sometimes necessary to cope with some problem or
effect some kind of trade-off.  This is a bore, but can at times be
justified.

Complexity is better justified when it does more than round the
edges off an uncomfortable design.  When one piece of complexity
solves more than one significant problem, or provides multiple
benefits, then it is really earning its keep.

The most radical thing about Ivip is the proposal to push mapping
data fast - a few seconds or so - to as many full database ITRs and
Query Servers as people feel like running.  I believe this
complexity is justified by multiple benefits:

  1 - Simpler mapping information - just an ETR address.
      (A cost of this is no explicit TE in the ITRs - end
      users need to do load sharing by using small micronets,
      including single IP addresses, and pointing some to one
      ETR, some to another etc.)

  2 - Removing the need for ITRs to test for reachability,
      make decisions about which ETR to use etc.  (ITRs will
      still need to do some PMTUD and fragmentation work.)

  3 - Removing from the map-encap system the process of
      determining in real-time where the packets are forwarded
      to.  Instead, that is controlled by the end-user, in
      whatever way they like.  This modularisation is a major
      feature of Ivip.

  4 - Support for a new kind of IPv4/6 mobility with optimal
      or typically close to optimal path lengths, without
      fundamental change in the TCP/IP stack or applications.

The future map-encap architecture is at least as big a step as
designing IPv6 - because if we get it right, it is assured of
widespread adoption.  So I think we should be bold and contemplate
additional and novel complexity if it brings us fruitful results.

---

From: Stig Venaas <[email-masked]>
Date: Fri, 15 Feb 2008 09:51:20 +0100

David Conrad wrote:
> We already have to change host stacks and applications to support IPv6. 
> Given the minimal deployment of IPv6 to date, particularly within the
> application space, this doesn't seem like a non-starter to me.
> 
> pull-based system, the only applications that would need to change would
> be those that are very sensitive to delays in the initial flow RTT.  To

I agree

> date, there has been mention of one (channel changing) on this list. 

I'm not sure if channel changing need to pose a problem. E.g. if they
can be streamed from the same source address. If they are streamed from
multiple addresses but covered by the same locator mapping (assuming
that you can get a single id-prefix/range to locator mapping from the
mapping service), this might also be fine.

> I'm sure there are others and I'd be interested in understanding their
> constraints, but I imagine those applications are in a distinct minority.

I'm thinking of trying to study some particular application, or DNS
resolve implementation or something, myself. If I find some time to
spare...

Stig

---

From: Lars Eggert <[email-masked]>
Date: Fri, 15 Feb 2008 11:30:53 +0200

On 2008-2-15, at 0:40, ext David Conrad wrote:
>> few billion hosts. Deploying something new at the network layer  
> within the application space, this doesn't seem like a non-starter  
> to me.

We've had to change (mostly) the network layer of hosts stacks for  
IPv6. But here we're talking about changing transport layers and (UDP- 
speaking) applications. That's a different magnitude of change.

> ported to IPv6) wouldn't need to change.  Even in the worst case  
> pure pull-based system, the only applications that would need to  
> change would be those that are very sensitive to delays in the  
> initial flow RTT.  To date, there has been mention of one (channel  
> changing) on this list.  I'm sure there are others and I'd be  
> interested in understanding their constraints, but I imagine those  
> applications are in a distinct minority.

I'm a bit hesitant to make predictions without having at least  
simulated things, but TCP is somewhat sensitive to packets being  
delayed, lost or reordered during the first few round-trips, and  
that's independent of what application runs on top of TCP.

Obviously, for a long bulk transfer, a performance decrease due to  
delay/loss/reordering during the first few round-trips won't matter  
much when looking at performance of the whole transfer, but for  
shorter transfers (think web traffic), the user-perceived performance  
decrease can be substantial.

Now, that said, if this is serious or not depends a lot on how  
frequently these events will happen, or, in other words, what caching  
or pre-fetching of mapping information a given RRG proposal employs,  
and what the overall traffic matrix is like.

I'll pitch again the idea of combining mapping-based routing with an  
explicit-feedback scheme. If we're making changes to routers that may  
potentially decrease end-to-end performance of some communication  
sessions in some cases, providing the communicating end systems with  
congestion/capacity information that lets them offset the performance  
loss or maybe even increase performance seems to be a very attractive  
option, IMO.

Lars

---

From: [email-masked] (Noel Chiappa)
Date: Fri, 15 Feb 2008 10:00:25 -0500 (EST)

>> You can have systems which are 'pure pull' but which have the property
    >> that queries are answered "a lot closer and faster", close to as
    >> fast as a push system - just add intelligent caching.

    > if you expect those local caches to accumulate a large body of
    > information - large enough to make reliance on the slow ALT global
    > query server system rare enough not to be much of a problem

First, I'm not assuming any particular design (especially as it stands
today), be it ALT, CONS, or anything else.

Second, it's not at all clear to me that a cache needs to accumulate "a large
body of information" to get a fairly high hit rate. (Or, at least, to get
over the knee where you get into the diminishing returns region, where adding
extra cache entries only slightly increases the hit rate - at that point, to
get a big increase in performance, you have to go to full push.)

    > time. This has the severe cost that you either need to accept low rates
    > of mapping change by end-users, or you need to add complexity in the
    > form of a "notify" system when the mapping is changed.

This doesn't currently seem to be a major problem with DNS; why do you think
it will be a problem here?

Yes, if we want to do e.g. mobility with this mapping layer, we'll have to
look at the question of outdated mappings, but there are a number of
approaches to solving that, depending on where one wants to land in
performance/complexity/overhead space.

    > the worst case situation will occur frequently enough to be a blight on
    > this new kind of address space: where the mapping and initial packet
    > delivery depends on the slow and not necessarily reliable ALT network.

Who's talking about ALT?

    > mapping data out to lots of ITRs and full database query servers.

If you're sending it to ITRs, it's full push.

The notion of having a widely distributed set of full database query servers
is an interesting one; let me ponder that. Right off the bat I see some
issues (e.g. how do you assemble a "full database" when it's inherently a
distributed artifact - it's the same problem as 'how would you assemble a
full DNS database), but without thinnking it through I don't want to give an
opinion.

    > But what if the ETR is not reachable?

We clearly need to have mechanisms to track whether ETR's are up or not, lest
we produce the same kind of 'black hole' problems we have with routers that
die.

    > This reliance on ETRs for functions other than decapsulating packets
    > and responding to reachability and PMTU probing strikes me as
    > problematic and overly complex.

First, I'm not sure we want ITRs individually probing ETRs to see if they are
alive. This kind of approach was rejected in other circumstances (e.g.
host-router reachability) because of traffic load issues.

Second, why would it be "problematic and overly complex" to have other
functionality in the ETR's? Once the ITR gets the EID->RLOC binding for a
given EID, the ITR is normally only communicating with the ETR at that RLOC
anyway, so it's natural to emplace maintainence functionaltiy in the ETR -
it's doing anything *else* that's more complex.

    > For one thing, you have to have inter-ETR communication.

Why? Unless you want fast (i.e. less than an extra round-trip delay)
handover, all the ETR has to know is 'this EID is no longer at me' - from
there, the ITR has to do another binding lookup.

Again, there's a cost/performance/complexity tradoff.

    > stuff, since an ETR .. would need a way of proving to ITRs all around
    > the Net (and the ALT network etc.) that it is one of the authoritative
    > sources of mapping information for an EID which is assigned to an
    > end-user.

No. If you did have ETRs providing a new binding, they'd just have to get a
signed binding (from the entity which is normally authorized to hand out
those bindings), and pass that along.

	Noel

---

From: Marshall Eubanks <[email-masked]>
Date: Fri, 15 Feb 2008 11:07:14 -0500

On Feb 15, 2008, at 4:30 AM, Lars Eggert wrote:

> IPv6. But here we're talking about changing transport layers and  
> (UDP-speaking) applications. That's a different magnitude of change.

The experience with Multicast in this century has been that stack +  
application changes, even with vendor support (such as IGMPv3 + SSM),  
take about a decade to propagate into something like a critical mass.  
Hasn't, for that matter, IPv6 also taken about a decade to get to  
where we can consider IPv6 only plenaries ? If anything, I would
regard that delay as likely to increase in the future.

There is probably a role for changes intended for the far future, but  
I wouldn't do it lightly.

Regards
Marshall

>
>> And to be clear, most applications that exist today (that have  
>> been ported to IPv6) wouldn't need to change.  Even in the worst  
>> case pure pull-based system, the only applications that would need  
>> to change would be those that are very sensitive to delays in the  
> shorter transfers (think web traffic), the user-perceived  
> performance decrease can be substantial.
>
> frequently these events will happen, or, in other words, what  
> caching or pre-fetching of mapping information a given RRG proposal  
> employs, and what the overall traffic matrix is like.
>
> I'll pitch again the idea of combining mapping-based routing with  
> an explicit-feedback scheme. If we're making changes to routers  
> that may potentially decrease end-to-end performance of some  
> communication sessions in some cases, providing the communicating  
> end systems with congestion/capacity information that lets them  
> offset the performance loss or maybe even increase performance  
> seems to be a very attractive option, IMO.
>
> Lars
>
> --

---

From: Marshall Eubanks <[email-masked]>
Date: Fri, 15 Feb 2008 13:23:45 -0500

On Feb 15, 2008, at 3:51 AM, Stig Venaas wrote:

>>> even
>> IPv6.
>> would

 From my point of view, this could be viewed as just a CDN design  
constraint (i.e.,
if streams need to be sources from the same /56 to have fast change,  
then they can be), but I
know of others who would not like this.

Here is a toy model : I am sending you a 2 Mbps video stream, and you  
are on a 100 Mbps connection.
I have a 2 second "group of pictures" or GOP in the MPEG sense, so  
you cannot
see a good image until on average 1 second has passed, and ~ 2  
seconds in the worst case. To this
time has to be added the RTT time (you to signal me that you want a  
new channel, and the data
to get from me to you), plus some time for routers and servers to do  
their thing.

Fast channel change / fast start means that I send to you initially  
at a fast rate, so that the first 2 seconds
of a new channel are sent in, say, 0.04 seconds (using your full wire  
rate). If your RTT is 100 msec,
and internal stuff takes, say, 50 msec, you could be receiving a new  
channel 200 msec after you
push the button on the remote, and those sorts of times are viewed as  
the desired goal.
After 0.05 seconds (in this toy model), the rate drops down to the  
long term value. Something like this is done by, e.g., Quicktime  
Streaming Server right now, with initial QTSS bandwidths being  
routinely x2 or x3 the long term. (Both live streams and VOD can  
benefit from this.)

If all of this is coming from one server farm, all is well and good.  
But, what if (like Joost) these
streams are being provided by P2P ? Each new P2P node would have this  
delay, which will definitely hinder fast start and may caP2P scales  
well for fast start, as you can have more nodes send stuff
in the beginning, but this would destroy that advantage as well. So,  
people looking to replace CDN's with P2P meshes for streaming would  
probably not like this.

I have seen various plans for in-line advertising where ads come from  
different IP addresses than the regular stream, and this would also  
mess that up, although there might be engineering work-arounds to that.

Also, you consider not just unicast, but SSM multicast (would every  
SSM group join be delayed which the SSM unicast address is being  
resolved ?).

Note, too, that many simple web pages have in them pieces from other  
servers and other IP addresses (banner ads
are frequently done this way) and so this could put a real  
performance hit on web page load times in the real world.

Regards
Marshall

---

From: Stig Venaas <[email-masked]>
Date: Fri, 15 Feb 2008 20:08:06 +0100

> On Feb 15, 2008, at 3:51 AM, Stig Venaas wrote:
[...]
> delay, which will definitely hinder fast start and may caP2P scales well 
> for fast start, as you can have more nodes send stuff
> different IP addresses than the regular stream, and this would also mess 
> that up, although there might be engineering work-arounds to that.

Right. I'm with you. After posting, I also started thinking of various
cases where having source addresses within some common prefix/location
for all the streams would be difficult.

> Also, you consider not just unicast, but SSM multicast (would every SSM 
> group join be delayed which the SSM unicast address is being resolved ?).

Yes, I expect that would be the case, unless the mapping already was in
place.
> are frequently done this way) and so this could put a real performance 
> hit on web page load times in the real world.

Yes, I can think of some ways to reduce the problem, but anyway, I think
it's good to study such scenarios.

Stig

---

From: David Conrad <[email-masked]>
Date: Sun, 17 Feb 2008 10:54:24 -0800

Lars,

> (UDP-speaking) applications. That's a different magnitude of change.

The lack of backward compatibility has meant we have had to change  
every network aware application ever written to support IPv6. Granted,  
many of those changes are 'simple' API changes (some of which might  
even be hidden by compatibility libraries), but they are still changes.

What I presume you are concerned about is the potential need to change  
application (or transport layer) logic to support a potentially longer  
RTT for an initial packet.  However, since applications (and transport  
layer implementations) already need to deal with the potential of a  
delayed or dropped initial packet, it seems to me this is a question  
of optimization rather than feasibility.

Regards,
-drc

---

From: Stig Venaas <[email-masked]>
Date: Sun, 17 Feb 2008 20:31:47 +0100

David Conrad wrote:
> The lack of backward compatibility has meant we have had to change every 
> network aware application ever written to support IPv6. Granted, many of 
> those changes are 'simple' API changes (some of which might even be 
> hidden by compatibility libraries), but they are still changes.
> 
> delayed or dropped initial packet, it seems to me this is a question of 

My experience is that some applications don't handle delayed or dropped
initial packets all that well. This may be okay since it rarely happens,
but it's a problem if it becomes the norm. Part of the problem here is
how the application/transport can distinguish between a delay/drop due
to a cache-miss (i.e. mapping not in place) and a packet dropped for
other reasons. From a single application point of view it may be good
to retransmit the first packet multiple times in a short period of time,
to make sure that one gets through as soon as possible. I'm not sure if
it's good if applications did that in general.

Anyway, this bring me back to the point I made earlier, that we really
would need to look at some applications. I guess you have some knowledge
how say BIND behaves? E.g., it looks like the host command will wait for
5s if the first request is dropped. Is a 5s delay to DNS lookups
acceptable?

Stig

> optimization rather than feasibility.
> 
> Regards,
> -drc
> 
> 
> --

---

From: Lars Eggert <[email-masked]>
Date: Mon, 18 Feb 2008 16:12:37 +0200

On 2008-2-17, at 21:31, ext Stig Venaas wrote:
> but it's a problem if it becomes the norm.

Exactly!

> Part of the problem here is
> other reasons.

Right. Most transport-related mechanisms assume that loss equals  
congestion, and they react by taking load off the (apparently)  
congested path.

Wireless networks are one example where loss can be due to corruption,  
and transports have struggled to deal with this case. (And, in truth,  
it's been addressed because the MAC designers have figured out how to  
deal with this, and not because the transport protocols had adapted.)

If the routing system becomes a new source of loss or re-orderings  
that aren't due to congestion, the issues will be similar, i.e.,  
congestion control will be triggered without congestion being present.  
How serious this is depends on how frequently that can happen, exactly  
as you state above.

> From a single application point of view it may be good

It wouldn't be a good idea from a congestion control perspective,  
because the initial phase of a connection is exactly when the  
transport protocols have the least knowledge about the path  
characteristics (bandwidth, delay, etc.), because they have no sampled  
path state, so that's when they need to be most conservative.

> 5s if the first request is dropped. Is a 5s delay to DNS lookups

DNS applications are a good case to look at, as is web traffic, SIP  
stacks, etc.

Lars

---

From: Eliot Lear <[email-masked]>
Date: Mon, 18 Feb 2008 16:22:57 +0100

Hi Stig,
> but it's a problem if it becomes the norm.

Yes, this is why I did NERD.  But let's please make a distinction 
between delay and drop.  I think they can and should be viewed separately.

Eliot

---

From: David Conrad <[email-masked]>
Date: Mon, 18 Feb 2008 08:55:06 -0800

Stig,

On Feb 17, 2008, at 11:31 AM, Stig Venaas wrote:
> initial packets all that well.

Which applications?

> would need to look at some applications.

Indeed.  And what it would take to get said applications to tolerate  
an initial packet delay better.

> I guess you have some knowledge how say BIND behaves?

I had some knowledge of how a previous incarnation of BIND behaved  
about 5 years ago.  BIND has evolved in the face of real-life Internet  
considerations.

> E.g., it looks like the host command will wait for

Not sure it is relevant since the DNS makes assumptions that would not  
necessarily hold in a pull-based mapping system for EID/LOC  
separation, e.g., a 5 second time out that can be interrupted by the  
user can be acceptable.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Mon, 18 Feb 2008 11:18:37 -0800

Lars,

>> but it's a problem if it becomes the norm.

Data? While I'm perfectly willing to accept this is the case (really),  
I would prefer to understand what these applications are, why they are  
the way they are, and what (if anything) can be done to mitigate the  
problem.

> Most transport-related mechanisms assume that loss equals  

...
> characteristics (bandwidth, delay, etc.), because they have no  
> sampled path state, so that's when they need to be most conservative.

Indeed.  And since they have no sampled path state, being the most  
conservative typically means having a timeout at some fixed value,  
potentially exponentially increasing that timeout after the  
retransmit. I'm not sure how I see this needing to change if you add a  
few hundred more msec delay for the first packet.

> DNS applications are a good case to look at, as is web traffic, SIP  

DNS isn't a particularly good case (IMHO) as it isn't all that  
sensitive to initial delay.  Web traffic (particularly due to "link  
dense" web pages) might be.  SIP stacks, maybe, but I don't see why  
they'd be all that sensitive to initial delay (since people are  
already used to dealing with significant delay during call setup).   
Maybe ENUM-related services?

Regards,
-drc

---

From: Jari Arkko <[email-masked]>
Date: Mon, 18 Feb 2008 14:06:09 -0800

Folks,

It is indeed true that in some cases there is no such thing as a free
lunch. And keeping the Internet up and running is important. As we
invent new pieces of technology -- such as a new routing architecture in
this case -- there may effects to the other parts of the Internet system.

However, we should not jump from this to the conclusion that we do not
need to understand or measure the effects to applications because "we
have to do it anyway". We do need to understand the effects, for a
variety of reasons:

- We want to choose the best solution, so understanding the differences
between approaches on this aspect is necessary.

- If the effects are truly significant, we may have to go back to the
drawing board to invent better solutions.

- If it turns out that a certain effect is unavoidable, we want to make
sure that it is still acceptable compared to the alternative of not
having any solution at all. I.e., while fixing the Internet please don't
break it.

- If the effects can be avoided by a significant increase of complexity,
will it be worth the effort?

- Deployment incentives for people installing either the new routing
system or the needed host/apps modifications matter.

In any case, I am completely in the dark with regards to how serious
this problem is in real life. And frankly, this thread is not giving me
much new information in that regard. It could be that its not a problem
at all. Or it is. I would suggest that we take this issue seriously and
make sure there is someone making a measurement of what the actual
impacts would be. Data needed! We don't necessarily have to have a
complete new routing system to test or simulate this; it would be
sufficient to introduce the drop/delay-the-first-packet effect.

The other thing we should do is to go back to the tradeoff discussion
that David started; I found that useful. The questions that I have are:
1) Wouldn't a hybrid scheme be able to reduce the incidence of this
problem? 2) David's tradeoffs assumed that we operate either in push or
pull model. However, has it been established that we actually need a
separation architecture that needs these mappings? What about research
ideas such as compact routing?

Jari

---

From: Marshall Eubanks <[email-masked]>
Date: Mon, 18 Feb 2008 17:26:24 -0500

Dear Jari;

> sufficient to introduce the drop/delay-the-first-packet effect.

Set this up for the plenary in Dublin. See who complains.

Or, set up such a net in the terminal room and invite people to test  
their applications, web sites,
etc.

Regards
Marshall

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 18 Feb 2008 23:18:11 -0800

> ideas such as compact routing?

We have to be careful of introducing a hybrid, more complex solution  
just to solve a dropped packet for the very first flow between a  
source-site and destination-site.

A robust system should also consider what problems not to solve.

Dino

---

From: Lars Eggert <[email-masked]>
Date: Tue, 19 Feb 2008 09:46:08 +0200

> between delay and drop.  I think they can and should be viewed  

 From a transport perspective, it matters whether delay can cause  
packet reordering to happen or not.

If not, the only effect will be, er, more delay. If yes, to most  
transports the reordering will look like a loss and cause a congestion  
control response. Which will decrease performance more than delay  
without reordering would.

Lars

---

From: Eliot Lear <[email-masked]>
Date: Tue, 19 Feb 2008 13:50:40 +0100

> packet reordering to happen or not.

Right.  But if the delay is sparse (every few minutes) and we can limit 
discussion to the "middle" of a communication, this begins to sound like 
a tractable problem, if it even is noticeable.  There are still odd 
ducks out there, and right now DNS is the biggest thing I see quacking.

> without reordering would.

And now we have to ask ourselves under what conditions such a thing 
would happen.  If it happens during a routing transition, perhaps it is 
a reasonable cost.  If it just Happens, perhaps not so much.

Eliot

---

From: Jari Arkko <[email-masked]>
Date: Tue, 19 Feb 2008 05:35:25 -0800

> A robust system should also consider what problems not to solve.
I very much agree -- that's why it would be very useful to know if the
first packet problem is bad enough that we need that extra baggage.

Jari

---

From: Marshall Eubanks <[email-masked]>
Date: Tue, 19 Feb 2008 10:59:40 -0500

Dear Jari;

I have spent enough time in places 200 to 300 msec RTT away from home  
to be fairly confident that most
things will work. I suspect that the people on this list represent in  
aggregate many years of 200+ msec RTT experience, and I haven't heard  
anyone saying otherwise. Of course, significant amounts of out-of- 
order or dropped packets will cause problems, but there is an  
existence proof for you.

However, it is not enough to work; it also has to be salable.

I don't think that any system that imposes significant user delays  
will be commercially viable if other options are available, and it is  
hard to see how they would not be.

If I represent a service provider without these delays, and you  
represent one with these delays, I can guarantee
you that you will not like my marketing materials pointing this out,  
and telling potential customers something like "ask how long it takes  
Jarinet to connect to the web." I have been on the loosing end of  
such campaigns; it is hard to fight simple facts with complicated  
explanations of why said facts are irrelevant most of the time.

So, the delays cannot be "significant," which I would regard as  
meaning "being noticeably larger than typical DNS delays." Any such  
solution with significant user delays will be hammered in the  
marketplace.

This is indeed likely to impose significant engineering constraints.

Regards
Marshall

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 09:20:55 -0800

Earlier, Dino wrote:
% We have to be careful of introducing a hybrid, more complex solution
% just to solve a dropped packet for the very first flow between a
% source-site and destination-site.
%
% A robust system should also consider what problems not to solve.

It likely would be helpful if proposals would clearly document which
problems they are solving and also which they are not solving.

Ran

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 09:22:26 -0800

Earlier, Lars Eggert wrote:
% From a transport perspective, it matters whether delay
% can cause packet reordering to happen or not.

Are you thinking of the 4.3-Reno BSD issue where tcp_input()
performed visibly better if TCP packets arrived in order ?

Or something else ?
If something else, which issue(s) are you thinking of ?

Thanks,

Ran

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 19 Feb 2008 09:58:54 -0800

> first packet problem is bad enough that we need that extra baggage.

We should have plenty of data on this by summer IETF.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 19 Feb 2008 10:01:37 -0800

> problems they are solving and also which they are not solving.

If this is a comment against LISP, the lisp-alt draft indicates that a  
Data Probe or a Map-Request can be sent over the LAT topology. So if  
you don't mind drops, the Map-Request is sent to trigger a Map-Reply.  
If you don't want any packet loss (dropped intentionally at the ITR),  
then the Data Probe is used to solicit the Map-Reply.

So we don't state the problem, we state the solution. Is that not  
sufficient?

Dino

---

From: [email-masked] (Noel Chiappa)
Date: Tue, 19 Feb 2008 14:11:51 -0500 (EST)

> From: Marshall Eubanks <[email-masked]>

    > many simple web pages have in them pieces from other servers and other
    > IP addresses (banner ads are frequently done this way) and so this
    > could put a real performance hit on web page load times in the real
    > world.

Perhaps, but I'll note that interspersing of content in this manner also, in
theory, requires a DNS lookup for each of these included pieces, since they
are generally referred to via a DNS-based URL. (I can make that assertion
because I regularly trawl my web cache to find new sources of advertising
content which I can block, and I don't see many which are IP-address based.)

If the DNS resolution isn't proving problematic, it's probably because a
relatively small number of sites are involved (my list of blocked sites is
not that large), and the DNS records for them are getting cached relatively
close to the user machines (e.g. in a local DNS server). Ad content seems to
come either from the same server as the rest of the content, or from a small
number of ad providers.

That being the case, I would think the same would be true of EID->RLOC
bindings; there won't be that many, and in a system with decent caching,
requests for those bindings will be resolved 'close' to the ITR.

As a more general comment, I don't think we're going to be able to predict
accurately the performance of this kind of system (especially if it makes
heavy use of caches, as I think any well-designed system of the kind ought
to), nor the actual problems we are likely to see in service.

Networking work has a long history of theoretical analyses that purport to
show that approach X won't work - only to find that it does, in practise. (I
am reminded of one series of analyses by a faculty member at MIT-LCS which
'showed' that contention networks such as Ethernet wouldn't work 'well' in
high-load situations, which were a factor in the decision to work on token
rings at MIT. Ethernet, as we all know, does work quite well in practise,
because they basically never operate for long in those high-load regions.)

So I think probably the best thing to do is prototype a system, and try it
and see what happens.

	Noel

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 11:16:31 -0800

> problems they are solving and also which they are not solving.

Later, Dino wrote:
% If this is a comment against LISP, ...

My comment was notabout any particular proposal.

It is a suggestion that both "problems addressed/solved" and
also "problems out-of-scope/not solved" be very clearly,
very explicitly, and fully documented for each proposal.

Ran

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 19 Feb 2008 20:44:37 +0100

Noel,
> close to the user machines (e.g. in a local DNS server

Several researchers have analysed the performance of DNS. MIT did an 
interesting analysis in 2000, the main results form this analysis are 
the following :

Our most surprising, non-obvious findings and conclusions are:

     * About a quarter of all DNS lookups never get an answer. More than 
50% of the DNS-related packets in the wide-area correspond to such lookups!
     * The DNS retransmission protocol appears to be overly persistent: 
while most successful answers are received in at most 2-3 
retransmissions, the lack of an answer or response causes a much larger 
number of retransmissions and a corresponding number of DNS packets 
traverse the wide-area.
     * Replacing all (i.e., for non-name server hosts) the A-record 
TTL's to a value as small of 10 minutes is not likely to degrade the 
scalability of DNS in any noticeable way. This is because of the 
heavy-tailed nature of accesses to names.
     * The scalability of DNS has little to do with its hierarchical 
organization or to good A-record caching. Most of the DNS name space is 
a flat, two-level structure. A-record caching does not seem to add much 
more to the per-host or per-application caching done by end clients 
today. Rather, the scalability derives from the good name space 
partitioning achieved by the cacheability of NS records, which avoid 
load on the root and top-level name servers.

See http://nms.csail.mit.edu/projects/dns/

I think that for the DNS the RRG should probably also consider looking 
at real data...

Olivier

---

From: Brian Dickson <[email-masked]>
Date: Thu, 21 Feb 2008 21:43:10 -0500

> sufficient to introduce the drop/delay-the-first-packet effect.
While the impact to TCP-based things is likely to be 
noticable-but-modest (TCP has to handle loss,
including first-packet loss), I think UDP protocols may need more scrutiny.

The short list of things that I think need further testing or analysis 
or experience, regarding loss
or delay to the first packet:
ntp (think about it!)
snmp
snmptrap
l2tp
all the various *lm things (licence managers) - how many IT help desks 
to we want to anger at once, really?

These are all in use, big time, and likely to be important to any site. 
All are UDP. All need to work,
reliably, predictably, consistently. (Especially ntp.)
> ideas such as compact routing?
I think it would be good to re-examine alternatives to push/pull mapping 
schemes. They exist,
and like everything, have trade-offs.

How do those compare to push/pull for map-encaps and the trade-offs there?

Brian Dickson

---

From: "William Herrin" <[email-masked]>
Date: Mon, 25 Feb 2008 18:56:04 -0500

>  or delay to the first packet:

Brian,

Say, that's a good one.

NTP clients querying servers through a TRRP path with oscillate the
servers between eligible and ineligible each time the delay between
NTP queries gets long enough to cause the address to time out of the
ITR causing what the NTP client will perceive as excessive delay and
jitter.

Solving this in the short term requires no changes to NTP but it does
require some attention to configuration for those who choose to be
early adopters of TRRP space. Far-away NTP servers won't be usable
over a TRRP path so you'll need a machine on BGP routable space to
make those far-away queries and then you'll have to query that machine
over the local routing infrastructure that doesn't pass through BGP or
TRRP. Fortunately, TRRP's ETRs already meet this criteria and many
machines capable of running a GRE endpoint are also capable of running
NTP.

In the long term, NTP client software could be adjusted to notice the
unexpected delay phenomenon and either shorten the time between
queries to the point where the destination stays in the ITR's cache or
recognize the delay as transient and retry a few seconds later
ignoring the transient delayed result. The ITR will generally only
have to re-cache the bottom-level component, so that cache oscillation
shouldn't be harmful.

This will be an issue for every map-encap scheme that relies on caching.

>  snmptrap

Generally local; outside TRRP's area of effect.

>  l2tp

Don't know for sure but I don't think there's a problem with l2tp or
pptp over TRRP-managed address space. VPN protocols are generally
already designed to work over nasty, unpredictable networks.

>  to we want to anger at once, really?

Generally local; outside TRRP's area of effect.

Regards,
Bill Herrin

-- 
William D. Herrin [email-masked] [email-masked]
3005 Crane Dr. Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

---

From: Christian Vogt <[email-masked]>
Date: Tue, 26 Feb 2008 12:26:04 +0200

> Could you expand on this?  Assuming a pull system something along  
> the lines
> of what I described to Robin, why wouldn't it work for global  
> mobility?

David,

I should have been more specific, and maybe we are even on the same  
page.

It is possible to use the indirection between edge and transit addresses
as a tool for global mobility support, but alone it is insufficient.   
Three
more components are needed:

(1)  Localized mobility support within an edge network, because the  
address
      indirection alone tracks host mobility only at the granularity  
of edge
      networks, not access links.

(2)  Dynamic and per-host mappings between edge and transit addresses,
      because the mapping for a single edge address changes when a  
mobile
      host moves to another edge network

(3)  Trust relationships between edge networks so that a mobile host's
      mapping can be changed by (or based on information of) a visited  
edge
      network

Item (1) could be realized through, e.g., Hierarchical Mobile IPv6 or  
Proxy
Mobile IPv6.  Item (2) could be realized through pull systems for  
mapping
resolution.  Push systems would likely create an unsuitably high  
amount of
state as well as signaling and processing overhead if they were to  
provide
the required fine-granularity and dynamism.

I analyzed the feasibility of mobility support through address  
indirection
also in my paper on Six/One Router [1], section 3.5.  The analysis  
considers
mobility support through address indirection in general where  
possible, and
looks at Six/One Router specifically only where necessary.

[1]  http://users.piuha.net/chvogt/pub/2008/vogt-2008-six-one-router.pdf

- Christian

---

From: "William Herrin" <[email-masked]>
Date: Tue, 26 Feb 2008 11:52:51 -0500

On Tue, Feb 26, 2008 at 5:26 AM, Christian Vogt
<[email-masked]> wrote:
>  address indirection alone tracks host mobility only at the granularity
>  of edge networks, not access links.
>
>  because the mapping for a single edge address changes when a mobile
>  mapping can be changed by (or based on information of) a visited edge
>  network

Hi Christian,

I'm okay with your first two points but the third makes an assumption
that might not be warranted. You do need a trust relationship but it
doesn't necessarily need to be between edge networks. The trust
relationship can also be from the edge networks to one or many
third-party address authorities.

Also, note that item 1 can be achieved through a standard push-based
IGP like OSPF given some form of AAA system for the client.

Regards,
Bill Herrin

---

From: Scott Brim <[email-masked]>
Date: Tue, 04 Mar 2008 13:38:23 -0800

Please pardon my lateness ...

On 2/19/08 11:11 AM, Noel Chiappa allegedly wrote:
> As a more general comment, I don't think we're going to be able to
> predict accurately the performance of this kind of system (especially
> if it makes heavy use of caches, as I think any well-designed system
> of the kind ought to), nor the actual problems we are likely to see
> in service.
> 
> Networking work has a long history of theoretical analyses that
> purport to show that approach X won't work - only to find that it
> does, in practise. (I am reminded of one series of analyses by a
> faculty member at MIT-LCS which 'showed' that contention networks
> such as Ethernet wouldn't work 'well' in high-load situations, which
> were a factor in the decision to work on token rings at MIT.
> Ethernet, as we all know, does work quite well in practise, because
> they basically never operate for long in those high-load regions.)
> 
> So I think probably the best thing to do is prototype a system, and
> try it and see what happens.

Thanks for that.  The "delay" question has been hard because useful 
statistics on connection behavior are rare or confidential and we never 
know what future scenarios are going to be anyway.  So it's been hard to 
say anything useful.  But the IETF is coming, so in the meantime here 
are a few thoughts:

   - Most traffic is within a region.

   - EIDs will still be allocated by RIRs (or the equivalent) out of
     assigned chunks, so there will be a great deal of contiguousness for
     EIDs within a region.

   - ALT routers for a particular EID prefix will be fate-shared,
     situated in the region where it is predominantly found.

   - Data paths follow a topological hierarchy and the delay issue is one
     of relative differences.

So one can conjecture that for a particular session, *if* there is delay 
(not yet granted), the likely case is that the session will see far less 
than the alleged "very long paths".  There will be some outliers due to 
small chunks of EID prefix moving to a completely different region -- I 
don't think there will be many of those.  There will obviously be 
sessions between regions.  For those, at a granularity of a domain, the 
ALT control path may in fact be the same as, or at least not differ 
much, from the data path.  The relative difference in round-trip time 
will be small.

That's if there is a delay.  The only time the question arises is the 
first time a site attempts to reach a particular EID prefix, or if it is 
contacting so many prefixes that it has to do some cache management.  We 
can predict connection/cache behavior for very large and very small 
sites.  If anyone has pointers to data on connection behavior over time 
for intermediate-size sites, please say so.

This is all extrapolation from the current reality, but at least it 
tries to be based in how networks work.  We should try to get the delay 
discussion on a solid foundation of fact if possible.  Others will be 
better at that than I.

Scott

---

From: Christian Vogt <[email-masked]>
Date: Tue, 18 Mar 2008 19:00:49 +0200

Bill -

Sorry for responding late...

> third-party address authorities.

You are raising an interesting point.  3rd-party authorities are
typically used for authentication.  But apart from authentication,
there needs to be a trust relationship between two edge networks based
on which one of the two can update a mapping on behalf of the other.
Having such trust be transitive through the 3rd party would be an
option, but maybe not a desirable one...

- Christian