Subject: Re: where the indirection layer belongs

From: Robert Honore <[email-masked]>
Date: Fri, 29 Aug 2003 11:02:17 -0400

Dear Keith Moore,

I do not want to seem to be ganging up on you, but your reply to Mr. 
Hain just prompts more questions.

> 
> You are missing something fundamental here - if a TCP connection breaks
> (isn't closed cleanly) then the two endpoints can get out of sync
> regarding how much data was delivered.  You can't fix this at higher
> layers without an unacceptable amount of complexity and overhead.
> This has nothing to do with the app/transport interface being a sacred
> invariant - it happens any time you try to layer something on top of
> transport that has to survive breakage of the transport layer.
> 
> (how many ways do I have to explain this?)
> 
You might have to explain this really often and in simple language too. 
  But why do you assert that it will take lots of complexity and 
overhead?  Can you point to some code where they tried this?  As far as 
I know, nobody has really given this an earnest try as yet.  At least 
not with any IP protocols.

> It's a lot less glue than the L4-L7 approach, and most of it has to deal
> with authentication that would be needed for any kind of
> remotely-originated routing update anyway, regardless of what layer it
> lived at.

What specific glue do you believe it requires for the L4 to L7 approach? 
  How does that compare to what is needed in an L3 solution?

> Yes you can do that but it presumes that the host knows a priori whether
> or not it needs the stabilization layer.  I would make the
> mechanism used to provide stable addresses a property of the network-
> either the network provides "reasonably" stable addresses (i.e. plenty
> of prior notice before changing them) or it provides a stabilization
> layer.  That way, networks that don't need it don't pay the overhead.

But I would argue that the host or at least the application's designer 
knows whether it will need the stabilisation layer.  Making the 
mechanism that provides the stable network addresses a property of the 
network leaves the question of how.  Even if that were achieved though, 
that still does not completely or effectively address the problem of one 
application process identifying its peer across the network.

Yours sincerely,
Robert Honore.

---

From: Robert Honore <[email-masked]>
Date: Fri, 29 Aug 2003 10:18:17 -0400

Dear Keith Moore,

Thank you for your reply.  It seems that we are without a forum though, 
since what we are discussing is, according to Tony Hain, not in line 
with the IPv6 working group charter.  Maybe we really do need a new 
working group for this issue.  Should we propose the formation of one?

Keith Moore wrote:
> These "capabilities" should be regarded as bugs which are being fixed.
> 
> In particular, the fact that IPv6 hosts can, in ordinary circumstances,
> have multiple addresses has led people to believe that it's reasonable
> to expect IPv6 apps to deal with an arbitrary number of addresses per
> host, some of which work to send traffic to the destination and some of
> which don't, and to have this behavior vary from one source location to
> another.  First, nobody has ever explained how these hosts can reliably
> determine which addresses will work.  Neither source address
> selection nor multi-faced DNS are satisfactory answers.  Second, this
> robs apps of the best endpoint identifier they have.
> 

I agree that the "bug" in this picture here is that nodes can have 
multiple addresses, some of which work and some of which don't work in 
different circumstances.  It really should be that an address which is 
advertised for a node should be valid for communication with that node 
under all circumstances.

I disagree with you if you are implying that the node should not be 
allowed to have multiple addresses on an interface, and I never agreed 
with the notion that the IP adress should double as a node's identifier 
or as the identifier of anything other than the interface with which it 
is associated.  If the IP address as the identifier of the endpoint is 
the reality with which we must live, then I can live with that, but 
don't ask me to consider that the better circumstance.

> 
> No, it's not worthwhile.  Any kind of routing needs to happen below the
> transport layer rather than above it.  That's not to say that you can't
> make something work above the transport layer, but that to do so you
> have to re-implement routing, acknowledgements, buffering and
> retransmissions, duplicate suppression, and windowing in this new layer
> when transport protocols already provide it.
>

I never said anything about forcing the application to talk directly to 
the network layer as you seem to imply by your second sentence.  It 
might have come out wrong, but what I was trying to say was that for 
applications that need it, what is wrong with there being a standard 
(choose your favourite name) adaptation or stabilisation service or 
interface or protocol between the application and the transport layer 
(like Tony Hain suggests)?

Besides, even if you mean that the presence of such an stabilisation 
layer (to use Mr. Hain's nomenclature), would require the implementation 
of routing, acknowledgements, buffering etc., you are not necessarily 
routing, acknowledging, retransmitting etc., the same data that the 
transport is, are you.  You might be doing that for higher level objects 
where the transmission of any one might have required the establishment, 
usage and teardown of one or more transport connections.  Do you find I 
am engaging in excessive sophistry if I were to argue that if that is 
what the application needs then that application should be able to 
implement it?

> 
> Good question.  My best answer so far is: stable enough so that the
> vast majority of applications don't have to implement additional logic
> to allow them to survive broken TCP/SCTP/etc. connections, or (to put it
> another way) stable enough so that failures due to address/prefix
> changes are not a significant source of failure for most applications
> (as compared to, say, uncorrectable network failures and host failures).

Am I to presume that you are not including in your categorisation of 
uncorrectable network failures and host failures, the possibility of one 
"home" of a multihomed host going down, while still leaving the host 
reachable by one of its other "homes"?  As far as I can tell, none of 
that is well addressed as yet.

> 
> IMHO, apps should be able to assume that an advertised address-host
> binding is valid for a minimum of a week.  This is a minimum - it
> should be longer whenever possible.  (however there's no requirement to
> maintain addresses longer than the nets will be accessible anyway -
> i.e., you don't expect the addresses for the ietf conference net to
> remain valid after the net is unplugged...but they shouldn't be reused
> within a week either.)
> 

I have no real objection to the "address-host binding" being valid for a 
minimum of a week or any duration greater than two round-trip times 
node-to-node.  But an address-to host binding isn't really one, is it? 
By that I mean that the real and effective binding is to an interface. 
If I may quote RFC 1883, it defines an address as follows.

"   address     - an IPv6-layer identifier for an interface or a set of
                  interfaces."

While we have regarded the node and its interface as one and the same, 
it really isn't, even though we can get away with that most of the 
times.  My point is that we haven't even considered the other half of 
the problem as far as the application is concerned.  We also need a 
higher level object which we can associate with the node itself or the 
application endpoint and with which we can associate an identifier. 
Such an association will need to remain valid at least for the duration 
of the application's run-time, I believe.

> 
>>And the second question is what should be the context of that
>>identifier's validity?
> 
> 
> The identifier should be unique within the entire Internet.  In some
> cases (e.g. small networks without connectivity to the Internet core)
> "very likely to be unique within the Internet" would probably
> suffice.  That doesn't mean that you can send to any point from any
> other point, because there will still be access controls.  But if an
> address is advertised for a host, and you have permission to send to
> that host, you should be able to use that address to send to that host.
>

If you accept the statement I made above, the minimum context of that 
association's validity should be the collection of nodes that are 
currently participating in that application's protocol.  I then have no 
fundamental disagreement with your assertion that such an identifier 
should be valid within the entire Internet.

> 
> The first one I explained above in slightly more detail.  The second one
> should be obvious.  If an address becomes invalid because of a topology
> change somewhere distant in the network, how is a layer above layer 4
> going to know about it?  It doesn't have access to routing protocol
> updates - those happen at layer 3 and aren't normally propagated to
> hosts anyway.  When you think about it, you probably don't want hosts to
> have to listen for information about distant state changes in the
> network - that would place a tremendous burden on small hosts and nets
> with low-bandwidth connections.

The second one becomes obvious only if you restrict the definition of 
endpoint to the interface.  If instead the definition of the endpoint is 
taken to be the process or object that is supposed to be the final 
recipient of the data that is forwarded by the IP and the transport 
layers, then it is no longer so obvious.  Sometimes I wonder if we 
aren't treating together concerns or problems that we should be treating 
separately.

I don't want *application processes* to be needing to listen for 
information about distant state changes in the network, but I do want my 
application processess to be able to adapt or recover when something 
happens that affects its communication with a peer *process* while 
neither of the cooperating processes has indicated any intention to stop 
participating in that communication.

Yours sincerely,
Robert Honore.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Thu, 28 Aug 2003 15:10:00 -0500

I wasn't "there" at the time, but if

----- Original Message ----- 
From: "Robert Honore" <[email-masked]>
To: "Tony Hain" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Tuesday, August 26, 2003 12:40 PM
Subject: Re: Solving the right problems ...

>  Problem is that right now, every time
> an application needs such a service, the application programmer must
> code his own provisions for it, and none of the propositions I have
seen
> discussed so far seems to deal with that.

couldn't have been a quote about congestion control in 1980, I'm
surprised.
And now, of course, we accept that transport layers are responsible
for
congestion control.

This makes me think that having something underneath the application
take
care of multiple interfaces is likely a good idea.

I'd also go farther on Robert's point - if we don't think of something
fairly
quickly, there are likely to be a good number of applications that
expect to
handle multiple intefaces (with varying degrees of skill) lying around
when
we think of something later.

And this seems to me, to be very analogous to firewalls (we "needed"
them
because we didn't have end-to-end security, but once we had them, we
didn't need end-to-end security nearly as badly), NATs (we "needed"
them because we didn't have IPv6, but once we had them, we didn't
need IPv6 nearly as badly), and other sources of architectural
drift...

Spencer

---

From: "vinton g. cerf" <[email-masked]>
Date: Fri, 29 Aug 2003 07:44:21 -0400

Keith,

as one thinks about VPNs, tunneling, mobility and the like, assuming a week is probably a bad idea. 
Think a bit about the way AOL's IM tracks the binding of an IM handle to an IP address - it is quite dynamic.

Vint

>remain valid after the net is unplugged...but they shouldn't be reused

Vint Cerf
SVP Technology Strategy
MCI
22001 Loudoun County Parkway, F2-4115
Ashburn, VA 20147
703 886 1690 (v806 1690)
703 886 0047 fax
[email-masked]
www.mci.com/cerfsup

---

From: Melinda Shore <[email-masked]>
Date: Tue, 2 Sep 2003 12:02:40 -0400

On Thursday, August 28, 2003, at 04:10 PM, Spencer Dawkins wrote:
> take care of multiple interfaces is likely a good idea.

We already do have something in that general direction, rserpool,
which I think is something that's generally unknown and then often
misunderstood by those who've heard of it.  It doesn't, however,
provide any kind of support for rendezvous or something similar for
dealing with translated or otherwise gateway-needful addresses.

Melinda

---

From: Keith Moore <[email-masked]>
Date: Tue, 2 Sep 2003 17:35:55 -0400

> Maybe I read your paper on project SNIPE too quickly, but it was not 
> immediately clear that the problems you mentioned  were a specific 
> result of an attempt to make the application resilient against
> (sudden) changes in IP address. 

that wasn't quite the purpose of SNIPE.  SNIPE didn't really anticipate
that the network would change IP addresses out from under it, but rather
that processes (and thus their IP addresses) would need to be accessible
from multiple addresses (in order to be able to use the most efficient
network medium available), and to be able to migrate from one address to
another (in order that the system as a whole could continue to operate
for long periods of time without interruption).

still, the mechanisms we built for SNIPE (and then discarded) were very
similar to what people are now proposing as a way to cope with changes
in IP address, and our experience with SNIPE seems like a good
indicator of how well those mechanisms could be expected to work.

---

From: Robert Honore <[email-masked]>
Date: Tue, 02 Sep 2003 17:02:06 -0400

Dear Keith Moore,

Maybe I read your paper on project SNIPE too quickly, but it was not 
immediately clear that the problems you mentioned  were a specific 
result of an attempt to make the application resilient against (sudden) 
changes in IP address.  More specifically, it was not clear from that 
report that the additional complexity did come from the attempt to 
provide the kind of resilience we are seeking, or from the rather 
ambitious goals of project SNIPE.  I will re-read more slowly and 
carefully this time.

Yours sincerely,
Robert Honore.

Keith Moore wrote:
> (regarding the complexity of putting a general-purpose layer to survive
> address changes between L4 and L7)
> 
> 
>>overhead?  Can you point to some code where they tried this?  As far
>>as I know, nobody has really given this an earnest try as yet.  At
>>least not with any IP protocols.
> 
> 
> I tried this in connection with a project called SNIPE that I worked on
> several years ago.  SNIPE was an attempt to build a very
> reliable distributed computing environment that supported, among other
> things, the ability to access a computing resource via multiple
> addresses (mostly in order to exploit high-bandwidth local networks
> not necessarily using IP), and the ability of both hosts and processes
> to migrate to other addresses.  It used a DNS-like service similar to
> RESCAP (for those who remember that) to register the addresses at which
> a process was accessible, and it attempted to provide TCP-like streams
> on top of TCP and this registry that would survive changes in those
> addresses.  Basically I found that you can get such a service to work
> reasonably well and reasonably efficiently if endpoints don't move
> without adequate notice.  OTOH if hosts or processes do move without
> adequate notice then you end up needing to implement the mechanisms I
> mentioned earlier, and that involves extra copies and extra overhead. 
> 
> The reason I am preposing that the two problems (changing addresses
> with adequate notice and changing addresses without adequate notice) be
> treated separately is that by trying to make a single mechanism serve
> both purposes you end up with a lot of inefficiency and/or cost that
> aren't needed in most cases.  And that's true (for different reasons)
> regardless of whether you insert that single layer between L3 and L4 or
> between L4 and L7.
> 
> 
> Thought I'd said this already: buffering of data until acknowledged by
> the peer, generation and processing of acknowledgements, retransmission
> of lost messages due to broken connections, windowing (so you don't
> degrade to stop-and-wait), duplicate suppression.  You also need to
> multiplex control and data information over the same stream and to probe
> for other addresses when you get a failure on the one you were using.
> 
> 
> If you work on the problem at (or just above) L3, transport protocols
> already have the code needed to recover from lost messages, so you don't
> have to reimplement that.  You basically need a mechanism by which the
> layer can realize it needs to start routing packets differently, and do
> so.  You probably need multiple ways that the layer can get that
> information because the remote address can change for a variety of
> reasons and in lots of different places in the network.   (That's
> equally true for the L4-L7 layer as for the L3-L4 layer, but the L4-L7
> layer isn't in a position to get some of that information.  The L3-L4
> layer can potentially recover from address changes more quickly but to
> do that safely it has to be able to authenticate and establish a basis
> for trust in a wider variety of information sources.)
> 
> 
>>>Yes you can do that but it presumes that the host knows a priori
>>>whether or not it needs the stabilization layer.  I would make the
>>>mechanism used to provide stable addresses a property of the
>>>network- either the network provides "reasonably" stable addresses
>>>(i.e. plenty of prior notice before changing them) or it provides a
>>>stabilization layer.  That way, networks that don't need it don't
>>>pay the overhead.
>>
>>knows whether it will need the stabilisation layer. 
> 
> 
> It can't know that reliably unless the network without the stabilization
> layer has well-defined properties - e.g. the network won't change
> addresses of a network without advertising those changes with a minimum
> amount of advance notice.  If addresses can potentially change at
> arbitrary times with no assurance of stability then every app needs the
> stabilization layer (or provide its own means of recovery).
> 
> 
>>Making the 
>>network leaves the question of how.  Even if that were achieved
>>though, that still does not completely or effectively address the
>>problem of one application process identifying its peer across the
>>network.
> 
> 
> Well, there's no way I can supply all of the details in a few email
> messages.  I've been trying to find time to write them up.  In the
> meantime I want to discourage too much momentum behind trying to solve
> all of the address changing problems in any one place - particularly
> between L4 and L7, because I already know that that won't work well.
> 
> Keith
> 
> 
>

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Fri, 12 Sep 2003 10:48:18 -0500

Ok,

A very very late response to this thread... or at least part of
this thread, for some reason a while back I was knocked
off of the ietf list and thus only get things cc'd to the ipng list :-0
(and yes, I could re-subscribe but... I get so much email its hard enough
  keeping up... sigh..)

Now, I am going to comment on all of the bits of this thread
that I have been able to sort through (that went dually to ipng)...

Tony Hain wrote:

>In the ongoing saga about topology reality vs. application perception of
>stability, it occurs to me we are not working on the right problem. In short
>we have established a sacred invariant in the application / transport
>interface, and the demands on either side of that interface are the root of
>conflict.
>
>Mobile IP, and the multi6 DHT work are attempts to mitigate it through
>slight of hand at the IP layer, while SCTP attempts to mask the topology
>reality in the transport layer. (These are probably not the only examples,
>but they are the ones that come to mind this morning.) Yet none of these
>really do the job in all cases.
>

SCTP seems to me to do most of what you propose in this "layer"... assuming
an SCTP that includes the ADD-IP extension I can (and I have done this
with the KAME stack)

1) setup a SCTP association (connection in TCP terms) between two 
endpoints lets say
    10.1.2.1 port 2222 <----------------> 10.1.3.5 port 3333

2) on 10.1.3.5 move the cable to the 10.1.4.5 network

3) on the machine type "ifconfig fxp0 10.1.4.5 netmask 0xffffff00 ; 
route add default 10.1.4.4<cr>"

4) And the association stays up .. the application is un-aware that the 
topology under the machine
    changed.

Now of course if the application wants to be aware, it can "subscribe to 
events" that let it know
that it happened. This means it DOES hide the events letting the app 
keep a stable view of
the endpoint its talking to and YET at the same time allows some apps 
(ones that need or want
to know) find out. This seems to me to be EXACTLY what Tony is trying to 
get to... so why
would we re-do what we have already done?

We could do has Keith has proposed.. put it at a layer BELOW the 
transport. Now, what
do we end up with?

1) A NAT like thing that is playing games with the IP addresses and 
checksums (aka the psuedo-header)
<or>
2) Putting a psuedo IP address (the identifier) on the machine for the 
app to use apart and
    seperate from the real physical IP address in use.

3) ??others that I can't think of??

If we choose <1> it seems ugly to me.. I don't like NATs and now we just
put one IN your host... what else will break?

If we choose <2> and arguably you can do this without a new layer.. 
There are several
papers on how and I know I did such a thing long ago at NYNEX for their 
voice-dialing
system... you end up with some other problems that are also in <1>... 
which all boil down
to the fact that the transport has knowledge of failures your upper or 
lower layer will
NOT have (hey I just retransmitted to IP-A .. need its the Nth time
I have done so). This is key to detection that there is a 
problem....this information will
just not be present in either approach.

Now long ago, Qiaobing Xie and I started down this path of trying to get 
fault tolerant
applications. SCTP was a direct result of that.. we put the path 
redundancy in the
transport where you could use the information it holds to better make 
fail-over choices on
paths. But there is another aspect to this as well.. the other half of 
the equation if
you will... that is I have

APP-1   <-------------------------------------> Peer-Of-App-1

Note I say Peer.. it may be a server .. it may NOT and you might be 
doing some
peer-2-peer networking...

Lets say that Peer-Of-App-1 is redundant and I can talk to any of the 
peers. And
more so each side of this (APP-1 and the peers) are multi-homed. We 
(Qiaobing
and I) divided this problem in to two pieces..

1) The path level multi-homed redundancy portion... i.e. what became 
SCTP.. lets keep the
     transport up as long as there is connectivity between the two no 
matter which
    IP addresses come/go/up/down what have you.

2) The app not caring WHICH Peer its talking to.. as long as it gets 
done what
    it needs to. This other redundancy we put in what is NOW the RSERPOOL
    working group. The app gets an identifier for Peer-Of-App-1.. and if 
there
    is a failure it can silently move to an alternate peer.. and of 
course we envisioned
    the connection between the two being SCTP.. so all that path-level 
redundancy was
    handled at transport...

I just don't see the need for this to tell the honest truth..

A) We have SCTP which Tony has not indicated (at least as far as I can 
see in this
      thread) how his new layer does anything different then what is 
already there.

B) We have an ongoing effort in rserpool to finish out the upper level 
redundancy (were almost
      done I think.. or close :>)

So why do we need to re-invent a solution???

This seems to me like the introduction of yet another NAT like 
mechanism... this is percisely
why IPv6 is where its at today... call your local ISP, ask them about 
IPv6... what will they answer you?

I have two ISP's and have talked to MANY people in both.. and you get either

a) huh?
<or... less often>
b) Oh, yeah, I did hear something about that.. but hey its just 
something I need
    to read about someday in the future... (future being around the time 
I retire).

Adding this layer just makes NO sense to me...

R

>
>We all agree that applications should not be aware of topology. At the same
>time,  application developers insist on the right to pass around incomplete
>topology information. There are arguments that the IP address is overloaded
>as an endpoint name. I don't particularly buy that, because the real
>endpoint is either a protocol, or a specific port of a protocol. From one
>perspective, when applications use addresses in referrals, they are
>specifying that routing through a particular point in the topology will
>reach the desired endpoint named 'transport/port-id'. But that is a
>semantics and perception discussion which doesn't help reach the goal.
>
>In any case, what applications need is a stable reference to other
>participants in the app. Yet we know that current and projected reality says
>that topology is not stable, or consistently reachable. This could be due to
>technology issues of the underlying links, or simple policy. Either way, the
>network as seen by the transport layer is not, and will never be stable or
>consistently reachable from everywhere. Given that, the direct interface
>with the transport layer creates a failure mode for an application expecting
>stability. 
>
>Where this leads us is to the sacred invariant, and the need to solve the
>problem in the right place. I suggest it is time for the IETF to realize
>that the protocols are no longer being used in the network of 1985, so it is
>time to insert a formal stabilization layer between application & transport.
>
>
>Such a layer would be responsible for managing intermittent connectivity
>states and varying attachment points of the transport layers below.
>Applications that interact with this layer would be insulated from the
>inconsistencies being experienced by the transport layer. It would also be
>reasonable for this layer to manage multiple simultaneous transport
>interactions so that the application perceived a single data path to its
>peer. With appropriate trust between the stack and the network policy, this
>would even simplify the process of QoS markings for parallel related data
>sets. 
>
>The next place that leads us is to a name space. At this point I see no
>reason to avoid using the FQDN structure, but acknowledge that the DNS as
>currently deployed is not even close to being up to the task. The protocols
>are not so much the issue as the deployment and operation model that is
>focused on a limited number of nodes operated by a small community of
>guru's, where the expectation is that any changes occur on the order of
>several days or longer. What we ultimately need in a name service to support
>the suggested layer is the capability for every consumer device with
>electrons in it to automatically and dynamically register its current
>attachment information with rapid (that doesn't mean sub-second, more along
>the lines of a cell phone that powers up away from its home) global
>convergence. As there are multiple trust boundary issues involved (because
>not every device will be subscribed to a service), making this scale will
>require pushing the database distribution out to smaller pockets. Making it
>reliable for Joe-sixpack will probably require that part of the
>infrastructure exists on his side of the interconnect & trust boundary from
>any other networks. Automating the attachment of a massive number of small
>dataset servers will probably require something with better scaling
>characteristics than the current DNSsec deployment model.
>
>Since many networks have usage policies established around the sacred
>invariant, there will need to be some recommendations on how to apply those
>policies to this new layer. We could even consider a protocol between this
>layer and a policy entity that would aggregate applications into a policy
>consistent with the what the network can deliver for various transport
>protocols. This distribution of policy awareness would have much better
>scaling characteristics than per app signaling, as well as the ability to
>locally group unrelated transports that any given app might be using.
>
>Bottom line is that we are spending lots of time and effort trying to force
>fit solutions into spaces where they don't work well and end up creating
>other problems. We are doing this simply to maintain the perception of
>stability up to the application, over a sacred interface to a very unstable
>network. Stability to the application is required, but forcing applications
>to have direct interaction with the transport layer is not. Yes we should
>continue to allow that direct interaction, but we should be clear that there
>are no guarantees of stability on that interface. If the application does
>not want to take care of connections coming and going to the same device at
>potentially different places, it needs to have an intermediate stabilization
>layer available.
>
>Tony
>
>
>--------------------------------------------------------------------
>IETF IPng Working Group Mailing List
>IPng Home Page:                      http://playground.sun.com/ipng
>FTP archive:                      ftp://playground.sun.com/pub/ipng
>Direct all administrative requests to [email-masked]
>--------------------------------------------------------------------
>
>
>  
>

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Keith Moore <[email-masked]>
Date: Fri, 12 Sep 2003 16:57:39 -0400

> >stability, it occurs to me we are not working on the right problem. In
> >short we have established a sacred invariant in the application / transport

[...]

Yes, SCTP does *most* of this.  It can handle the case where one party of a
connection moves, and it is able to tell its peer that it has moved.  IIRC, it
doesn't handle concurrent moving of both connection endpoints (even if both
endpoints are in the same network), nor does it handle cases where timely
notice cannot be provided.  Nor does it handle rendezvous or referral because
in those cases there's not a constantly-open connection over which to signal
locator changes.  So I don't view the SCTP approach as a complete solution,
but it might be useful as part of a solution.

Also, I'm not insisting on a separate layer.  As far as I can tell, some of
the changes needed could take the form of backward-compatible modifications to
TCP and UDP and perhaps SCTP also, and I certainly think it's worth
considering those techniques.  Whether this is best handled by changes to
layer 4 (and perhaps layer 3 also if we need support from the routers), or a
new layer between 3 and 4, is not something I claim to have analyzed in
detail.

(OTOH, I am insisting that it's not reasonable to try to solve the entire
problem above layer 4.  For different reasons - mostly economic - I suspect
it's not reasonable to try to solve the entire problem below layer 4 either,
but I'm less sure of this.)

> transport. Now, what do we end up with?
> 

I don't think the two are inherently that different.  Say you define a new
identifier that can be associated with a connection endpoint, and you build a
mechanism that allows a host to map that identifier to one or more locators. 
Now you can tweak TCP to allow it to define either or both of its connection
endpoints (what is now the pseudo-header) in terms of those identifiers rather
than in terms of the locators currently in use, and you tweak the demux layer
on hosts that support this extension so that it can accept multiple
source,destination locator pairs and map each of them into the same TCP
channel that is defined in terms of endpoint identifiers.  Then you define
ways of doing signalling via TCP options so that the hosts can learn about new
locators and deprecated locators for their peers.  I see this as a
variant of your #2, but it also has a vague resemblance to NAT.  

However, I don't think this has any of the problems of NAT.  Essentially what
it would accomplish is to cleanly separate connection endpoint identifiers
from locators, which people have been saying was a good idea for a long time. 
And I think it could be done in such a way that it is fairly compatible with
existing apps that use TCP. 

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 12 Sep 2003 17:44:07 -0400

> >identifier that can be associated with a connection endpoint, and you build
> >a mechanism that allows a host to map that identifier to one or more
> >locators. Now you can tweak TCP to allow it to define either or both of its
> >connection endpoints (what is now the pseudo-header) in terms of those
> >identifiers rather than in terms of the locators currently in use, and you
> >tweak the demux layer on hosts that support this extension so that it can
> >accept multiple source,destination locator pairs and map each of them into
> >the same TCP channel that is defined in terms of endpoint identifiers. 
> >Then you define ways of doing signalling via TCP options so that the hosts
> >can learn about new locators and deprecated locators for their peers.  I
> >see this as a variant of your #2, but it also has a vague resemblance to
> >NAT.  
> >
> mucking the IP header is bad.. if we could reach some solution that is NOT
> mucking the IP header then it might be ok... 

In my mind the IP header still contains the addresses of the locators in use,
and layer 3 (at least that part of it) works entirely the same as it does now.
 But the TCP pseudo-header might contain different identifers than those
addresses, and as far as TCP is concerned the (source address, source port,
destination address, destination port) would no longer necessairly be the
connection identifier - there would be a mapping function between those and
the real connection identifier, which would be defined in terms of endpoint
identifiers and ports.

> ok.. but I think you are really talking about a "new" TCP...

If it turns out to be feasible, there's a lot that can be leveraged by making
mods to TCP rather than moving to SCTP or some other protocol.  In particular,
you can ease transition (one end could be running new TCP and the other old
TCP and they could still talk as long as neither end moved)  and minimize the
mods to existing apps.  And it may be easier to get host stacks to implement a
TCP exstension than to implement a new protocol, even assuming similar amounts
of code for each.

and of course neither SCTP or TCP would be sufficient by itself.  we still
need a suitable identifier,  a way to map those into locators, and a way to
maintain those mappings.

I'm still undecided about whether it is better to modify existing transports
or to do a mobile-IP like approach.  The latter has tunneling overhead but
works for all transports and in some sense the changes are simpler.

---

From: "vinton g. cerf" <[email-masked]>
Date: Sat, 13 Sep 2003 09:56:15 -0400

I am a strong proponent of trying to find a way to create a new set of end identifiers that would be insensitive to the changing of IP level addresses. It seems to me that we would find ourselves working pretty hard to tease apart the current strong binding of IP and TCP (pseudoheaders etc) but it may be well worth the effort. For one thing, it might lead to the ability to carry TCP segments over multiple Source/Destination pairs between the same hosts (labeled by a single end point identifier each) in addition to allowing for rebinding of endpoint identifier and IP address. The rendezvous and signalling problem of concurrent motion is not unlike the challenge of TCP's simultaneous-INIT - you have to get the fixed point right to make it all work. We have other fixed points in the Internet, notably the root hint file, so perhaps it is not unreasonable to consider another fixed point concept to facilitate simultaneous rebinding of IP and endpoint identifiers. I suspect this ge!
 ts pre
tty messy when you start to think about multicast but that's territory that also needs exploring. We would also want to look very carefully at the potential spoofing opportunity that rebinding would likely introduce. 

Vint

Vint Cerf
SVP Technology Strategy
MCI

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Fri, 12 Sep 2003 16:25:52 -0500

Keith Moore wrote:

>notice cannot be provided. 
>
Yep.. and neither would a layer put under or over the transport. And I am
not sure about the timely issue. In the case of moving you only have a
1 RTT time lag... that may be considered not timely enough.. I guess.

Also to handle con-current moving you need some rendevous point... aka
a mobile-ip home agent or its equivalant.. this same issue would exist
for any layer (upper or lower)...

> Nor does it handle rendezvous or referral because

Yep.. it solves one part of the problem.. can we find ways to solve
the other parts.. I think yes..  I know Michael Tuexen has some
ideas on the referral side.. he voiced them at the last SCTP bakeoff.. if
referral means sort of a proxy setup where some other host will be
handling the actual connection.. Michael was kind of vague.. and we
did not thrash out any details.. but I think he will be putting forward
a draft somewhere down the road ... the old idea->napkin->draft  :->

>new layer between 3 and 4, is not something I claim to have analyzed in

ok.. I can agree with that.. there is ALWAYS room for improvement :>

>but I'm less sure of this.)

I think we are in FULL agreement here.. I don't see it working well
above layer 4. This is why we (Qiaobing and I) only handled failover
when a server broke above layer 4 in our model.. linkish related things
new IP address.. multi-homing et.al is best done at or below 4...

mucking the IP header is bad.. if we could reach some solution that is NOT
mucking the IP header then it might be ok... I am just not convinced 
yet... having
the transport aware seems to me a cleaner solution. .but of course you knew
I would say that :->

>existing apps that use TCP. 

ok.. but I think you are really talking about a "new" TCP... by the time
you tweak this and massage that you have re-written/modified/... (fill in
your choice word) TCP into something it was not before... as long as
we are at it lets fix the head-of-line blocking problem too.. after all
as Kacheong would say.. its just code... of course this then just becomes
TCP picking up all of the ideas of SCTP and becoming it :-0

I think originally sigtran would have wanted to modify TCP if they would
have been allowed.. instead of rolling a new transport.. now that we have
the new transport.. how much do we back-port ?

R

---

From: Dave Crocker <[email-masked]>
Date: Sat, 13 Sep 2003 07:30:23 -0700

Randall,

RRSh> Now of course if the application wants to be aware, it can "subscribe to
RRSh> events" that let it know that it happened.

That sounds remarkably like a "presence" service.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Sat, 13 Sep 2003 14:46:05 -0500

vinton g. cerf wrote:

>I am a strong proponent of trying to find a way to create a new set of end identifiers that would be insensitive to the changing of IP level addresses. It seems to me that we would find ourselves working pretty hard to tease apart the current strong binding of IP and TCP (pseudoheaders etc) but it may be well worth the effort.
>

We did this in SCTP.. we did NOT place a pseudoheader in specifically so 
that this would not be
a problem.. especially since SCTP had to deal with multiple addresses..

> For one thing, it might lead to the ability to carry TCP segments over multiple Source/Destination pairs between the same hosts (labeled by a single end point identifier each) in addition to allowing for rebinding of endpoint identifier and IP address.
>
Sounds just like what we have in SCTP as well .. at least with the SCTP 
ADD-IP extension :>

> The rendezvous and signalling problem of concurrent motion is not unlike the challenge of TCP's simultaneous-INIT - you have to get the fixed point right to make it all work. 
>
We actually made the simultaneious-INIT in SCTP without a fixed point.. 
it was a
bit tricky but it does work :> However if you want to do dual ended IP
address changing you DO need some fixed point somewhere. Someone once
made a proposal to me on how it could be done with just a temporary
state repository.. at least for the both moving and already established 
case.

For the moving and NOT yet established this gets harder.. the fixed point
needs to be something like a mobile IP home agent.. with a lot
less functionality.. i.e. tell me where X is... and you get back a
current location... more of a DB dip then anything...

>We have other fixed points in the Internet, notably the root hint file, so perhaps it is not unreasonable to consider another fixed point concept to facilitate simultaneous rebinding of IP and endpoint identifiers. I suspect this gets pre
>tty messy when you start to think about multicast but that's territory that also needs exploring. 
>

Yuck.. I that one makes my head hurt a bit.. I am sure it can be done but
we would need to think about it very hard :-0

>We would also want to look very carefully at the potential spoofing opportunity that rebinding would likely introduce. 
>

This is one of the reasons the authors of ADD-IP have NOT pushed to get 
it done.. some more
work needs to be done on this area...

R

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Sat, 13 Sep 2003 14:48:09 -0500

>That sounds remarkably like a "presence" service.
No, what it is is a socket call you make that "subscribes" do address
events on the socket. Thus when a peer adds and deletes an
address you get a "notification" message up the socket buffer
telling you that an address was added and one was deleted.

Take a look at the current SCTP sockets document... or for
that matter you can play with the KAME SCTP stack and
write some code, bring up an association and then do some
ifconfig's ...

R

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Sat, 13 Sep 2003 14:48:19 -0500

R

---

From: Pekka Nikander <[email-masked]>
Date: Mon, 15 Sep 2003 20:45:02 +0300

vinton g. cerf wrote:
>> We would also want to look very carefully at the potential spoofing 
>> opportunity that rebinding would likely introduce.

> work needs to be done on this area...

http://www.ietf.org/internet-drafts/draft-nikander-mobileip-v6-ro-sec-01.txt
is a background document, produced by the MIPv6 route optimization
security design team, that tries to explain the security desing
in MIPv6 RO.  I think that most of the threats and much of the solution
model would most probably apply also to SCTP ADD-IP and, of course,
also other multi-address multi-homing solutions.

--Pekka Nikander

---

From: Dave Crocker <[email-masked]>
Date: Mon, 15 Sep 2003 10:37:27 -0700

Randall,

>>That sounds remarkably like a "presence" service.
RRSh> No, what it is is a socket call you make that "subscribes" do address
RRSh> events on the socket.

oh.  ok.  I thought you were referring to a protocol issue, not a local
programming issue.

I suppose it is interesting that the language you used in your previous
statement could have either meaning.  Not sure _how_ it is interesting,
though...

d/

---

From: Keith Moore <[email-masked]>
Date: Mon, 15 Sep 2003 10:42:24 -0400

> > > Now of course if the application wants to be aware, it can
> > > "subscribe to events" that let it know that it happened.
> >

Both are needed.  That is, you need a way for the host IP stack to know
that a peer has moved to a different locator, and you need timely
notification that this has happened.  This is a lot like a presence
service, but it needs to operate at layer 3, and it needs to act in
response to sending IP packets to the old locators (polling a presence
server is either too slow or - if you do it often enough - too
wasteful).

And yes, you probably also need a socket call that the app can use to
ask that it be notified of changes in peers' locations, because
location changes can also result in changes in the level of service
available to the app. though hopefully most apps won't need to use it.

Keith

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Mon, 15 Sep 2003 13:54:05 -0500

Pekka Nikander wrote:

>> This is one of the reasons the authors of ADD-IP have NOT pushed to 
>> get it done.. some more
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
Pekka:

Interesting reading.. I had heard of the RR check from someone but had
not researched it in detail.. nice document :>

Now as to the applicability in SCTP and ADD-IP...

There is a difference with mobile-ip in that an SCTP association is already
established. Each node CN and MN have "connection" state. There has
been a 64bit random value exchanged and the "ADD-IP" which is equivialant
of the "BU" can be verified with this random state that the ends are
maintaining. The real issue shows up in that if you are worried about
an ease-dropper that can "see" the initial INIT/INIT-ACK exchange
between the two peers. In that case it would then have the 64bits of 
randomness
and could "inject" the false ADD/DEL that would hi-jack the association. Of
course it could do other things too like knock down your assocation as well
by sending a false ABORT chunk....

It is good to see that the routing infrastructure is believed to be 
non-compromised
in MIP case. If we can make the same assumption then with one minor
tweak we can add a mechanism to SCTP to authenticate the ADD-IP with
private-public key pairs shared in the INIT/INIT-ACK. The obvious
problem with this would be if the infrastructure was compromised and you
had a true man in the middle who could intercept the INIT/INIT-ACK 
packets and
change the keys... but that goes away if we make the same assumption MIP 
did :>

Michael Tuexen and I were working on a seperate draft for this.. and 
were also
somewhat concerned about the compromised routing structure too. If we don't
have to worry about that our job just got easier :>

Thanks

R

---

From: Dave Crocker <[email-masked]>
Date: Sat, 13 Sep 2003 07:30:23 -0700

Randall,

d/

---

From: "Randall R. Stewart (home)" <[email-masked]>
Date: Sat, 13 Sep 2003 14:46:05 -0500

R

---

From: Jari Arkko <[email-masked]>
Date: Mon, 15 Sep 2003 22:18:23 +0300

> by sending a false ABORT chunk....

Yes. Unless you are encrypting the whole session, on-path attackers can
already do almost anything. They can start a session for you. They can
abort a session for you. They can hijack a session from you. They can
modify a session.

> change the keys... but that goes away if we make the same assumption MIP 

The question you have to ask is: What is the difference between the
"Internet as is" and "Internet with ADD-IP (or MIP)". You can do the
analysis case by case, such as for plaintext communications and
for cryptographically protected communications and with or without
the compromised infrastructure. For instance, assuming plaintext
SCTP packets, presumably in the current Internet all on-path attackers
will be able launch the attacks I listed above. But I hope that
not everyone in the whole Internet will be able to e.g. disconnect SCTP
sessions from a given host. The same properties should stay if you
add a feature such as ADD-IP. On the other hand, if we assume that
the routing infrastructure is compromised, then even in the current
Internet I can go and intercept your plain sessions and cause all kinds
of interesting problems. I would allow this to happen even with ADD-IP,
as long as it does not make the problem worse.

With cryptographic protection (e.g. IPsec) on the SCTP packets,
you should be safe even from on-path attackers. Again, if you add
ADD-IP feature the same property should stay. Note that there are
some DoS attacks that remain regardless of cryptographic protection.
For instance, by interfering with ARP/ND you could block the flow
of packets.

--Jari