Subject: [Tsvwg] multiaddressing

From: Wesley Eddy <[email-masked]>
Date: Mon, 15 Nov 2004 13:40:47 -0500

In draft-crocker-mast-analysis-01 (now expired), the concept of
"multiaddressing" for solving multihoming and mobility problems is
explained.

The HIP groups and multi6 are focusing on multiaddressing solutions
built for "simple" transports that can only bind to a single address
with no means to update or extend that binding when it might be required
by mobility or multihoming.  Adding a wedge-layer to map dynamic locators
to fixed identifiers fixes these problems and allows the transport to go
untouched and remain simple.

A drawback of such wedge-layers, Mobile IP, and other systems that hide
mobility (or multihoming) from the transport is that current transport
mechanisms are not nearly "agile" enough to deal with end-to-end path
changes that can pop up instantly.  The transport is where all the path
estimates are stored (congestion control variables, RTTM, PMTUD, etc) and
making non-trivial path changes without some explicit signalling to the
transport can be dangerous.

If we're going to see multiaddressing wedge layers deployed, then it seems
like we should have a slightly richer interface between the wedge and the
transport so that reinitialization and reestimation of path state can happen
rapidly.

Architectures that don't include such wedges are what we were
considering in draft-eddy-tlmarch-00 (presented last week).  The idea of
rapidly fixing the transport's path estimation variables is roughly
visible in draft-swami-tcp-lmdr-04.  For architectures that *do* include
wedges, some similar form of signaling and responding to path changes is
still _required_.  Signaling has to occur both between the wedge and
transport locally and remotely, so that both half-connections can be
fixed.  Responses to these signals will have to be implemented
per-transport protocol.

It's not entirely clear to me how we can combine the strengths of a
multiaddressing wedge layer with the rapid-response of path property
estimates that we can have without the wedge in a transport layer
mobility scheme.  Thoughts?

-Wes
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 24 Nov 2004 09:13:15 -0500

> untouched and remain simple.

I previously brought up the idea that a multiaddressing wedge should at
least pass a bit up to the transport to notify it of mobility events.

Another consideration, in comparison to systems where the transport does
all the book-keeping of addresses and handles mobility and multihoming
on its own, is that the transport<->wedge interface will likely not be
rich enough for the transport to do the sorts of things that SCTP is
capable of, like retransmiting on alternate paths, load balancing
between interfaces, etc.  I'm not sure that the wedge interface could be
made rich enough to support those types of transport decisions without
completely negating the benefit of its simplicity, and certainly we
don't want the wedge to be doing those types of things silently itself
without the transport's consent.

It looks like this issue is another point in favor of transport-based
mobility/multihoming schemes.  I'm still in favor of passing the
book-keeping off to a common wedge somehow, but it looks like there's
still quite a bit of utility in having the transport be able to query
and richly interact with that wedge, rather than have it just
transparently "make things work".

-Wes

---

From: Mark Allman <[email-masked]>
Date: Wed, 24 Nov 2004 11:48:26 -0500

(This is not really my ball of wax, but a thought ...)

> certainly we don't want the wedge to be doing those types of things
> silently itself without the transport's consent.

Your assumption.  Not mine.

At the risk of forming an unholy marriage of buzzwords, HIP + XCP ==
clean separation of concerns (amongst IP, HIP, XCP, TCP/DCCP/SCTP/blah).

(Where I use "HIP" to mean some form of node identifier that isn't the
IP address and "XCP" to mean some congestion control that is not built
into the transport and with fine-grained path information from the path
itself.)

allman

--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 24 Nov 2004 11:58:35 -0500

> clean separation of concerns (amongst IP, HIP, XCP, TCP/DCCP/SCTP/blah).

That's not going to work so well, without some heavy modifications.  If
you let the wedge do load balancing, for example, it's going to be
shuffling things the transport sees as going on/off a single virtual path
onto multiple paths.  Then the wedge will be passing up XCP data from
multiple paths to the transport, that looks like it's coming from a
single path.  How is the transport to interpret this?  Would you propose
to make the wedge figure that out too by somehow aggregating the XCP
data and passing up smoothed versions reflecting the virtual path?

-Wes

---

From: Randall Stewart <[email-masked]>
Date: Wed, 24 Nov 2004 12:14:26 -0500

Ok, I will finally comment a bit on this :-0

>>"multiaddressing" for solving multihoming and mobility problems is

The crocker draft was just an attempt to take the
SCTP add-ip draft and duplicate it into a lower level.. I never
thought it would work since the reason we have been holding
the add-ip draft is the fact that there are security issues
with it. These should be solved as soon as we finish the
work on the auth draft.. but you can't do that without state which
means a shim layer can't do it.. unless you want to mirror all
the transport state at a layer below transport.. ughly and far
from simple... I know that first hand.. I am working on NAT
and SCTP and let me tell you its tricky when restarts can
show up .. and thats not even the multi-homing .. just single
homed SCTP :-0

> without the transport's consent.

I just don't see how you can have anything simple below transport
doing this.. It just gets more and more complicated.. IMO we should
leave it where per-connection state can be maintained so one can
do the necessary things...

R

> ------------------------------------------------------------------------
> 

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Ted Faber <[email-masked]>
Date: Wed, 24 Nov 2004 09:34:06 -0800

> data and passing up smoothed versions reflecting the virtual path?

Ummmm, without *some* path continuity/locality congestion control is
pretty difficult.  How often do you expect paths to change?  If it's
per-packet with high probability, this is a very tricky thing.  If the
paths change once a minute (for example), it's a minor issue - not much
worse than starting a new connection every minute.

You can define an unsolvable problem here.  Do you have to?

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG 
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 24 Nov 2004 12:44:55 -0500

> worse than starting a new connection every minute.

I was talking about load balancing over multiple _static_ paths, and
claiming that a wedge could/should not provide this service.  Perhaps I
wasn't very clear.

> You can define an unsolvable problem here.  Do you have to?

I don't see load balancing over multiple static paths (with congestion
control) as unsolvable,  It can be done in SCTP.  I think we had a minor
translation error between what I meant, what I said, and what you read
:).

-Wes

---

From: Mark Allman <[email-masked]>
Date: Wed, 24 Nov 2004 12:50:46 -0500

Wes-

> That's not going to work so well, without some heavy modifications.

I know I'm a little funny, but I don't mind modifying things that don't
exist. :-)

> If you let the wedge do load balancing, for example, it's going to be
> shuffling things the transport sees as going on/off a single virtual
> path onto multiple paths.  Then the wedge will be passing up XCP data
> from multiple paths to the transport, that looks like it's coming from
> a single path.  How is the transport to interpret this?  Would you
> propose to make the wedge figure that out too by somehow aggregating
> the XCP data and passing up smoothed versions reflecting the virtual
> path?

I don't understand how XCP could possibly make this situation worse.  In
a non-XCP world the transport that is trying to assess such a path has
the same issues and less information.  In the load balancing example you
give my head might explode without XCP.  TCP will probably settle on the
lowest common denominator.  Probably.  In that case, the information XCP
provides sure sounds handy to me --- precisely because implicit
information will look like such a tangled mess.

HIP+XCP sure sounds like an interesting avenue to think about from here.

allman

---

From: Ted Faber <[email-masked]>
Date: Wed, 24 Nov 2004 10:07:54 -0800

> translation error between what I meant, what I said, and what you read

Good diagnosis, doc. :-)

Multiple static paths is trickier than single paths but solvable.

I think Mark's point of separations of concerns is right on here.  Each
path would be congestion controlled individually and the balancer could
pick.  But since each path is basically a transport connection, why not
put the load balancer above transport, say in the application where it
has some app-based knowledge of how to prioritize traffic?

---

From: Randall Stewart <[email-masked]>
Date: Thu, 25 Nov 2004 06:35:12 -0500

> translation error between what I meant, what I said, and what you read

Jana at the U-Del is doing his PHd thesis on this very thing. There
are several papers on the subject at:

http://pel.cis.udel.edu/

(look for Concurrent Multipath Transfer - CMT)

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 25 Nov 2004 06:39:30 -0500

> Multiple static paths is trickier than single paths but solvable.

Yep
> has some app-based knowledge of how to prioritize traffic?
Well.. that works sort of.. except there are other problems.. when
a path switch occurs fast during the balancing you will get a
problem with the way the cwnd's grow (some of Jana's earlier work
shows this.... the only solution we found .. if the transport is not
invovled is to disable fast retransmit... which is ugly)...

And as far as XCP goes.. I would not hold ones breath waiting
for routers to provide all this wonderful information... it can
be done but its expensive and not something your likely to see
in production anytime soon.

R

---

From: Janardhan Iyengar <[email-masked]>
Date: Mon, 29 Nov 2004 12:14:06 -0500 (EST)

Hi Mark/all,

> information will look like such a tangled mess.

Implicit feedback is a mess if the transport is unaware of the existence
of multiple paths. If the transport is aware of multiple paths, and has
knowledge of data scheduling on multiple paths (as against a wedge/shim
under the transport), load balancing can be done without anybody's head
exploding. Such a transport *can* separate per-path information, of course
with added effort. We have managed to get SCTP to do exactly this
separation - see
http://www.cis.udel.edu/~iyengar/publications/2004.spects.iyengar.pdf

regards,
jana

---------------------------------------------------------------
Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
Protocol Engineering Lab  --   CIS   --  University Of Delaware
---------------------------------------------------------------

---

From: Janardhan Iyengar <[email-masked]>
Date: Mon, 29 Nov 2004 12:35:21 -0500 (EST)

> > control) as unsolvable,  It can be done in SCTP.  I think we had a minor

I agree. See our work on Concurrent Multipath Transfer -
http://www.cis.udel.edu/~iyengar/publications/

> pick.

Each path can also be congestion controlled individually within the
transport itself.

> But since each path is basically a transport connection, why not put the
> load balancer above transport, say in the application where it has some
> app-based knowledge of how to prioritize traffic?

I want to point out a subtle difference here. Though each path is close to
a transport connection, there is significant information sharing that
occurs across paths within a transport connection used over the multiple
paths. Looking at throughput as a measure of performance, we found that
using an application with multiple transport connections on individual
paths performs significantly worse than using a single transport
connection that spans multiple paths (See
http://www.cis.udel.edu/~iyengar/publications/2004.icon.iyengar.pdf,
Figure 1, Figure 6 and Section V-A in particular). This performance
difference was an unexpected result for us, as we were hoping that CMT
would match up to the performance of an ideal load balancer at the
application layer (AppStripe). CMT outperformed AppStripe by a significant
measure.

I agree that app-based knowledge of traffic prioritization is not used, we
assume throughput to be our measure of performance.

regards,
jana

---

From: Ted Faber <[email-masked]>
Date: Mon, 29 Nov 2004 10:14:44 -0800

This may be responding more to Jana's mail, but I'm going to be a little
lazy and leave it here where I typed it.  I am bad.

> invovled is to disable fast retransmit... which is ugly)...

I haven't tried to do this, but (I'm assuming that you'll just disregard
the rest of this paragraph, as you should) it's clear that the load
balancer needs to have some special magic in it.  It seems to me that
just putting a bunch of transport connections down and striping the data
with no regard for the properties of the application and the transport
will have problems.  A good load balancer would need to take into
account how aggressive each transport needs to be (for example, the
folks in DCCP are finding out that TFRC behaves oddly if it's supporting
a fixed rate connection and slow/fixed connections in XCP may confuse the
fairness algorithm).  Even a system like TCP needs to probe the network
for spare capacity or reduced capacity. For applications that send data
that's of different value (bits that make the background look spiffy vs.
bits that draw an opponent in a vidoe game, perhaps) the balancer might
want to take that into account, too.

Those are all hard problems, and it's not clear there are good,
efficient (economical?) solutions. Still, I don't see how tackling them
in the transport makes things any easier and in the case of integrating
application info, it makes it harder.  IMHO, transport should provide a
set of quasi-static (the paths change infrequently) congestion
controlled pipes that the application can use to do its work.  My design
sense says that transport should provide a building block that can be
used to implement this multiaddressing/multipath load balancing/mobility
thingie.

Are there simple changes to transport that would help this?  Are there a
few things that this multiaddressing/multipath load balancing/mobility
thingie would require of transport connections that would facilitate
building using transport connections?  Answering that question seems
more fruitful to me than trying to put a big new function in transport.
And if it turns out to be easier than I think it is to build a
multiaddressing/multipath load balancing/mobility thingie, migrate it
down the stack when it's done.

> in production anytime soon.

Yeah, I understand some of the problems and I'd like to understand more.
I do think that XCP has been pretty clever at minimizing new state
needed in the routers, but there are lots of questions about doing the
computations.

I'm not expecting to see it tomorrow, but with all the additional
computing power Moore keeps bringing, it's probably wise to think about
how we can use it to help with congestion, if we can.

---

From: Janardhan Iyengar <[email-masked]>
Date: Mon, 29 Nov 2004 14:35:45 -0500 (EST)

Hi Ted/all,

> application info, it makes it harder.

I agree. As far as solving problems is concerned, load balancing in the
transport is probably no simpler than in the application. The application
has better knowledge and understanding of the application's needs and
demands, and the transport has better understanding of path/network
characteristics. Either ways, information may need to be consolidated and
exchanged across layers.

> IMHO, transport should provide a set of quasi-static (the paths change
> infrequently) congestion controlled pipes that the application can use
> to do its work.  My design sense says that transport should provide a
> building block that can be used to implement this
> multiaddressing/multipath load balancing/mobility thingie.
>

An application layer solution migrated down the stack still remains a
solution *on top of* the transport layer, not *in* the transport layer. I
do not think that developing a solution inside of transport is the same as
developing a solution on top of transport, and migrating it down. I have
found that there are benefits in doing load balancing *in* the transport,
as against *on top of* the transport. These performance benefits may not
dictate design choices, but IMHO, are certainly of value in making these
choices.

regards,
jana

---

From: Janardhan Iyengar <[email-masked]>
Date: Mon, 29 Nov 2004 17:15:02 -0500 (EST)

Hi Ted/all,

> > down the stack when it's done.

There is also the question of what services are common enough to be pushed
into a common layer below the application layer, to avoid repeated
implementations in applications. If load balancing is a service that most
applications would use (if provided) for increased throughput, the service
should, IMHO, be provided in transport. When there are demonstrable
benefits in adding some services to transport, is there a reason for
keeping such services outside transport?

regards,
jana

---

From: Ted Faber <[email-masked]>
Date: Mon, 29 Nov 2004 15:28:18 -0800

> keeping such services outside transport?

I imagine you've herad the rumors that there might be layers between
transport (4) and application (7).  I haven't really seen them, either,
but it seems this load-balancing whatsis might be a good thing to put
there, at least conceptually.  :-)

I say this because I have a hard time imagining one load-balancing
whatsis that is good for everyone, but with DCCP & TCP and maybe XCP out
there, I can imagine a reasonably sized set of load-balancing whatsi
that snap on top of the various transports (maybe more than one
transport at once) and provide different types of load balancing
behavior.  You might want to implement your whatsi in libraries or call
them protocol layers.  I think there may be reasons to think of them
either way.

I think that approach lets me keep simple transports (well, simplish -
TCP and SCTP are somewhat intricate) while trying out some of these load
balancing ideas in a way that can be passed around between apps and
developers as well.  You do need the transports to play nice - maybe
export current estimated throughput and RTT info - but I like the
modularity of talking to transports rather than subsuming transports
better.

Now, it may be that the cost of that modularity is crushing, but I
didn't hear you say that layering cost was the showstopper.  Did I just
miss that?

---

From: Ted Faber <[email-masked]>
Date: Mon, 29 Nov 2004 15:30:23 -0800

> dictate design choices, but IMHO, are certainly of value in making these

I'm a slow thinker.  Can you give me an example of a benefit to doing
load balancing in transport that's not an implementation workaround, but
a problem with using the transport service to do what you want?

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Nov 2004 00:05:56 -0800

>>I thought I should add one more point to my earlier response:
...
>>keeping such services outside transport?

A number of reasons; these are some of the reasons that IP is outside 
TCP, and why the multi-6 WG is focusing on multhoming as a shim that is 
outside the transport protocol as well.

In particular, data unit multiplexing is IP's job. Making it TCP's job 
too means that different TCPs - or SCTP and TCP, etc. - can have 
different muxing systems that either interfere or invert priorities.

The argument that "this is something apps need, so let's share it and 
put it in the transport layer" holds even moreso for pushing it into IP; 
there are many transport protocols that would benefit from load-balanced 
muxing, and putting it even lower makes it a generic pseudo-link 
service. The further benefit is that whatever muxing you do at the 
single node might be applicable further down the line as well, in 
routers, and so should be at the network layer optimally.

There's also the idea, as Ted mentioned, of something outside TCP; this 
can be somewhat parallel, ala the Congestion Manager. That was an 
attempt to help multiple TCPs play nice using a single mechanism, rather 
than replicating it through many TCPs.

Joe