Subject: PILC ASYM document

From: Hari Balakrishnan <[email-masked]>
Date: Wed, 01 Mar 2000 18:42:48 -0500

Reviving an old, old thread about draft-ietf-pilc-asym-00.txt (Balakrishnan & 
Padmanabhan):

On Mon, 22 Nov 1999 10:23:46 -0600, Spencer Dawkins wrote:

> Van Jacobson: RFC 1144 header compression, in one-byte mode (as opposed to
> five-six-byte mode), handles 3000:1 asymmetry. Are we seeing connection
> paths that are more asymmetric than this? What's the point?

This comment was from the PILC minutes of the Washington DC IETF (11/10/99).

It has come to our attention from Aaron and Spencer (chairs of the working group) that we had perhaps not clarified what we meant by asymmetry, and that Van's comment above implied that a push-back was needed on this document.

Venkat and I don't believe that this assessment is necessarily correct for the following reasons:

1. The document draft-ietf-pilc-asym-00.txt isn't about bandwidth asymmetry alone; it defines (perhaps inelegantly) "asymmetry" to be all cases when the characteristics of one direction "significantly" impact performance in the other.  It goes on to describe several kinds of interactions, including bandwidth asymmetry (exacerbated in the presence of bi-directional traffic), media-access asymmetry that causes latency variability, error/loss rate asymmetry.  All of these are _real_ and not just of academic interest.

2. If you have a bandwidth-asymmetric network and ACKs aren't going across a loss-prone path, VJ header compression solves the problem.  Completely.  The draft says as much.

3. But there are important, real cases when things aren't quite so simple:

   (i) Media-access.  Many MAC protocols are RTS/CTS based, requiring a synchronization handshake before communication, with each failed RTS triggering an exponential backoff followed by retry.  This avoids unnecessary data collisions that may otherwise result.  Examples include many packet radio networks and the 802.11 spec.  In such networks, it has been observed that latency can be highly variable, and as a result data and ACKs interact badly.  The consequence is that the RTO becomes very large because rttvar is large.  When a timeout occurs, the idle times are prohibitively large.  In such situations, performance would improve a lot more from *infrequent* acks (and of course you need methods to deal with this at the sender, as described in the draft) than from *smaller* (header-compressed) ACKs.  (Infrequent ACKs need to be dealt with without increasing burstiness, and the draft talks about this at length.)

   (ii) Bi-directional traffic.  I didn't think this was such a common occurrence, but I recently received this message from a DSL customer, which I repeat verbatim below.  (The customer recently graduated from MIT.)

--------------------------------------------------------------------
Date: Mon, 31 Jan 2000 11:47:05 -0800 (PST)
From: [email-masked] (Nelson Minar)
To: [email-masked]
Subject: assymetry and TCP performance?

I thought I was so smart after taking Hari's class last semester, but I'm having a hard time diagnosing a simple problem.

I'm on PacBell DSL here in California. It's pretty good, 1.5 Mbits/sec downstream and 128 kbits/sec upstream or so. Reliable.

Except one problem - when someone is downloading a file off of my web server, saturating the 128kbits/sec upstream link, then all my performance goes to hell. Pinging the gateway takes 2 seconds round trip, and things like DNS or web page fetches are either terrible performance or just don't work.

I'd expect things to be a bit sluggish, but not anywhere near this bad. Is there anything simple written about assymetry and TCP performance? I have some guesses about what's going on, but I'd like to understand this problem better.

Thanks!
  Nelson
--------------------------------------------------------------------

	One might think that this is an uncommon situation, but if you look at the way the Internet is evolving, nodes serving information from people's homes are becoming increasingly commonplace and are likely to dramatically increase in the next few years.  I predict that situations like Nelson's above will not be rare in the near future, when we start putting all our computers and myriad other devices in our homes on the Internet.

   (iii) Loss rate asymmetry: Some of the techniques (e.g., sender adaptation) described in the draft handle this case as well, which is observed in many real systems including mobile ones where end stations transmit at much lower power and contend for reverse slots whereas the forward channel is from a high-powered base station and is therefore relatively loss-free.

Given these situations, we believe that they are worth clearly documenting in a draft.  I don't believe that these are of purely academic interest.  

We are very interested in your reactions to this.

Thank you!

Hari

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 2 Mar 2000 00:09:28 -0700 (MST)

> From: Hari Balakrishnan <[email-masked]>

> ...
> > I don't understand the a priori relevance of such considerations.
> > TCP is not a MAC protocol, and in general is supposed to be blithely 
> > ignorant of oddities in the link layer.
>
> And indeed it is.  Making it robust to infrequent ACKs is an all-round win.  No 
> one is suggesting any link-specific modification to TCP.

I don't understand that.  Where do you have infrequent ACKs in an
active TCP connection except where a strange link layer is being
either very smart or too smart by half, and eating ACKs?  What
happened to the notion of "ACK pacing"?
Making TCP robust to infrequent ACKs is a special change particularly for
links that like to preferentially eat some special kinds of upper layer
frames.  It might well be a good idea, but with no offense intended, 
more than academic assurances would be nice before we rush out
and modify all current TCP implementations.

> > > ...
> > > Except one problem - when someone is downloading a file off of
> > > my web server, saturating the 128kbits/sec upstream link, then all
> > > my performance goes to hell. Pinging the gateway takes 2 seconds
> > > round trip, ...
> > 
> > The problems there are irrelevant to the draft.  First, no matter what
> > you do, you can't get more than 128 kbit/sec from a 128 kbit/sec link.
> > Second, such experiences show that saturating a link with mixtures of
> > interactive and bulk traffic through with routers that no any notion of
> > priority queues is a bad idea, which is not exactly a new discovery.
>
> Gosh, who said anything about interactive traffic!  This happens when you're 
> doing bulk traffic in both directions.

Nelson's only quantitative performance measurement (that I recall) involved
interactive stuff, his 2 second pings.  He didn't say anything (as I
recall) in terms of bits/sec that showed the downstream pipe was not full.
It might not have been full, and one would hope an MIT alumni would do a
complete set of measurements.  However, the left coast does affect people.
I've even known people on the right coast within a rifle shot of MIT to
blow off a little private steam in wildly overstated complaints about lame
ISP services.

>  ...
> > Those problems are not new.  In the good old days, many of us suffered
> > exactly such situations as Nelson's, except that our links were 10 or 50
> > times slower.  Please consider ancient asymmetric dial-up modems including
> > those from (I think) Courier and modems that were sort-of assymetric like
> > Telebits.
>
> Evidently you decided to do nothing about the problem then, same as now.  :)  
> (Sorry, couldn't resist, given your general tone.)

Actually, that's wrong.  We have various link layer things including data
compression, VJ header compression, and various priority queueing at least
partly because of the old problems.  Note that the things we do have
received a lot of real life testing and serious measuring before they were
documented in RFC's.

Besides, one of my favorite sayings goes something like "anyone can always
find something to add to anything."

Some of what you propose might be very good things.  One or two meet my
personal prejudices for what's right and needed dead on.  However, as far
as I can tell, you've offered only academic or qualitative and anecdotal
support for them.  In other words, what kind of RFC should your draft
should become?

> > >    (iii) Loss rate asymmetry: Some of the techniques (e.g., sender
> > > adaptation) described in the draft handle this case as well, which
> > > is observed in many real systems including mobile ones where end
> > > stations transmit at much lower power and contend for reverse slots
> > > whereas the forward channel is from a high-powered base station
> > > and is therefore relatively loss-free.
> > 
> > That sounds like the old argument that the upper layers must be
> > modified for every new link layer that comes along, regardless of 
> > whether how different it really is (think about Telebits).  
> > Some people reflexively resist that argument.
>
> In general, reflexive reactions aren't well-advised in a scientific discipline

In general, that is false.  This isn't a scientific discipline, but if it
were, Occam's Razor would be a reputable reflexive reaction.  In an
engineering discipline like our's, Occam's Razor as well as the rule of
thumb that says you don't change things without good, quantitative reasons
are wise reflexes.

>Furthermore, many of the solutions advocated in the draft are link-layer ones

The whole point of your proposals is to affect TCP.  Most of your proposals
("solutions"?--sheesh!) might large effects on TCP, assume TCP has certain
behaviors (e.g. RTT measurement won't be trashed by ACK compression or
performance won't be zapped by re-ordered TCP segments), or assume TCP
will be changed at least slightly (e.g. made "robust to infrequent ACKs").

> The changes proposed to TCP itself are all ones that DO NOT degrade TCP 
> behavior or performance over any other paths, and often, they improve them.
>
> Ergo, I believe they are a good idea; the solutions are not of the form where 
> they improve performance in one situation and degrade it in several other 
> places.

I'm sure you think so.  Over the years I too have had wonderful ideas that
I was sure would have lots of wonderful results and no bad effects.  I
was even right on the second half about one or two of them.   In other
words, if you're proposing a BCP or STD, it's far too early.  Still, the
ideas would be interesting to test seriously, and I don't mean a couple
of master's thesis simulations suitable for IEEE publication.  (I hope no
one has done any of those for these ideas since I don't intend any
particular offense by that point.)

And characterizing your proposals as "solutions" again?  I'd go for
"proposed solutions", "possible solutions," or even "likely solutions",
but you're a *long* way from bald "solutions".  You haven't even offered
convincing evidence for all of your characterizations of the problems!

> ..
> > The interesting question is what happens to a draft after it has been
> > published.  How people are supposed to respond to this draft?  Is it
> > intended to become a Proposed Standard or a BCP?   Or would it become
> > just another Informational RFC?  Or is it a call to start some experiments?
>
> Frankly, whether or not the draft exists makes no difference to me personally
> However, I think that not being aware of some important issues (not merely 
> academic ones) that affect TCP performance over some classes of real networks 
> is not a good situation.
>
> It isn't a call for more experiments necessarily, although that would be 
> welcome.  I assumed it would be informational.  I don't know for sure though
> I suppose it also depends on the WG chairs.

Urging serious experiments of the ideas is a good idea.  However, you keep
talking about "solutions," which would be strong for an official working
group Informational RFC.

Vernon Schryver    [email-masked]

---

From: "Falk, Aaron" <[email-masked]>
Date: Thu, 2 Mar 2000 14:26:51 -0500

> just another Informational RFC?  Or is it a call to start 

Vernon-

The ASYM document is in our charter to be published as a BCP. This choice
was made before a document was written and the assumption was that it would
contain a) a description of the problems, b) proposed well-understood
mitigations (implemented at the layer of your choice), and possibly,
well-caveated proposals for future protocol development. The PILC charter
does not include developing any new protocols. However, if we find something
that we believe should be developed, we can approach the IESG about
rechartering the working group.

The working group has several options: 

- the topic can be discarded as not relevant and the draft allowed to expire
(or submitted as a non-wg draft by the authors), 

- the draft can be submitted by the working group as an Informational or BCP
RFC, 

- and/or we can seek to develop some protocol changes and an STD. 

A key to determining what will happen will be how large the consituency is
and deep the need is. Hence this discussion...

Before the working group could take on developing a new protocol, a
compelling argument needs to be made for the need of such an activity. Hari
and Venkat have made their case that this is a real problem and I'm looking
to the mail list for validation that the problem merits further definition
and the proposals merit further development by this working group.

I appreciate your comments and invite more.

--aaron

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 2 Mar 2000 15:07:01 -0700 (MST)

> From: "Falk, Aaron" <[email-masked]>

> rechartering the working group.

It is clear Hari's document cannot conceivably be (part of) a Best Current
Practices in the immediate future.  I think Hari agrees with that, despite
his unqualified uses of "problem" and "solution."  Before ACK regenerating,
ACK-only scheduling, or the other ideas could be listed in a BCP, we need
real testing to see what they break.  (Claims that nothing would break
are obvious nonsense; every worthwhile change in a system as big as the
Internet breaks something that someone somewhere values.   Worse, no one
knows enough about the system to predict accurately what will break.)

The many ADSL networks provide many test beds for some hacks on free
UNIX and Linux kernels.  Some artificial throttling and random filtering
might ADLS links similar to wireless links.  If the ideas are safe
enough, users of modified systems would not notice the change.

> and deep the need is. Hence this discussion...

The issues (note I didn't say "problems") listed in Hari's document have
long had a significant consituency outside the IETF.  (Yes, I'm yet again
refering in part to cable modem games with ACKs.)  I think the issues are
more worthy than many other IETF activities, such as the zillions of MIB's.

> I appreciate your comments and invite more.

The practice followed by Van Jacobson, Sally Floyd, and others has worked
pretty well over the years.  They do lots of simulating, testing, and more
than prototype implementing before publishing anything, or at least before
announcing solutions or proposing IETF sanctioned changes in widely
deployed implementations.  A recent example is the change in the initial
TCP window, which was done at least partly under the auspices of the TCP
Implementors working group.  (I don't mean to say anything about which WG
should own which territory, but only offer a relevant mailinglist archive.)

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 3 Mar 2000 11:57:30 -0700 (MST)

} From: mukul goyal <[email-masked]>

} THis is a diversion from the topic on hand ...

it seems relevant to me.

} It might be a good idea to have a document that lists accepted definition
} of concepts like this.  

More of a definition of ACK clock, pacing, etc. would be good,
especially in a document that mandates or at least encourage
practices that, as Craig Partridge said, "muddy the spacing".

  ....

From: Dr G Fairhurst <[email-masked]>

] The well proven approach taken by Sally Floyd, et al (just using
] the name quoted on the list, no offence to Sally) is a good one
] for some problems.  But it does not always work.

Yes, doing "lots of simulating, testing and more than prototype
implementing before publishing anything, or at least before announcing
solutions or proposing IETF sanctioned changes in widely deployed
implementations" does not always work.  Failure is part of inventing.
However, defining protocols (e.g. ACK regeneration) without doing all of
that other stuff is almost certain to fail.

Examining relevant history (e.g. asymmetric dial-up modems carrying
TCP/IP/SLIP and TCP/IP/PPP) and long standing, current industry practice
(e.g. ACK compression in the metropolitan cable TV modem systems of
Hybrid Systems installed for the last 6-10 years) is as good in an
academic paper as in an official WG RFC, even if only Informational.

  ....

> From: Venkat Padmanabhan <[email-masked]>

> ...
> We also have anecdotal evidence of similar problems in cable modem
> systems. We were hoping to hear experiences of other people confirming
> or refuting this.

How would such experiences be other than still more anectotal evidence
that could neither confirm nor refute the draft?  Don't we need some
prototype implementing and serious testing before advocating ACK
regeneration or compression?  (Not to mention re-ordering the TCP
segment stream by moving pure ACKs ahead of data)

>                   Barbed emails heavy in sarcasm unfortunately drown
> out whatever technical point was sought to be conveyed.

I haven't intend as much barbed sarcasm as seems to have been read.  But
if you keep twisting my tail by talking about a BCP openly based on a few
tests, some thinking, a few anecdotes, and not even a nod at historical
or current industry practice, I'll start meaning it.

> Regarding your implication that TCP should be ignorant of the oddities
> of the link layer, an analogy might illustrate why this argument isn't
> necessarily right.  HTTP/1.0 (an application-level protocol) ignored 
> artifacts of the underlying TCP transport protocol (e.g., slow start)
> and opened a new TCP connection for each request. Consequently, performance
> suffered.

That's a very good example.  Would it have been better to modify TCP/IP
to fit HTTP/1.0 or to modify HTTP/1.0 to fit TCP/IP?  Do you think a BCP
recommending that slow start be turned off for HTTP traffic should have
been written instead of the work on HTTP/1.1?  Do you think that the
tests that supported increasing the intial TCP window were not needed,
that it would have been wise to mandate that change without what passes
for due consideration in the IETF, running code?

> ...
> What we are talking about of here is not "mixtures of interactive
> and bulk traffic". Rather it is the interference between the acks of
> a downstream bulk transfer with data packets of an upstream bulk transfer.  

Did I fail to see the numbers in Hari's message from measurements that
supported the hypothesis that the downstream pipe was not full?  I didn't
keep a copy of the original message, and I remember only a mention of
ping as support for the hypothesis.  Do you agree that the 2 second ping
measurement contradicts or at best shows nothing about the hypothesis?

> ...
> I have no idea what Telebits is, but regardless we are not arguing 
> that the upper layers must be modified for every new link layer.

You really should look at Telebit and Courier modems.  You might find
that many of the problems listed in the document are 10 or 15 years
old.  That's not to say that they have already been solved or shouldn't
finally be solved, but that you might then have more than anecdotes.

> Rather we have attempted to document a set of problems related to
> asymmetry and an approach to addressing these that has a common 
> thread: reducing the frequency of acks -- a generalization of
> delayed acks, if you will -- and making the sender robust against
> disruptions in the ack stream. These have been derived from our
> experience/experiments with a few network technologies. Whether to 
> turn any of this into a proposed standard would depend on whether 
> there is a consensus on their relevance in a wider setting 

If "disruptions in the ack stream" did not mean intentional, consistent
filtering mandated by the same BCP, who could disagree with that?

> > Is it intended to become a Proposed Standard or a BCP?   Or would it
> become
> Good question. As Aaron states, the PILC WG's current charter precludes
> turning this into a proposed standard (of course, I realize that this
> technicality is not the only hurdle in the way!).

I don't see why that should be a hurdle.  The IETF is not yet the IEEE
where very heavy duty politicing happens before a committee gets to
own its turf, or the ANSI where the most weighty deliberations in any
committee concern defending its turf from other committees.  The IETF
is supposed to be about "running code" even before "rough concensus."
Let's have some running code, and then worry about the official documents.

>                                                   If there is sufficient
> consensus and supplementary/supporting evidence (e.g., experiences with
> other network technologies), then (parts of) the document can
> be turned into a BCP. ...

How could the WG publish a Best Current Practice about things that are
not Current Practice and are not known to be harmless, not to mention
Best?  As far as I can tell, no one has even started to demonstrate that
the various ideas for modifying the ACK stream do more good than harm.
For example, if ACK compression as great an idea as you think and my
prejudices agree, then why is it as if the large scale commercial
installations that use ACK compression do not exist?  Why do so many
knowledgable people only grimace when you mention ACK compression?

It's early and unnecessary to write a BCP that might be turn out to say
the equivalent of "disable slow start on all hosts because otherwise
HTTP/1.0 works poorly."

>                       If not, I guess it will be an Informational RFC.

What's wrong with doing some prototype implementing and serious testing
before worrying about publishing an official document or whether it should
be Informational, STD, or BCP?   The point of the IETF is not merely
publishing documents about ideas for protocol changes, but "running code."

---

From: Marie-Jose Montpetit <[email-masked]>
Date: Fri, 3 Mar 2000 15:46:25 -0800

I would also stress that while I never believed we should change the
Internet protocols to support new links with asymmetry or delay or noise (in
fact all pieces of the bigger pie), I think it is important to have the
different communities at least sharing the issues and have the people
designing the links (the subnets) be aware of what is running end to end
over them.

Marie-Jose
-----Original Message-----
From: Falk, Aaron [mailto:[email-masked]]
Sent: Friday, March 03, 2000 6:03 AM
To: 'Vernon Schryver'
Cc: [email-masked]
Subject: RE: PILC ASYM document

Vernon wrote: 
> Implementors working group.  

Vernon-

You've made an excellent point. Actually, getting researchers of the caliber
you mention is part of the goals of this group. (Sally, Van: are you
listening?) However, to get networking researchers engaged in a problem
there needs to be a clear articulation of exactly what the problem is. Part
of that discussion would be to propose some candidate solutions. This is why
PILC isn't developing any protocols -- we are supposed to be bringing
together user/developer communities
_who_may_not_be_experienced_in_Internet_protocol_R_&_D_ to document the
problems they see on links of interest. One possible outcome is that an
Informational RFC is published. Getting to that point will require a rough
consensus on the issues. The dialog will likely be of greater value than the
document. These dialogs can have profound influence decisions on IETF
activities in the future.

Some might say (& I heard it about TCPSAT), that this isn't relevant work
for the IETF, since the IETF develops protocols for the Internet. My
understanding is that the IETF's charter is not quite that narrow. PILC is
doing the advance work to identify whether a new protocol(s) needs to be
developed. Part of that process is requirements definition (such as is going
on in ECM). But before you can define requirements you need agreement on
what the issues are and whether they require fixing. *That* is what we are
doing now. 

(I might add that TCPSAT was very effective in giving the satellite
community a clue on issues surrounding TCP performance. This was/is a
community where most designers didn't have a background in Internet
protocols and didn't understand the performance/design tradeoffs. To a
certain extent, PILC is a generalization of the TCPSAT goals to a wider
community of interest.)

--aaron

---

From: Venkat Padmanabhan <[email-masked]>
Date: Mon, 6 Mar 2000 10:30:45 -0800

> That analogy is misleading.
> 
> HTTP _always_ runs over TCP, so there are widespread gains from
> optimising interaction between the two. It's a global (a World-Wide,
> even) optimisation with few tradeoffs.
> 

Good point. A more appropriate analogy may be TCP header compression 
over slow speed links wherein transport-layer specific knowledge is
exploited in the link layer of a particular class of links. I would place
techniques such as ack filtering/reconstruction (discussed in the draft)
in the same category -- link layer algorithms that utilize TCP-specific
knowledge without necessarily modifying TCP itself. In that sense,
I believe these are bang on target for PILC; basically, designers
of such links may need to spinkle some TCP-specific magic dust to
have their links perform well with unmodified end-to-end TCP.
(N.B.: The draft surveys a whole range of alternatives, some of which
do involve mods to the e2e TCP.) 

> TCP does not always run over e.g. some braindead ARQ-to-infinity link
> layer. Optimising TCP for such a special case would prove detrimental
> to interactions with other, different, link layers. It's a local
> optimisation with many tradeoffs.

Yes, it's a local optimization. But it may well be possible to 
confine it to just the problematic link (for instance, by using 
an ack reconstructor to undo the effects of ack filtering) thereby
minimizing/eliminating adverse interactions with e2e TCP.

By the way, as I mentioned in my previous message (as also during 
my presentation at the DC IETF), the problem of upstream packets
(acks in the case of a downstream transfer) being expensive to send 
may be a fundamental consequence of hub-and-spokes network architectures
rather than an artifact of braindead link layers.

-Venkat

Venkat Padmanabhan
Microsoft Research
[email-masked]
http://www.research.microsoft.com/~padmanab

---

From: Michael Orr <[email-masked]>
Date: Thu, 9 Mar 2000 15:12:49 +0200

> --aaron

	Put me down for "yes, it is a real problem, and worth
discussing/solving". 
	I am not quite sure if a new protocol is required, but if we don't
thoroughly 
	discuss, how will we decide? we need to see the trade-offs. 

--
Michael Orr
[email-masked]