Subject: Re: pilc minutes for IETF 48

From: Vernon Schryver <[email-masked]>
Date: Thu, 12 Oct 2000 09:06:17 -0600 (MDT)

> From: Reiner Ludwig <[email-masked]>

> ...
> > > OK, so my wording was a bit loose. With the "link's capacity" I mean the
> > > number of TCP segments (MSS) that fit into a *link layer RTT*.
> >
> >Given the speed of light and so forth, that must mean that the near end
> >of the link has buffering of at least 4x MSS.  The MSS is at least 1500,
> >so 4X is 6 KBytes, and 6 KBytes at 14000 bit/sec is 3.4 seconds.
>
> Good point. Although I don't agree to the "... must mean that the near end
> of the link has buffering of at least 4x MSS".

I did not use the word "must" to mean "required by standards,"
but to mean "unavoidably implied" by the hypothesis that the
link has 4X of RTT of buffering.

> It means that the link layer sender can have 4x MSS worth of bytes in 
> flight, i.e., unacked at the link layer. That sender would not need to have 
> any queue at all if it received the next packet from the higher layer (IP) 
> just when it has sent out the (last bit of the) preceding packet/frame.
>
> Instead I would say "... the near end of the link should have buffering of 
> at least 4x MSS".

You declare "should" while I infer "MUST NOT" (in the RFC sense).

> The point you raise is that the rule-of-thumb we discussed in the last mail 
> does not work on slow links.

I strongly disagree.  That you don't like the implications is insufficent
reason to amend the rule of thumb.  The reasons for the rule of thumb
are compelling and unavoidable.  Any link with frequent (not only typical)
delays of 500 milliseconds or more will be considered practically useless
by the majority of users.  A link that frequently suffers 3 second delays
will be completely rejected for any and all interactive applications.  It
does not matter why the link delay delay is frequently so high; it matters
only that essentially all IP applications other than email and possibly
instant messaging used by any except an insignificant minority in the 21st
Century are too frustrating to use with such delays.  Even email, when
the words means IMAP or POP, cannot at best barely tolerate 3 second RTT's.

On the other hand, if the hand-offs, fade, or whatever else that would
cause 3 seconds of delay are infrequent, then there are no reasons to need
4x MSS of buffering and still compelling reasons to not have it.

>                              Thus, I propose the following modification:
>
> The upper limit for the number of bytes allowed for queueing at a link per 
> flow before dropping occurs should be
>
> ("fixed worst-case e2e RTT excluding the link layer RTT" + "link layer RTT")
> x link speed.

That makes sense only with the familiar wireless-Telco reasoning based on
the premises that IP history is irrelevant and that anyone except telco
people cares about radio links.  Both premises are contrary to fact. 

10 or 15 years ago people used TCP/IP over 9600 bit/sec modems with 3-5
seconds of buffering.  They hated the results, and so either commonly
reduced the buffering or used only non-interactive applications.

No user wants to care about any link layer's RTT or other characteristics.
Users use TCP/IP for the applications, not to support telcos and telco
suppliers.  The only RTT that matters to users and applications is the
end-to-end RTT.  The reasoning says that people will change their behavior
and expectations to accomodate whatever a consortium of link layer vendors
finds convenient to sell is literally clinically insane, whether used by
the WAP forum now or the ATM Forum 10 years ago.
If you put too much buffering in any router or other box in the path and
so make the path RTT more than about 500 ms, you will do the otherwise
impossible and make people care about your link layer's characteristics.
Unfortunately, they'll care only about staying away from it.
(Yes, the ATM Forum's insanity was more complicated and not as obvious.)

Because of government help, it was once true that whatever was good for
PTT's was by definition good for their customers.  That is no longer true.

> >] If you further restrict this to say that this is the upper limit *per
> >] flow*, I would fully agree with you again.
> >
> >If each flow gets fair access, then yes.
> >Otherwise, the limit must be shared by all flows.
>
> I agree. In that case, one might need to have some idea of the average 
> number of concurrent flows sharing the link.

Why?  Enough memory for 200-300 ms of buffering for as many flows
as can possibly go to a single mobile phone is, if not free, far
cheaper than other things.

The number of flows that a single mobile phone (or IP host on the
far end of a 1985 vintage Telebit Trailblazer using the Packet
Ensemble Protocol (?) or PEP) is not infinite.  Among its bounds
are the reverse bandwidth consumed by TCP ACKs and the serialization
delays each flow suffers due to others.

Vernon Schryver    [email-masked]

---

From: Reiner Ludwig <[email-masked]>
Date: Fri, 13 Oct 2000 11:26:41 +0200

At 17:06 12.10.00, Vernon Schryver wrote:
>I strongly disagree.

That's OK. Then we can hopefully continue this productive (and I really 
mean it) discussion ...

>Any link with frequent (not only typical)

I strongly disagree ... see below ...

>reduced the buffering or used only non-interactive applications.

So let's stay with that example since 9600 bit/sec (sync. = 1200 bytes/sec) 
is exactly the ideal (no errors on the link) bit rate you get on 
state-of-the-art wide area wireless networks: IS95 and GSM.

Even if you flame over telco networks so much, if you really need to rely 
on wide area coverage (beyond nation-wide) then these networks are 
state-of-the-art.

When travelling I (and so do many other people) always and frequently use 
GSM to download e-mail from our Intranet and if I really need a piece of 
info I do HTTP. Believe it or not, there are even situations where I have 
to telnet, and it works OK.

I set my MTU to 1500 bytes since I'm mostly doing bulk data stuff and want 
to get highest throughput (I almost never do telnet and bulk data at the 
same time ;-). Now, the time it takes to transmit a 1500 bytes segment and 
the returning 40 bytes ACK over 1200 bytes/sec is about 1300 ms (not 
counting the PPP framing overhead). Like it or not, but for GSM we need to 
add another 200 ms latency (propagation delay if you prefer) *one-way* over 
the GSM hop. We are at an e2e RTT of 1700 ms by now. Adding a worst case 
Internet RTT of 500 ms which is not unlikely gets us to 2200 ms. Because I 
want to make sure that the link is always utilized you must allow me to 
queue at least one packet. Thus, we need to add another 1300 ms for the 
queueing delay of one single packet.
=> a sum of 3500 ms.

But I don't mind because I really need that e-mail to be downloaded and 
it's not so bad after all. Even telnet is acceptable if I really need to 
get something done.

As I write this e-mail, I'm dialing in via a landline modem with 28.800 
bits/sec which should still be a commonly found link speed. Modems usually 
have a *one way* latency of 50 ms (depending on the manufacturer you can 
find much higher numbers!). Doing the same math gets you to a sum of about 
1500 ms. Now I mind even less because 28.8 kbit/sec is OK for e-mail, WWW, 
and telnet.

That's why I can't agree with what you wrote above: "delays of 500 
milliseconds or more will be considered practically useless"

By the way, it's not always the telcos and the modem vendors that are 
responsible for excessive queueing. Check out the BSD operating system. By 
default it happily allocates a 50 packet large output buffer for a PPP 
interface. Given a TCP receiver window of 8 or 16 KBytes, most of those 
bytes will sit in BSD's output buffer. And then we are talking about RTTs 
of several seconds! See, e.g., 
http://iceberg.cs.berkeley.edu/papers/Ludwig-Sigmetrics99/index.html

Yes, I know that there are per route metrics that BSD4.4 uses to avoid 
that. But those metrics are discarded after each PPP session (when the 
route goes down) so that does not help much.

So, you haven't convinced me that the last version of the rule-of-thumb is 
broken.

///Reiner