Subject: Re: pilc minutes for IETF 48 - Persistence

From: Vernon Schryver <[email-masked]>
Date: Fri, 6 Oct 2000 09:32:52 -0600 (MDT)

> From: Dr G Fairhurst <[email-masked]>

> I don't think we differ as much as you may think we do.

I suspect we differ more than either of us thought.

> Sorry, this debate is less than clear.  I was talking only about WINDOW size,
> in many link ARQ schemes there is a fixed modulus or window which
> limits the maximum number of packets in flight.  This may be determined by
> either amount of buffer memory or size of frame number field (before everyone
>shouts and says "but not all" - I know there are exceptions). The HDLC standard
> works this way - and so do nearly all variants of it, I have seen.

Yes, but if you have a whole window's worth of packets that cannot get
through, then the link is very sick.  There's little to talk about besides
radio things like hand-off that should be inivisible to TCP except possibly
for a bunch of lost segments.

If you have one stuck packet that is keeping the window from advancing
and so you cannot get any other data through the link, then either your
window is too small or you have broken hardware or software that cannot
be fixed with ARQ or anything else (i.e. my #3).  There is absolutely
nothing that can or should be done in this case except possibly to change
vendors in a forklift upgrade.

> ...
> > Moreover, by Dr. Fairhurst's assumptions, either
> >   1. no other packets for any TCP stream are flowing because the link
> >      is very sick.
> >   2. this particular packet will soon get through because the problem
> >      was temporary.
> >   3. something is very badly broken in the design or implementation of
> >       the link layer, because one particular packet cannot get through
> >       but others can.

> ...
> 1 is what I call an OUTAGE - we have text already which recommends what
> to do (hang on to the last few packets if you can and send them once you
> are able - not quite the same as recommending persistence!)
>
> 2 may happen - but I have seen cases where this can take a long time
> - particularly with a big packet and periodic interference.
>
> 3. This happens too... and can be bizarre.
>
>The non-persistent argument seems to help with 2,3.  At least it clears out all
> that
> old link state and any buffers tied up, so that things can resume orderly
> in the
> future.

No, in those three cases:
  1. there is nothing to do.  The link is broken and has been broken for
     at least a little while.  Saving packets in the hope that the link
     might return might be good, but is probably a waste of cycles and
     memory, and quite possibly worse.

     You MUST NOT keep any TCP or UDP packet for longer than the MSL.
     You will break things if you somehow keep a packet for a long time
     and then eventually regurgitate it.

  2. this is ordinary bit rot.  If you often find problems with big packets
     and periodic interference, then you have misdesigned the link layer
     packet sizes by making it too big.

  3. there is nothing that can be done in this case except to ship all of
    the hardware and software back to the factory, and possibly to fire
    the poor fellow who messed up the FEC or CRC computation, buffer
    chaining, or whatever.

> What I'm trying to find out is what we should recommend when we have a
> high (or moderately high) packet loss rate.
>
> If we have only one TCP flow, do we care?
> Probably not, as long the ARQ tries (a few attempts) or forever.  The
> latter may
> even be better. Good versions of HDLC could work fine with segmentation.
>
> If we have many flows - are there other issues?
> I can think of these two:
>
> 1) The ability to classify UDP form TCP seems very desirable (although
> difficult with tunnels/encapsulation and fundamentally problematic with IPSEC)

It's wise to not attempt things that you know are impossible.  Given the
fact that IPSec exists, the fact that sooner or later people will get
paranoid about their bits, and the fact that people are already trained
to be paranoid about their voices on radios by analog cellular phone
scanners, it is crazy (as in failing to accept reality) to do anything
but assume IPSec makes classifying impossible.

> 2) I suggest sharing with other TCP flows can become an issue - perhaps less
> so if you are end-system connected to a mobile net, but more so if you are
> connecting
> a remote LAN with many hosts over a wireless (or any RF) link. Several factors
> here:
>     per-flow state,
>     head-of-line block when buffer/window exceeded,
>     forwarding delay (out of order delivery may be desirable)

Out of order forwarding is not desirable, but it is always far more
desirable to get the bits through the link than to second guess the upper
layers tolerance to re-ordering.
All IP protocols and applications are absolutely required to tolerate
absolutely all possible re-orderings of IP packets within the MSL.  Any
delaying of later packets merely because earlier packets need link layer
retransmitting would be *WRONG*.  And again, there is the MSL limit
on how long you can delay (including retransmitting) any packet.

When your window or buffer is full, what can you do beside hs head-of-line
block and drop incoming packets?  Mentioning that as an alternative makes
as much sense as mentioning that when the system melts in a fire, it must
stop working.

Per-flow state might be handy, but why would you not start with the sorts
of things that are already being done or considered for other sorts
of routers?  For example, why not do a modified RED?

Vernon Schryver    [email-masked]

---

From: Dr G Fairhurst <[email-masked]>
Date: Mon, 16 Oct 2000 16:31:21 +0100

> I suspect we differ more than either of us thought.

I wrote this to encourage more debate, pleased you responded.

> for a bunch of lost segments.

No, I disagree. Not so sick, for some types of link, just unfortunate: It's
not just cellular mobile we are thinking of.

> vendors in a forklift upgrade.

Window is indeed too small (at least in relation to the problem).

A link reset may well fix the problem.  If the protocol engines at each end
reinitialise
(and hopefully log), then you can get them back in sync with no memory issues.

>      memory, and quite possibly worse.

I'm not convinced the link ARQ platform will always be doing useful work
if the link is not moving data - so I'd like to let people think they could
use the cycles to do something (if they choose to do so). By "worse",
I presume you mean MSL (good point, see below) - are there other bad things
I haven't thought of?

>      and then eventually regurgitate it.

good point. MSL is always to be observed. Incidentally this means that true
reliability (with unlimited retransmissions) is not allowed for IP!

>      packet sizes by making it too big.

Sure, just a badly configured link, or very unfortunate error pattern. Neither
are impossible, and can be found in the Internet.

>     the poor fellow who messed up the FEC or CRC computation, buffer

Or do a link reset, possibly drop some frames, and continue.

> but assume IPSec makes classifying impossible.

I agree unless the IPSEC user chooses to signal the transport class (by some means)

or inhibits IPSEC, then we can not assume an ability to classify the transport
protocol.

> on how long you can delay (including retransmitting) any packet.

MSL - agreed.  "desirable" - should be "may prove beneficial in some cases"

> as much sense as mentioning that when the system melts in a fire, it must

Can drop from front or back....

> of routers?  For example, why not do a modified RED?

Has anyone done this?

> Vernon Schryver    [email-masked]

Gorry Fairhurst

------------------------------
http://www.erg.abdn.ac.uk/users/gorry

---

From: Reiner Ludwig <[email-masked]>
Date: Wed, 18 Oct 2000 06:14:13 +0200

>reliability (with unlimited retransmissions) is not allowed for IP!

If you check the PILC archive, you will find that I always mentioned the 
MSL as an upper bound for link layer error recovery persistency. In fact, 
the maximum RTO of 64 seconds is a more reasonable upper bound.

///Reiner

---

From: Reiner Ludwig <[email-masked]>
Date: Wed, 18 Oct 2000 12:03:38 +0200

At 10:40 18.10.00, Dr G Fairhurst wrote:
>Yes, but asserting that MSL/TCP RTO should be considered an upper delay bound,
>aren't you saying you don't think a fully reliable link ARQ (aka HDLC) is a
>good idea?

I'm saying (again) that link layer error recovery (aka ARQ) as, e.g., 
implemented by HDLC, on lossy links or links that suffer transient outages 
is a great idea, *if* that link also implements flow seperation. The 
maximum time the link layer sender tries to re-/transmit a single TCP 
segment should, however, be bound to 64 sec. (TCP's max RTO).

>P.S. 64 secs will usually equate to a very large number of 
>retransmissions...or
>a
>significant link outage.

True.

///Reiner

---

From: Lloyd Wood <[email-masked]>
Date: Wed, 18 Oct 2000 11:38:46 +0100 (BST)

> the maximum RTO of 64 seconds is a more reasonable upper bound.

Reasonable for whom?

Having an upper bound much more than several times the link delay
makes little sense. An upper bound that is a significant fraction or
more of the path delay makes little sense. And a telnet session with a
lag of 64 seconds is something I can do quite happily without.

By increasing persistency you're not increasing link
utilisation. You're not helping TCP(-friendly) congestion control
algorithms. What are you doing, exactly?

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: Kamesh Medepalli <[email-masked]>
Date: Wed, 18 Oct 2000 11:25:58 -0400

Zoran,

Two questions:

A. What is the air link data rate in your case?

B. When you say higher persistency, are you also increasing the aggressiveness
of the link layer recovery? Meaning, if the 3 round recovery scheme
retransmits 1,2 and 3 times then a 5 round one retransmits 1,2,3,4 and 5 times?

Zoran Kostic wrote:
> 
> Lloyd,
> 
> This is an example of why and when it makes sense to
> have an upper delay more than several times the path delay.
> Also, an argument (from yet another individual) in favour
> of "higher" persistency.
> 
> In a cellular network with mobile users, you will often find
> a user whose performance is just fine most of the time.
> Occassionally, it will hit nasty channel conditions, get
> stuck for a short amount of time and get out of trouble
> after not so long. Well, we do not want to break the
> TCP connection while it is stuck, because we know that
> it will get out of trouble. And statisctically, this happens often
> (in the cellular environment)!
> 
> We've run an extensive set of simulations investigating
> interactions between the the cellular environment and TCP.
> Indeed, having very persistent lower layer gives the best results
> from the performance point of view. Having less than 3 or
> 4 retransmissions always screws up things if the channel is
> messy or if the system is highly loaded (many users per cell),
> and infinite persistency gets decently mimicked by
> something on the order of 5-10 retransmission on the RLP
> layer.
> 
> So, as an answer to your question
> "What are you doing, exactly?", I offer the following:
> In the cellular environment, with large number of users,
> cumulative distribution functions expressing system throughput
> and delay show significant improvements if the persistency on
> the lower layers is high. So, we are doing a great deal here !
> 
> Yes, we want persistency. It is not clear that we need to have
> infinite persistency or always as high as 64 seconds, but having
> only 3 retransmission (as in IS99) is suboptimal.
> I actually think that variable persistency (as a function of
> cellular channel conditions) may make sense.
> 
> Zoran

---

From: Zoran Kostic <[email-masked]>
Date: Wed, 18 Oct 2000 10:00:32 -0400

Lloyd,

Zoran

---

From: Zoran Kostic <[email-masked]>
Date: Wed, 18 Oct 2000 14:42:27 -0400

Kamesh,

  * The airlink is the one used in EDGE - thereby there is variability
     of the rate between roughly 9 and 59 kbps.

  * The way I think of persistency on the link is the following: Attempt
     to transmit the block no more than X times. So, I do not have the
     concept of rounds, just plain number of  transmission attempts.

     On this layer I also use the mode 0 option, whereby if the channel
     or interference are really bad, I do not even try to retransmit.

   Zoran

---

From: Kamesh Medepalli <[email-masked]>
Date: Wed, 18 Oct 2000 15:18:16 -0400

Zoran,

 The reason I asked those questions is for this:

 1. It could be possible that for the TCP window size, MSS, RTT and error
    rate, the max throughput that TCP can achieve is greater than 59Kb/s.
    In that case, you *can* get away by spending more time at the link layer.
    This is because there is some room in the latency before TCP "feels" it.

2. If that is not the case, then it is good that the *extra*
   latency in recovering errored data has indeed paid off. However, there
   clearly needs to be a limit on the latency you are adding. TCP may just
   time out too.

3. Regarding the ARQ scheme itself, clearly the residual error decreases with
   increased retransmissions. Given a fixed number of retransmission, say 5,
   you could choose to retransmit, for e.g. in two ways:
   a. For every NAK the receiver sends retransmit and keep doing it for 
      5 NAKs. If after 5 attempts you still can't recover data, then you 
      leave it to TCP for recovery.
   b. The other option could be once you receive a NAK, send the retrans.
      twice and if you still receive another NAK, send the retransmission 
      3 times. If you still can't recover, then leave it
      up to TCP for recovery.
  The performance of these schemes depends on the TCP window size, RTT, MSS, 
  and ofcourse physical channel [correlated errors or uncorrelated errors]. 
  It is interesting how these many parameters come together to give a 
  sensible result in the end! We have summarized these results in our 
  VTC-2000 Spring  paper.
  
  Infact, as a result of these findings we left the recovery scheme at link
  layer [for CDMA-2000] up to the vendors and the telco operators. We did, 
  however, provide a default scheme in the RLP [Radio Link Protocol] 
  standard.

Kamesh

---

From: Reiner Ludwig <[email-masked]>
Date: Thu, 19 Oct 2000 06:22:08 +0200

>      concept of rounds, just plain number of  transmission attempts.

Since (wireless) links have so different properties (frame size, link 
speed, latency, LL-RTT, ....), I suggest to speak of LL-ARQ persistency in 
terms of max. delay an IP packet may incur due to LL-ARQ before it is dropped.

Usually, it's straight forward to translate from "transmit the block no 
more than X times" to the mentioned delay.

IMO, that will make it much easier to talk about these things without 
requiring detailed knowledge of the link in question.

///Reiner

---

From: Kamesh Medepalli <[email-masked]>
Date: Thu, 19 Oct 2000 08:49:45 -0400

> requiring detailed knowledge of the link in question.

I agree. I was just making a simple comment that within the latency bound 
there are different optimal speeds for different network level and wireless
link level scenarios.

Kamesh