Subject: FW: Some comments to the draft-ietf-pilc-error-00.txt

From: Nitin Vaidya <[email-masked]>
Date: Wed, 23 Jun 1999 11:31:33 -0700

> -----Original Message-----
      > From: Lloyd Wood [mailto:[email-masked]]
 
      > On Wed, 23 Jun 1999, Nitin Vaidya wrote:
      > 
      > >       > From: Lloyd Wood [mailto:[email-masked]]
      > >       ...
      > >       > What may be appropriate for a single link for 
      > >       > communication between
      > >       > its two dumb endpoints in isolation can be extremely 
      > >       > inappropriate for
      > >       > an internetwork or higher layers utilising 
      > that link, where
      > >       > the functionality is duplicated.
      > > 
      > > To belabor the point ...
      > > 
      > > I agree that ARQ "can" be inappropriate (as you state 
      > above), but it
      > > is also important to note that it is *not always* 
      > inappropriate.
      > 
      > Feel free to cite real examples where local ARQ is 
      > appropriate to use
      > in a TCP/IP internetwork, then, and not clearly bettered by
      > alternatives. ARQ is generally introduced by people who don't
      > appreciate the benefits of appropriate coding but think 
      > their link has
      > to have something to make it Look Good.

Sure. See the papers cited by Reiner in one of his earlier emails
(for instance, paper by Steenkiste @ CMU).

      > >       > A PILC-generated draft explaining why ARQ is bad would
      > >       > IMO be a laudable goal. The current errors 
      > draft doesn't discuss
      > >       > FEC or ARQ at all, and:
      > > 
      > > A draft looking at FEC and ARQ is a good idea,
      > 
      > yes.
      > 
      > > however, starting with
      > > the presumption that "ARQ is bad" is a bad idea 
      > 
      > Well, it's an entirely correct application of 
      > ramifications of the
      > end-to-end principle. "ARQ is bad" is Blindingly Obvious as the
      > conclusion to be reached, given interactions with 
      > TCP/IP illustrated
      > by Perry's example etc.

Sorry to disagree here. (By the way, by local ARQ I do not imply a link
that provides 100% reliability, but limited number of retransmits, in
general).

Would need to see more data to be convinced of the claim
that "ARQ is bad", beyond the argument that "end-to-end" principle 
says so (it is a principle, not a theorem).

      > > - a more balanced draft
      > > will be more useful. Local FEC, local ARQ, end-to-end 
      > ARQ (and, for that
      > > matter, end-to-end FEC) are all useful techniques. 
      > Needless to say,
      > > not every technique is applicable in all situations.
      > 
      > end-to-end ARQ? end-to-end FEC?

In my jargon, TCP retransmits = "end-to-end ARQ". An example of "end-to-end
FEC" is the use of FEC for multicasting. In this multicast scheme,
the multicast sender sends a "checksum" packet, which could be used
by the recipients to recover from a packet loss.

      > Reality check, please. I can't think of a single case 
      > *in a TCP/IP
      > internetwork* where use of these would be at all 
      > appropriate, and I
      > can't see the draft covering other situations.

Hope above clarification of the jargon helps.

- nitin

      > 
      > L.
      > 
      > 
      > > Enough said.
      > > 
      > > - nitin
      > > 
      > > Disclaimer: Speaking for myself ...
      > 
      > <[email-masked]>PGP<http://www.ee.surrey.ac.uk/Pers
      > onal/L.Wood/>
      >

---

From: "Gary Kenward" <[email-masked]>
Date: Wed, 23 Jun 1999 12:59:18 -0500

I've only been following this discussion intermittently, so I hope I'm not
repeating something already stated. 

The issue, IMHO, is that FEC and ARQ in some environments, complement each
other.
In the wireless data world, ARQ is often required to compensate for
situations 
where one or more transmitted packets are completely lost (e.g. the link
layer header
is unrecoverable).

Why not perform ARQ at the link layer? Well, for reasons already given,
because
of the delay. It is hard enough to provide low delay across a short wireless
hop. 
Regardless of how well you design the ARQ scheme, you always end up having
to
send the ACK (or NAK), and the retransmission across all the hops
end-to-end, not 
just the hop where the packet loss occurred.

Next, I am very sceptical that there is a "one-size-fits-all" solution for
all types
of links. Certainly in the wireless world, the different RF links have
different 
characteristics. Yes, there are similarities, but usually the degree of a
certain
type of impairment is different. For example, comparing mobile cellular
versus
wireless LAN: (in general terms) one is predominated by burst errors, while
the 
other is dominated by highly variant coverage patterns; both are caused by
similar
physical phenomena, but the impairments manifest themselves very differently
at the digital signal level.

One possible technology for a "one-size-fits-all" solution that could be
adaptive 
not only to different media, but also to time varying channel impairments,
are
hybrid FEC/ARQ techniques. In particular, the so-called Type II hybrid error
control. However, this technique adapts to local characteristics, not to a
chain
of links with different characteristics, and as far as I know, it is best
applied
at the link level.

(In Type II hybrid EC, the FEC on the first transmission is designed for
best
case conditions, and each retransmission adds to the error correcting power.
For example: the first transmission has no FEC, the second transmission has
the
associated parity bits for FEC. There are many schemes, each with trade-offs
in error robustness versus delay versus overhead).

Just some thoughts...
Gary Kenward

> From:	Lloyd Wood [SMTP:[email-masked]]
> Sent:	Wednesday, June 23, 1999 12:50 PM
> To:	Nitin Vaidya
> Cc:	pilc
> Subject:	RE: Some comments to the draft-ietf-pilc-error-00.txt
> 
> On Tue, 22 Jun 1999, Nitin Vaidya wrote:
> 
> >   > Local ARQ duplicates TCP checksum functionality and is against
> >   > end-to-end, as I point out above. TCP already does the checking; the
> >   > added ARQ delay messes up TCP's RTT estimates, in any case.
> > 
> > This may be true, but not necessarily harmful, if link level (local)
> > ARQ delay is not large relative to end-to-end delay.
> 
> ...and hey, finer-grained measurement and RTO handling are never going
> to be deployed anyway, so messing up measurement where such messing
> can be avoided must therefore be acceptable? Sorry, can't agree.
> 
> 
> >   > FEC and ARQ are orthogonal; TCP effectively duplicates ARQ
> >   > functionality and makes ARQ redundant, but it doesn't duplicate FEC
> >   > functionality. It can't. TCP can check the integrity of what is
> >   > delivered by the transmission process, but it can't improve on it.
> > 
> > Why is ARQ at link layer redundant? Think of ARQ as a form of
> > adaptive FEC - both make delays variable.
> 
> Adaptive FEC adds rather less coding/decoding delay variation than
> multiple mini roundtrips do. Yes, ARQ on the local link _can_ allow
> the link to achieve the channel capacity of a binary erasure channel -
> if infinite delay is allowed for ARQ to work, of course, which rather
> messes up TCP delay measurement.
> 
> ARQ's granularity compared to coding is coarse, to say the least;
> coding is IMO by far the better per-link solution.
> 
>  
> > So long as delay variations are not too large, no
> > harm in using ARQ at link layer.
> 
> Well, so much for achieving that channel capacity, then. Limiting
> delay variation means limiting local roundtrips means limiting ARQ
> resends, so then what's the point of ARQ if it never gets to perform
> well? Severely errored link conditions then degrade link performance
> badly as a result of limited permitted retries. Bad in itself, but the
> knock-on system effects are worse...
> 
> Please see Perry's recent end2end email, appended below, for a 
> clear illustration of why use of ARQ is misguided, and how
> ARQ use messes up end-to-end communication.
> 
> The resulting delays are a product of the ARQ process; limiting ARQ to
> prevent 'too large' variations just adds more complexity while
> limiting the intended benefits of the ARQ approach. You'd be
> better off junking ARQ completely as inappropriate to the overall
> system - which is what application of the end-to-end principle tells
> us anyway.
> 
> 
> > Just make sure that the local ARQ does not make packet delivery "too" 
> > out-of-order.
> 
> That's another bizarre compromise/specification/requirement in itself,
> IMO, crossing layers to boot. All these 'so long as' and 'just make
> sure that' are intended to ameliorate the fundamental mismatch between
> ARQ and end-to-end requirements, when clean design involves
> avoiding the complication of ARQ in the first place.
> 
> What may be appropriate for a single link for communication between
> its two dumb endpoints in isolation can be extremely inappropriate for
> an internetwork or higher layers utilising that link, where
> ARQ can really get in the way when you have asymmetric/variable
> routing/topology, too; local ARQ'd resends of routing updates strike
> me as being particularly unhelpful. ARQ and asymmetry don't mix.
> 
> IMO be a laudable goal. The current errors draft doesn't discuss
> FEC or ARQ at all, and:
> 
> http://www.ietf.org/internet-drafts/draft-ietf-pilc-error-00.txt
> 
>  2.4 Delayed Duplicate Acknowlegements [MV97, VMPM99]
>  [..]
>    It is preferrable [sic] to allow a local mechanism
>    to resolve a local problem, instead of invoking TCP's end-to-end
>    mechanism and incurring the associated costs, both in terms of
>    wasted bandwidth and in terms of its effect on TCP's window
>    behavior.
> 
> needs careful qualification in terms of the local mechanisms that can
> be safely applied without impacting TCP's behaviour, and the scope of
> the problems that can be resolved; distinguishing between errors as
> part of the link characteristic and unusual errors symptomatic of a
> local problem that can be resolved is important. Or just reference
> that not-yet-written draft...
> 
> cheers,
> 
> L.
> 
> the less said about various bizarre FEC/ARQ combinations, the better.
> 
> 
> > - nitin
> > ----
> > Disclaimer: Speaking for myself, not my employer.
> 
> <[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>
> 
> Date: Wed, 23 Jun 1999 08:52:02 -0400 (EDT)
> From: Perry E. Metzger <[email-masked]>
> To: [email-masked]
> Subject: When non TCP/IP literate people design cable modem systems
>  
> Some folks at Unnamed Cablemodem Provider have a lovely cable modem
> system that rarely drops packets, but which does appear to have
> astoundingly large queues and reliable retransmits in the data link
> layer. I'm sure they thought this was a Great Idea. I provide a sample
> unretouched ping sample below of what using this system is like.
>  
> (my personal favorite was
> 64 bytes from 206.1.51.15: icmp_seq=243 ttl=246 time=212681.418 ms
> but I've heard stories of 300,000ms and more. The losses appear to
> occur only later in the network path, not within the provider's net):
>  
> 64 bytes from XXX.X.XX.XX: icmp_seq=213 ttl=246 time=167892.549 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=214 ttl=246 time=166909.097 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=215 ttl=246 time=165921.709 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=217 ttl=246 time=166285.507 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=218 ttl=246 time=165302.197 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=219 ttl=246 time=164312.156 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=220 ttl=246 time=163328.912 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=221 ttl=246 time=162344.836 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=222 ttl=246 time=161355.764 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=223 ttl=246 time=174627.732 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=224 ttl=246 time=173649.189 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=225 ttl=246 time=172664.074 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=226 ttl=246 time=178160.748 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=227 ttl=246 time=177176.078 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=228 ttl=246 time=176190.470 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=230 ttl=246 time=174203.410 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=231 ttl=246 time=175363.403 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=232 ttl=246 time=174380.950 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=233 ttl=246 time=173398.209 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=234 ttl=246 time=172413.667 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=239 ttl=246 time=196856.713 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=240 ttl=246 time=195878.520 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=241 ttl=246 time=194906.446 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=242 ttl=246 time=193907.888 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=243 ttl=246 time=212681.418 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=245 ttl=246 time=210696.110 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=246 ttl=246 time=209721.004 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=247 ttl=246 time=208735.515 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=248 ttl=246 time=207753.737 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=249 ttl=246 time=206764.453 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=250 ttl=246 time=205798.128 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=251 ttl=246 time=204803.277 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=252 ttl=246 time=203810.780 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=253 ttl=246 time=202825.129 ms
> 64 bytes from XXX.X.XX.XX: icmp_seq=258 ttl=246 time=197842.556 ms
> 
> 
> 
> 
> 
> 
> 
>

---

From: Lloyd Wood <[email-masked]>
Date: Wed, 23 Jun 1999 21:55:57 +0100 (BST)

[Since this is related to Perry's cable modem observations in:

 Re: When non TCP/IP literate people design cable modem systems

 I'm cc'ing end2end for viewpoints on the utility of using per-link ARQ,
 as opposed to per-link adaptive FEC, with TCP/IP. What would _you_
 recommend if you were designing a link which TCP/IP traffic
 would travel over? And why? I thought the answer was obvious, but...]

On Wed, 23 Jun 1999, Nitin Vaidya wrote:

>       > >       > communication between its two dumb endpoints in
>       > >       > isolation can be extremely inappropriate for
>       > >       > that link, where the functionality is duplicated.
>       > > 
>       > > To belabor the point ...
>       > > 
>       > > above), but it is also important to note that it is *not
>       > > always* inappropriate.
>       > 
>       > Feel free to cite real examples where local ARQ is appropriate
>       > to use in a TCP/IP internetwork, then, and not clearly bettered
>       > by alternatives. ARQ is generally introduced by people who
>       > don't appreciate the benefits of appropriate coding but
>       > think their link has to have something to make it Look Good.
> 

Okay. Reiner cited, and you recommended I read:
  http://www.cs.cmu.edu/~prs/papers.html
  David Eckhardt and Peter Steenkiste, Improving Wireless LAN
  Performance via Adaptive Local Error Control, Sixth IEEE
  International Conference on Network Protocols (ICNP'98), Austin,
  October 1998, postscript. 

I've just read it. And from that, I'd like to point out:

  Abstract
  [..]
  A significant challenge is ensuring that systems with multiple
  layers or error control avoid compromising performance by
  duplication of effort.
  [..]
  5.3.3 Persistence of local error control
  [..]
  A potential drawback of persistent local retransmission is that it
  may delay packets significantly. This may result in interference
  with TCP retransmission [two references].
  [..]
  We conclude that retransmitting overly persistently, such as the
  perpetual retransmission implemented in our simple LLC, can hurt 
  performance via inter-layer duplication. However, giving up after 
  a few transmissions has the potential to seriously compromise TCP
  performance

It seems pretty clear to me from this that Eckhardt and Steenkiste
verified that local ARQ retransmissions are a Bad Idea.

OTOH, they really really _want_ to make local retransmissions work
despite their own evidence to the contrary, and they're investigating
retransmission a lot more than they're investigating adaptive FEC, as:

   7.3 Summary

   The above results suggest that FEC coding can build on top of local
   retransmission to further increase usable bandwidth in interference
   and attentuation environments.

suggests. Backwards and flying in the face of their own evidence as
far as end-to-end use is concerned, IMO; it appears they're focusing
on a local optimisation they're keen on, relying on the granularity of
TCP timers to avoid timeouts (that's not what the timer is for...),
and neglecting ramifications of resulting knock-on system effects.

That's not how you build a high-performance internetwork... and that's
your _real example_?

>       > >       > IMO be a laudable goal. The current errors draft
>       > >       > doesn't discuss FEC or ARQ at all, and:
>       > > 
>       > Well, it's an entirely correct application of ramifications of the
>       > TCP/IP illustrated by Perry's example etc.
> 

Please see the excerpts from that paper you recommended I read, above.

A limited number of local ARQ retransmits provides no benefits at all.
ARQ on the local link can only allow the link to achieve the channel
capacity of a binary erasure channel if infinite delay is allowed for
ARQ to work. You're proposing kneecapping ARQ in this respect, instead
of simply removing ARQ as superfluous at best and as damaging to
overall internetwork performance the rest of the time.

If you're going to kneecap ARQ like this, you're better off accepting
the overhead of adaptive FEC.

> says so (it is a principle, not a theorem).

Please see the excerpts from that paper you recommended I read, above.

(How big a network can you build if you're only using theorems?)

>       > > - a more balanced draft will be more useful. Local FEC,
>       > > local ARQ, end-to-end ARQ (and, for that
>       > > Needless to say, not every technique is applicable in all
>       > > situations.

That's an understatement.

> In my jargon, TCP retransmits = "end-to-end ARQ".

ARQ relies on _negative_ acknowledgments sent only when errors are
detected. Go-back-N sliding windows with positive acknowledgement is
not automatic repeat request. TCP retransmits are not ARQ. (Collision
detection on ethernet is not ARQ either, incidentally.)

> An example of "end-to-end

And that's not forward error control coding on a link.
Not by a _long_ way. It's not even remotely analogous.

L.

has anyone investigated the impact of local link ARQ on
topologies with asymmetric routing? FEC on the link should be better.
Much better.

>       > *in a TCP/IP internetwork* where use of these would be at
>       > all appropriate, and I can't see the draft covering other
>       > situations.
>

---

From: Reiner Ludwig <[email-masked]>
Date: Wed, 23 Jun 1999 14:55:45 -0700

>just the hop where the packet loss occurred.

What? 

Are we talking about link layer ARQ? 

What do you mean with "you always end up having to send the ACK (or NAK),
and the retransmission across all the hops end-to-end, not just the hop
where the packet loss occurred."?

///Reiner