Subject: Mostly NiTs on link-id (-07 from IETF archive)

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 11 Dec 2001 13:35:28 +0000

While awakening several hours too early, after failing to adapt to US time
zones, I read the link-id again.

These comments refer to the ID revision 07 as held in the IETF archives.
Comments are mainly NiTs, and some consistency issues between pilc drafts.

Gorry Fairhurst

---------

In the section "Bandwidth on Demand (BoD) Subnets"

The ID says:
   "The resource shared may be a terrestrial wireless hop, a
   satellite uplink, or an end-to-end satellite channel."

These techniques are also used in other places than those listed. This
statement should be more inclusive. I suggest, for example:

   "Examples of the resource that may be shared include a terrestrial
    wireless hop, a cable modem uplink, a satellite uplink, and an
    end-to-end satellite channel."
    
---------

In the section "Recovery from Subnetwork Outages"

Para 6 of the ID says
   " Only a single packet per TCP connection, including ACKs, need be held
   in this way to cause the TCP sender to recover from the additional
   losses once the flow resumes.  [ARQ-DRAFT]"

Reference should come before the full stop.
 
---------

In the section "Recovery from Subnetwork Outages"

Para 7 of the ID says

 " Because it would be a layering violation (and possibly a performance
   hit) for IP or a subnetwork layer to look at TCP headers (which would
   in any event be impossible if IPSEC [RFC2401] encryption is in use),

*** But, still not sure about this - In [arq-id], section 3.3, we discuss
flow separation from the ARQ viewpoint and the practical difficulties in
doing this. Some of this surely applies here, in particular how is a network
device to identify TCP flows when IPSEC ESP is being used?
Life gets more tricky when the link includes tunnels/MPLS/etc. etc.

This problem issue should at least be highlighted somewhere!

It continues:

 ..."it would be reasonable for the IP or subnetwork layers to choose, as
   a design parameter, some small number of packets that it will retain
   during an outage."

The above sentence referes to one or more layers. Suggest:

 ..." it would be reasonable for the IP or subnetwork layers to choose, as
   a design parameter, some small number of packets that will be retained
   during an outage."

---------

In the section " CRCs, Checksums and Error Detection"

The final paragraph on IPv6, may be better placed as the third paragraph of
the section, since it follows logically the thoughts in paras 1,2. The rest
of the section seems to talk about link CRCs, which applies to IPv6 and
IPv4.

---------

In the section "CRCs, Checksums and Error Detection"

The paragraph below is possibly misleading:
   "For link layers that can differentiate between flows, it may be
   appropriate to reduce the error detection level for certain flows
   with large numbers of small packets, such as voice flows.  As such
   flows also benefit significantly from header compression, this should
   only be combined with a header compression scheme that is robust
   against residual bit errors [RFC3095]."

The requirement is not that the link layer is able to "differentiate between
flows" which simply means separation of the IP (or TCP) flows - something
which again, is not always possible or easy in the general Internet - it
also assumes that the link is able to identify the ***actual***
transport/application requirements. Reducing the CRC coverage / power should
only be done with the knowledge that the ***application*** can correctly
utilise corrupted information, doing this with arbitary application
protocols carries a serious danger that the link may deliver data with
errors which usually remain undetected by the application and hence can
corrupt the user data. This is evil. On the other hand, it can be good
if you have a suitable audio/video format and codec that's able to
accommodate loss using "erasure" techniques.

This paragraph needs to be changed.

---------

In the section "How TCP Works"

Paragraph 4:
   "TCP uses sequence numbering and acknowledgments (ACKs) on an end-to-
   end basis to provide reliable, sequenced, once-only delivery.  TCP
   ACKs are cumulative, i.e., each implicitly ACKs every segment
   received so far.  If a packet is lost, the cumulative ACK will cease
   to advance."

The last sentence isn't too clear. Slightly better:

"If an ACK is not received, the acknowledgment value carried in the
cumulative packet will cease to advance"

---------

The section "Delay Characteristics"

Says some important things, but also partly re-iterates what is said
elsewhere in the document on the impact of delay and also in the [arq-id].
It may be better to re-edit this or place it near the start? at the end?
of the document.

---------

The section "Bandwidth Asymmetries"

Since ASYM is now a working group document, which has been agreed, I think
you really should replace "[BPK98]" with a reference to the asym-id.  The
text in the link-id now pre-dates the text in the asym-id, which now talks
about much more than simple differences in link bandwidth and the few
mitigations cited in the link-id.

The ID says:
 "Some subnetworks may provide asymmetric bandwidth and the Internet
   protocol suite will generally still work fine."

Yes, but asymmetry may also arise because the forward and return paths used
by a TCP connection use different subnetworks with different available
capacities as well. This may be better:

 "Some subnetworks may provide asymmetric bandwidth (or may cause TCP
    packet flows to be experience asymmetry in the capacity) and the
    Internet protocol suite will generally still work fine."

I also dislike the work "bandwidth" here, because it's not bandwidth that
matters, but available capacity. Bandwidth asymmetry may lead to asymmetric
capacity.

The ID says:
   "Therefore, when the ratio of the bandwidth of the subnetwork carrying
   the data to the bandwidth of the subnetwork carrying the
   acknowledgments is too large, the slow return of of the ACKs directly
   impacts performance. "

Suggest you replace "bandwidth" by "available capacity" as below:

   "Since ACKs are generally smaller than data
   segments, TCP can tolerate some asymmetry, but as a general rule
   designers of subnetworks should avoid large differences in the
   incoming and outgoing bandwidth."

Is this really the correct recommendation? - I don't agree.

The phrase "should avoid" seems very strong, the asym-id gives many examples
of situations other engineering trade-offs result in high asymmetry.

I would suggest the best advice is something like:
   "Since ACKs are generally smaller than data
   segments, TCP can tolerate some asymmetry, but as a general rule
   designers of subnetworks should be aware that subnetworks with
   significant asymmetry can result in reduced performance, unless
   issues are taken to mitigate this [asym-id]."

The ID says:
   "One way to cope with asymmetric subnetworks is to increase the size
   of the data segments as much as possible.  This allows more data to
   be sent per ACK, mitigating the slow flow of ACKs.  Using the delayed
   acknowledgment mechanism [Bra89] that reduces the number of ACKs
   transmitted by the receiver by roughly half can also improve
   performance by reducing the congestion on the ACK channel.  These
   mechanisms should be employed in asymmetric networks."

The above paragraph is not advice to a link designer!

Anyway, this is one of a number of ***mitigations***, it only works in some
cases, it doesn't always help much - see [asym-id].

Can we delete the above paragraph?

The ID says:
   "Several researchers have introduced strategies for coping with
   bandwidth asymmetry.  These mechanisms generally attempt to reduce
   the number of ACKs being transmitted over the low bandwidth channel
   by limiting the ACK frequency or filtering out ACKs at an
   intermediate router [BPK98].  While these solutions mitigate the
   performance problems caused by asymmetric subnetworks, they do have
   some cost. Therefore, as suggested above, bandwidth asymmetry should
   be minimized in subnetwork designs."

This isn't in agreement with the discussion in the [asym-id]. Suggest you
replace this paragraph with:

   "Several strategies have been identified for reducing the impact of
   asymmetry of the network path between two TCP end hosts, e.g. [asym-id].
   These techniques attempt to reduce the number of ACKs transmitted over
   the return path (low bandwidth channel) by changes at the end host(s),
   and/or by modification of subnetwork packet forwarding. While
   these solutions may mitigate the performance issues caused by asymmetric
   subnetworks, they do have associated cost and may have other
   implications. A fuller discussion of strategies and their implications
   is provided in [asym-id]"

---------

In the section "Buffering, flow & congestion control"

Para 4 of the ID ends by saying
   "as a rule of thumb, each node should have
   enough buffering to hold one bandwidth*delay product's worth of data
   for each TCP connection sharing the link."

Is bandwidth - is this "link bandwidth"?
Is delay also "link delay" - or is it "path delay"?
- Can we make this clear?

---------

In the section "Multicasting"

There is no statement about IPv6, requiring multicast support at the network
layer - Thus is needed even if multicast is not natively supported by a
subnetwork.

---------

In the section "Broadcasting and Discovery"

There is no statement about IPv6!!!!!
 
---------

In the section "Security Considerations"

Paragraph 5 "The majority of the working group held that" --- I think
someone's already said something, but this is not necessarily true or useful
to say!!!

-----

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Dec 2001 13:16:45 -0800

> zones, I read the link-id again.
...

> There is no statement about IPv6!!!!!

Agreed. My initial impression is that IPv6 asserts that it is not

subject to the broadcast requirement, since it uses other, somewhat
undefined (anycast) mechanisms for discovery. Anycast, IMO, is
undefined because it requies multicast which typically requires broadcast.

It's a chicken-and-egg issue - as you noted above, IPv6 requires
multicast, but multicast typically requires broadcast on subnets,
yet the assertion appears to be that IPv6 doesn't need broadcast
since it uses multicast instead.

Anyone have a better perspective on this? (I can coordinate updates to
those sections...)

Joe

---

From: Dan Grossman <[email-masked]>
Date: Wed, 12 Dec 2001 22:34:43 -0500

Comments inline:

>     end-to-end satellite channel."

pen
Well, if we define BoD that broadly, then there are number of places in the
section that need to be rewritten to take into account cable and terrestrial
wireless.   Rereading the section, it appears that the principle holds for these
thing, although since propagation delays are not as large, the performance
issues are not as great.  That being the case, a little wordsmithing would
generalize.

Incidentally, terminology check:  a "data over cable television upstream
channel."

> Life gets more tricky when the link includes tunnels/MPLS/etc. etc

We also discussed this in the section on reliability and error control.  Maybe
best to use the same words:  "when a link layer can identify individual
flows...." and point to the ARQ draft.

> .

The point is that there has to be some positive way for the subnetwork to find
out about application requirements in trading off delay jitter/packet loss for
errored packets.  There are several ways to do this, such as signalling (RSVP),
DSCPs, local knowledge if the link in quesiton is connected to the host.

Incidentally, we should also point out  that the IP header checksum is weak, and
that indiscriminate forwarding of known errored packets runs a risk that
undetected errors in the IP packet header will cause misdelivery.

> matters, but available capacity. Bandwidth asymmetry may lead to asymmetric

Unfortunately, misuse of "bandwidth" to mean "available capacity" is all too
common.  I agree that as a technical document, we should use technically correct
terms.

> of situations other engineering trade-offs result in high asymmetry.

Strongly agree.

> Can we delete the above paragraph?

We might want to mention delayed ACKs in previous paragraph, if we do this.

> \

We had this discussion about two years ago.  The conclusion was that it was path
delay.  Which of course the link has absolutely no way to know.  Mark (I
beleive) suggested that link delay might be an approximation.  I think that we
left it ambiguous because the ambiguities spared us having to deal with a lot of
internal inconsistencies in the architecture.  But we should try again.

> someone's already said something, but this is not necessarily true or useful

Thank you. [please infer gracious tone of voice here]

---

From: "Marie-Jose Montpetit" <[email-masked]>
Date: Thu, 13 Dec 2001 06:07:17 -0500

snip/snip>
> Incidentally, terminology check:  a "data over cable television upstream
If we really want to extend BoD to all potential channels then even the
wireless LANs would apply (look at 802.11e; you could define BoD over it).
Do we want to just say the the technique applies to any shared media that
has a framing structure?

Marie-Jose

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 13 Dec 2001 09:44:27 +0000

> Do we want to just say the the technique applies to any shared media that

My main comment was that the list should not be exhaustive. It **may** apply
to a wide variety of channels. I would like the ID to have some examples, my
chosen ones were suggested to coincide with those with issues described in
ASYM-ID, there could be more or fewer.

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 13 Dec 2001 10:08:35 +0000

----------

>Gorry:

I didn't mean it as a replacement text, more as guidance to the editor,
I would prefer to see much less description, I wrote this to help explain
the problem.

> Dan Grossman:
> errored packets.  There are several ways to do this, such as signalling
> (RSVP), DSCPs, local knowledge if the link in quesiton is connected to the
> host.

Well, these QoS signals don't (necessarily) tell the link layer sufficient
information about the trade-offs it can possibly make in terms of delay -
versus  jitter - versus loss. This is a tricky area.

The section in the link-ID is about partial coverage checksums (or CRCs)
which do not cover the entire L3 datagram.

Specifically, my comment is that the IETF does not seem to have a framework
for partial checksums yet (DCP mentions this, as does ROHC WG for voice-only
services). However we should be explicit that this requires a clear
KNOWLEDGE that the end host applications can accept this type of error, and
that network/transport headers MUST be covered. The DEFAULT behaviour in the
absence of a concrete application requirement is that the link CRC MUST
cover the entire L3 datagram.

----------

> Incidentally, we should also point out  that the IP header checksum is
> weak, and that indiscriminate forwarding of known errored packets runs
> a risk that undetected errors in the IP packet header will cause misdelivery.

Yes, this is  related to this section, but I'd suggest for PARTIAL coverage
- I think we should mandate - i.e. say CRC MUST cover the network layer
header - for IPv6 this must be the case anyway!

Gorry

---

From: Dan Grossman <[email-masked]>
Date: Fri, 14 Dec 2001 10:17:48 -0500

> Do we want to just say the the technique applies to any shared media that
The common feature in question is not a framing structure (it does not exist 
for TDMA, for example), but rather a contention/grant/reservation scheme.   

If we accept this as the defining feature of BoD, 802.11 in PCF mode (which I 
believe is the basis for 802.11e, although I haven't been following it 
closely) is not a bandwidth-on-demand scheme, since it uses polling.

BTW, the term "Bandwidth-on-Demand is badly overloaded.  Can we come up with a 
better one?  Perhaps Contention/Reservation would work.

Also, the draft is pretty negative toward BoD.  I hope that Mark will tone 
that down a bit.
> Marie-Jose
Dan

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 14 Dec 2001 16:13:34 +0000

See comments at the end.

> Also, the draft is pretty negative toward BoD.  I hope that Mark will tone

We do need to get the definition of the term correct. I'm not sure this is
the same set of issues as in ASYM though where we didn't particularly
differentiate between types of reservation.

Can I suggest two extremes?

Reservation - where a fixed capacity is assigned for a fixed (possibly much
less than a session) time period. This is a rate-based assignment, and can
look very much like a "circuit switched subnetwork" to the IP traffic.

BoD - To me, I tend to think of the assigned capacity varies on a time frame
of the order of the link RTT.  Capacity may be based on measurement of
the current transmit queue size and/or arrival rate and/or a traffic
predictor. The short-term, or *DYNAMIC* element is the key bit.

However, I accept there are a variety of uses of the term,  and real systems
are often hybrids of various techniques, so we *MUST* be careful of wording
here so that the "audience" of the RFC understands.

Given what you are saying, I suggest we drop the DAMA and TDMA references,
or at least make this an example.

How do the 3GPP people understand the term BoD?

I like "BoD" as a section heading, because it's widely used, but it is
imprecise. It may encourage people to read the section!

Gorry

---

From: Markku Kojo <[email-masked]>
Date: Fri, 14 Dec 2001 21:08:18 +0200 (EET)

Hi,

I have a few comments/suggestions as well.

On Tue, 11 Dec 2001, Gorry Fairhurst wrote:

[snip]

> This problem issue should at least be highlighted somewhere!

The first one relates to some extent to this comment.

Current text in I-D section "TCP vs Link-Layer Retransmission":

  "However, when a link layer can identify individual flows and apply
   ARQ selectively [ARQ-DRAFT], then the link ARQ persistency should be
   high for a flow using reliable unicast transport protocols (e.g.,
   TCP) and must be low for all other flows.  Setting the link ARQ
   persistency larger than the largest link outage allows TCP to rapidly
   restore transmission without the need to wait for a retransmission
   time out. This generally improves TCP performance in the face of
   transient outages.  However, excessively high persistence may be
   disadvantageous; a practical upper limit of 30-60 seconds may be
   desirable. Implementation of such schemes remains a research issue.
   (See also Section "Recovery from Subnetwork Outages")."

As it is not trivial to get this implemented correctly in general
case and because there are caveats in doing this, the text should
stress it more strongly and say that [ARQ-DRAFT] discusses these
in more detail.

In particular, if the link layer identifies individual flows and
applies high ARQ persistency, it must also implement some form of
discard mechanism. Otherwise, a single TCP connection could
fill up its link queue and end up accumulating packes at IP layer,
effectively delaying any competing delay-sensitive traffic. An
explicit reference to the later section on buffering flow&congestion
control would be useful here as well.

In addition, if the IP layer supports flow classification (and
scheduling), it should be possible to turn off the flow separation
(classification) across the link queues at the link layer and
allow the IP layer to do the flow classification and forward
each IP flow to an appropriate link service as discussed in
QoS considerations section later.
Link layer may, of course,  do further clasification within a
single link queue, if desired.

Current text in I-D section "Buffering, flow & congestion control":

  "Some subnets may have flow or congestion control mechanisms in
   addition to packet dropping.  Such mechanisms can operate on
   components in the subnet layer, such as schedulers, shapers or
   discarders, and can  affect the operation of IP forwarders at the
   edges of the subnet.  However, with the exception of  RFC2481
   explicit congestion notification (discussed below), IP has no way to
   pass explicit congestion or flow control signals to TCP."

Last sentence: true, but maybe could say that it is possible to
implement a local congestion signal from IP to TCP in the local
IP stack. This is more an implementation issues, but would be
very useful when the first-hop link of the TCP-sender is the
bottleneck link. To aid the IP layer to give this signal, the link
layer is advised to provide a notification to the IP layer
when the link queue(s) are about to become full. For example,
if the local link layer implements active queue management with
ECN and the local IP stack understands the local congestion signal,
the link layer could preferably be configured to provide
a local cingestion signal, that is, send a notification directly
up to the local IP layer instead of setting the ECN bit on the
forwarded packets. This would allow TCP-sender react much faster
to upcoming congestion on the local link interface.

The rest are mostly nits:

-at least the following references are missing from the reference list:
 RFC3048, RFC2990, RFC1633, RFC2205, RFC2208, RFC2210, RFC2211, RFC2212,
 RFC2474, RFC2475, RFC2597, RFC2598

- [MBDL99] need fullreference information, there are more of
  these with only limited reference details.

Thanks,

/Markku