Subject: RE: Draft on Advice for Internet Subnetwork Designers

From: Marie-Jose Montpetit <[email-masked]>
Date: Wed, 1 Mar 2000 15:50:38 -0800

My own comments below.

-----Original Message-----
From: Dan Grossman [mailto:[email-masked]]
Sent: Wednesday, March 01, 2000 2:17 PM
To: Falk, Aaron
Cc: [email-masked]'; Phil Karn (E-mail)
Subject: Re: Draft on Advice for Internet Subnetwork Designers 

Comments:

1) Shortly before the Washington meeting, I sent a suggestion to change the 
Introduction/Overview section as follows.  

> ...seminal paper "End-to-End Arguments in System Design" [SRC81]. This
paper 
> argued that many -- if not most -- network functions are best implemented
on 
> an end-to-end basis, i.e., at the higher protocol layers.  Duplicating
these 
> functions at the lower levels can be needlessly redundant or harmful. 
> However,  there are also cases where duplication can be justified by 
> considerations such  as performance, availability or security.  Examples 
> might include link layer  retransmission on an unusually lossy channel,
e.g., 
> mobile radio, encryption  of the entire link layer protocol over a shared 
> medium, or  redundant paths at the link layer for fault recovery.  
> Duplication of functions should be done only based on an understanding of 
> system level implications, including any side-effects on the higher
layers.

I still believe rather strongly that this is more appropriate than the
current 
wording.  Since it has been quite a while since the last discussion of the 
document on the mailing list,  a reminder seems appropriate.

I agree. Strangely enough I was also trying to come up with words for the
introduction as it was apparent from the disucssions in Washington that the
purpose of the LINK draft was not fully understood. While the actual wording
is probably variable from one person to another I think the message should
be clear that the LINK draft is there to help to have a system view hence to
help sub-network designers get a flavor for the end to end aspects of
deploying IP subnets and make sure these subnets integrate in the IP Network
of Networks.

2) Some initial thoughts on what we should do with the empty section on 
buffering, congestion and flow control.  Let's discuss on the list, and I'll

draft text after the discussion:
a) [Following owes a great deal to a side-discussion I had with Mark Allman]

TCP needs lots of buffering.  An ability to buffer bursts of up to (fixed 
delay)*(bottleneck rate) per TCP connection in each node is ideal if 
impractical.  Problem is that neither the fixed delay nor the bottleneck
rate 
is necessarily known at any node.  A good approximation is fixed delay of
the 
link * link bandwidth.  For links that have variable delays, the
approximation 
should be based on the max.  Also, the maximum number of TCP connections per

node is very difficult to determine a priori, and is not subject to any kind

of admission control (except perhaps at a PEP).  A SWAG has to suffice.
Some 
kind of mechanism to measure delay on the link can be useful in autotuning 
buffer sizes.  In general, more buffers are better than fewer.  For example,

TCP performance in very early ATM switch implementations suffered horribly 
from the small buffer sizes.

Yes but if I remember this is also when the traffic statistics were
basically ill undestood. While I do agree that more buffering is better than
less, I would also raise the issue of buffering delays especially if a path
goes through more than one subnet. Then do you have to implement some queue
management at each point?

b) There is a rough consensus (not unanimous) in the research community that

active queue management is important to improving fairness, link utilization

and throughput. Reference RFC 2309.  Followed by a brief introduction to 
active queue management, with RED as an example. An important part of the 
concept is that active queue management attempts to keep average queue
depths 
short, while accommodating short term bursts.  It may be advisable for
subnets 
that perform queueing and discarding to consider implementing some form of 
active queue management, especially if large aggregates of TCP connections
are 
likely to share the same queue.  It may be less effective in the case, for 
example, of an ATM switch that implements per-VC queueuing.  Note, 
incidentally, that active queue management algorithms in general are highly 
sensitive to parameter settings and reference Borden and  Fioriu paper.  

Again I agree. But do my question qould be to which level of detail we want
to go? And is active queue management a link issue?

c) If the subnet does fragmentation and reassembly, make sure that if it 
becomes necessary to discard, then discard entire frames, not just scattered

fragments. For ATM, this specifically means using EPD/PPD, and reference ATM

Forum TM 4.0 specification.   Reference Romanow paper. 

Yes and I think we could even suggest that the subnet data unit (SDU?) have
some markings for this.

d) If the subnet includes flow control that shapes the rate at the edge of
the 
subnet to match a share of the bottleneck bandwidth in the subnet (e.g., ATM

ABR), this has the effect of pushing congestion to the edge of the subnet.
In 
such a case, adequate buffering is needed at the edge of the subnet (see a 
above).  Whether or not there is benefit in this kind of flow control is 
controversial, and there have been simulation studies that go both ways.  
Anecdotally, some large networks have used ABR sucessfully to carry IP 
traffic, but no published results.

This also has links into the subnets that use bandwidth on demand and yes
the buffering is an issue again. I have seen BTW RSVP to ATM results (S.
Giordano) but not ABR.

e) Flow control mechanisms that mark but do not shape frames in excess of
fair 
share of the bottleneck rate (e.g., frame relay FECN and BECN) are at best 
useful only in the context of IP ECN (RFC 2481).  New subnets would do well
to 
have flow control mechanisms that have that semantics (i.e., feedback signal

based on EWMA buffer size, feedback set probabilistically proportional to 
difference between two thresholds, or always set when second threshold 
exceeded) so that feedback could be 'bubbled up' from the subnet layer to
the 
IP header.  

Yes. How about some bubbling up of air interface conditions into control
messages or cost functions?

3) I have some ideas on QoS, which I'll try to organize and send out over
the 
next few days.

Any link will have to address the resource negociations/reservations and
some of the diffserv issues; the active queue management you addressed
already.

Marie-Jose

---

From: Cannara <[email-masked]>
Date: Wed, 01 Mar 2000 18:57:59 -0800

Another comment -- one point that was raised quite a while ago, but remains
overlooked in IP6 is that dependence on TCP/UDP checksums is irrelevant to
routers.  The IP checksum, now absent in default IP6 pkts raises the
possibility of bad router memory creating downstream problems for routers that
cache route paths via address analysis.  Any one router or level 3 switch with
some bad buffer/register mem can wreak havoc in later router decision making.
This seems to be one of a few curious steps backward taken in IP6.

Alex

---

From: Phil Karn <[email-masked]>
Date: Thu, 2 Mar 2000 00:20:05 -0800

Thanks for all the comments. I'll work on incorporating them into the
draft this weekend after I get back from the IAB workshop.

Phil

---

From: Dan Grossman <[email-masked]>
Date: Thu, 02 Mar 2000 17:53:32 -0500

> management at each point?
The model I've been working with (especially since that side discussion) is 
that if the subnet does queueing (buffering, scheduling, discarding), then the 
queueing systems in the subnet has exactly the same effect as queues in 
routers.  This may be worth stating.  Thus, recommendations about having a 
window size worth of buffers per TCP connection,  using active queue 
management, etc. in the subnet would follow from similar recommendations at 
the IP layer.  

As with IP layer forwarding, the objective is not to have large buffers that 
are always full, but rather the capability to absorb large bursts.  Since many 
of the kinds of subnets we're talking about will be (or will contain) the 
bottleneck link, one might expect that an entire window's worth of data might 
easily wind up at a single node in the subnet.
 
> to go? And is active queue management a link issue?
Probably not much more detail than the sketch.  Details could be by reference.
> Yes and I think we could even suggest that the subnet data unit (SDU?) have
Can you explain?
> the buffering is an issue again. I have seen BTW RSVP to ATM results (S.
RSVP to ATM is an unrelated issue.  Raj Jain and some of his students did a 
lot of simulation papers on this a few years ago.  So did Marty Borden and 
Nanying Yin (I think).  And there were some advocacy papers in the ATM Forum.  
I think it's fair to say that there was no consensus among the researchers in 
this area.  In the meantime, I have heard that there have been some sucessful 
deployments, but the network operators involved have not published their 
results.
> messages or cost functions?
I'm not sure what you're suggesting.  The semantics that I wanted to 'bubble 
up' is identical to that of the ECN in RFC 2481 (see my observation above). 
Thus, a queue in the subnet could provide a filtered indication to the TCP 
layer that it was congested just as if were a queue in a router.   What 
semantics in an air interface could map to any existing or potential semantics 
in IP?

> Marie-Jose
Dan

---

From: Phil Karn <[email-masked]>
Date: Tue, 7 Mar 2000 23:11:11 -0800

>routers.  This may be worth stating.  Thus, recommendations about having a 

Yes. In addition, a subnet that provides lots of buffering must also
provide QoS hooks that the IP layer can use. Otherwise, meaningful
IP-level QoS will be difficult or impossible to provide.

Consider a subnet that can buffer, say, 1 sec of traffic but only
provides FIFO service. Then 1 second becomes the minimum delay that
the router driving the link can provide to any IP packet regardless of
how important its TOS bits say it is.

I ran into this problem in the early 1990s when dialup modems began to
incorporate error control with internal buffers holding several
seconds of data. Once the router sent something to the modem, it could
not say "hey, take this new packet and put it ahead of the ones I just
sent you". Even though my router gave telnet traffic priority to ftp
traffic, telnet sessions performed poorly because each telnet packet
had to wait for all the FTP data in the modem's internal queue to
drain first.

I'm not sure what to say about this in the draft except that subnets
that implement more than a minimal amount of internal queuing should
probably provide some sort of QoS hook that IP can use to implement
its own QoS mechanisms.

Phil