Subject: Comment on draft-ietf-tsvwg-byte-pkt-congest-05.txt

From: Fred Baker <[email-masked]>
Date: Mon, 14 Nov 2011 15:52:06 +0800

May I make one more comment on this document?

I came into the twvwg meeting today to be present for the discussion, which turned out to mostly not happen - there was a slide and a comment from the mike, and that was the summary of the discussion.

So...

Grepping for the word "consensus", I find:

>    Consensus has emerged over the years concerning the first stage:
>    whether queues are measured in bytes or packets, termed byte-mode
>    queue measurement or packet-mode queue measurement.  Section 2.1 of
>    this memo records this consensus in the RFC Series.  In summary the
>    choice solely depends on whether the resource is congested by bytes
>    or packets.
> ...
>    The IRTF Internet congestion control research group (ICCRG) has set
>    itself the task of reaching consensus on generic forwarding
>    mechanisms that are necessary and sufficient to support the
>    Internet's future congestion control requirements (the first
>    challenge in [RFC6077]).  Therefore, we defer the question of whether
>    packet congestion might become common and what to do if it does to
>    the IRTF (the 'Small Packets' challenge in [RFC6077]).
> ...
>       *  Removed language about consensus

Did we really remove the language about consensus?

In the section pointed to, it reports this consensus:

> 2.1.  Recommendation on Queue Measurement
> 
>    Queue length is usually the most correct and simplest way to measure
>    congestion of a resource.  To avoid the pathological effects of drop
>    tail, an AQM function can then be used to transform queue length into
>    the probability of dropping or marking a packet (e.g.  RED's
>    piecewise linear function between thresholds).
> 
>    If the resource is bit-congestible, the implementation SHOULD measure
>    the length of the queue in bytes.  If the resource is packet-
>    congestible, the implementation SHOULD measure the length of the
>    queue in packets.  No other choice makes sense, because the number of
>    packets waiting in the queue isn't relevant if the resource gets
>    congested by bytes and vice versa.
> 
>    Corollaries:
> 
>    1.  A RED implementation SHOULD use byte mode queue measurement for
>        measuring the congestion of bit-congestible resources and packet
>        mode queue measurement for packet-congestible resources.
> 
>    2.  An implementation SHOULD NOT make it possible to configure the
>        way a queue measures itself, because whether a queue is bit-
>        congestible or packet-congestible is an inherent property of the
>        queue.
> 
>    The recommended approach in less straightforward scenarios, such as
>    fixed size buffers, and resources without a queue, is discussed in
>    Section 4.1.

I think the statement is fine as far as it goes, but it has two remaining issues. One is the question of when an interface is bit/byte congestible and when it is packet-congestible; I'm sure that means something specific to the authors, but will be far less clear to an implementer - every interface has a rate, and few if any measure their rates in packets per unit time, so specifically what is in view as a "packet-congestible interface"? The other is that it only really applies to interfaces to a unique and non-blocking channel, like a serial line. When the congestible interface or queue is on a shared resource such as the fabric in an input-queued switch, an 802.11 interface, etc, the mathematics really involves two queues - the interface queues (in a statistical sense if not a hardware version) for access to the channel, and the packet is enqueued toward the interface, and both arrival/departure distributions are important. 

Consider, for example, an AQM implementation on a WiFi channel. We have all been on busy 802.11 networks; we have this experience in our own history at the IETF. The dynamics are fairly strange. At the IP layer, sessions are generally between hosts on the WiFi network and hosts somewhere else. At the WiFi layer the AP is a proxy for the remote hosts, and is therefore in essence an extremely busy member of the WiFi system that otherwise operates in an approximation to a round-robin fashion. The effect is to build a bufferbloat scenario into the AP. Due to the shared nature of the medium and the fact that retransmissions are driven by absolute time (an RTO happens after a certain point in time), what one is really looking for is a packet's queue occupancy being measured in time. Bytes are a reasonable analog to time (which is what makes bytes reasonable for a bit-congestible interface) for noncontested interfaces, but a shared interface can fail to emit a transmission for an arbitrary time interval. So I might find myself, if some variant on Blue is in use, bumping the mark/drop probability when the queue becomes full and *also* any time one dequeues a packet that has been waiting longer than some time interval.

As to the "SHOULD NOT" regarding configuration, this probably makes a lot of sense in academic circles. Speaking as a vendor, I'm going to do what my customers tell me to. I'm happy enough with a default configuration following this recommendation, but if my customer wants it different, I'm not going to slap his hands.

As a side note, I would generally suggest that we step aside from "RED" and talk about "AQM". Blue, SFQ-Blue, and AVP are AQM algorithms that are likely superior to RED from an operational viewpoint.

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Nov 2011 01:49:40 -0800

Hi, Fred,

On 11/13/2011 11:52 PM, Fred Baker wrote:
...
(existing text about bit/byte vs. packet congestion)
> I think the statement is fine as far as it goes, but it has two
> remaining issues. One is the question of when an interface is
> bit/byte congestible and when it is packet-congestible; I'm sure that
> means something specific to the authors, but will be far less clear
> to an implementer - every interface has a rate, and few if any
> measure their rates in packets per unit time, so specifically what is
> in view as a "packet-congestible interface"?

Some examples in the text might be useful.

Bit/byte congestion:
	- storage that assigns space based on the packet size
	- interfaces whose output bandwidth is overloaded
	- processing that is length-dependent (e.g., full packet
	hashes or encryption)

Packet congestion:
	- storage that assigns space on a per-packet basis,
	i.e., that has fixed slots (for max packets)
	- interfaces whose output slots are overloaded
	(with fixed slots per packet)
	- processing that is per-packet dependent (e.g.,
	header processing, header hashing, tunneling)

It is useful to note that there are some variants of these, e.g., where 
there are a small number of different buffer sizes (coarsely 
per-bit/byte), or ones with combined bottlenecks (there can be multiple 
places where resources are under contention!)

It's useful to explain what to do when there are combined bottlenecks in 
both places (alleviating only one might not suffice). I wonder what the 
algorithm should be in that case...

 > The other is that it
> only really applies to interfaces to a unique and non-blocking
> channel, like a serial line. When the congestible interface or queue
> is on a shared resource such as the fabric in an input-queued switch,
> an 802.11 interface, etc, the mathematics really involves two queues
> - the interface queues (in a statistical sense if not a hardware
> version) for access to the channel, and the packet is enqueued toward
> the interface, and both arrival/departure distributions are
> important.
>
> Consider, for example, an AQM implementation on a WiFi channel. We
> have all been on busy 802.11 networks; we have this experience in our
> own history at the IETF. The dynamics are fairly strange. At the IP
> layer, sessions are generally between hosts on the WiFi network and
> hosts somewhere else. At the WiFi layer the AP is a proxy for the
> remote hosts, and is therefore in essence an extremely busy member of
> the WiFi system that otherwise operates in an approximation to a
> round-robin fashion. The effect is to build a bufferbloat scenario
> into the AP. Due to the shared nature of the medium and the fact that
> retransmissions are driven by absolute time (an RTO happens after a
> certain point in time), what one is really looking for is a packet's
> queue occupancy being measured in time. Bytes are a reasonable analog
> to time (which is what makes bytes reasonable for a bit-congestible
> interface) for noncontested interfaces, but a shared interface can
> fail to emit a transmission for an arbitrary time interval. So I
> might find myself, if some variant on Blue is in use, bumping the
> mark/drop probability when the queue becomes full and *also* any time
> one dequeues a packet that has been waiting longer than some time
> interval.

This is the combined example I suggested above AFAICT.

Joe

> As to the "SHOULD NOT" regarding configuration, this probably makes a
> lot of sense in academic circles. Speaking as a vendor, I'm going to
> do what my customers tell me to. I'm happy enough with a default
> configuration following this recommendation, but if my customer wants
> it different, I'm not going to slap his hands.
>
> As a side note, I would generally suggest that we step aside from
> "RED" and talk about "AQM". Blue, SFQ-Blue, and AVP are AQM
> algorithms that are likely superior to RED from an operational
> viewpoint.

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 30 Nov 2011 10:51:34 +0000

Fred,

Sorry for the (very) belated reply.

You're right we didn't remove all occurrences of the consensus word 
(deliberately). The summary of changes should have been more specific.

We removed the language about consensus solely over whether byte-size 
should be taken account of in the network or at the end-systems, 
which is what I thought you & David Black thought was overstepping the mark.

The other two occurrences of the word 'consensus' I think are valid 
and can remain.
- Once Sally had pointed out (1997) that there's a distinction 
between i) using bytes to measure the queue and ii) whether you 
drop/mark a particular packet because of its byte-size, there has 
been no controversy on the issue #i. So the role of this RFC is 
indeed to declare consensus on issue #i.
- The IRTF having set itself the task of reaching consensus on 
something is surely a perfectly OK use of the word consensus.

I'll answer the second part of your email along with a response to 
Joe's follow-up.

Bob

At 07:52 14/11/2011, Fred Baker wrote:
>I came into the twvwg meeting today to be present for the 
>discussion, which turned out to mostly not happen - there was a 
>slide and a comment from the mike, and that was the summary of the discussion.
>
>So...
>

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 30 Nov 2011 12:49:20 +0000

Joe & Fred,

(I've added the this context, given I've taken so long to reply - sorry):
 >    queue in packets.

>Some examples in the text might be useful.

We did give examples at the point where we define 
'packet-congestible' earlier (pasted below).

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
1.1. Terminology and Scoping

    Bit-congestible vs. Packet-congestible:

       Examples of packet-congestible resources are route look-up engines
       and firewalls, because load depends on how many packet headers
       they have to process.  Examples of bit-congestible resources are
       transmission links, radio power and most buffer memory, because
       the load depends on how many bits they have to transmit or store.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

>Bit/byte congestion:        - storage that assigns space based on 
>the packet size
>It is useful to note that there are some variants of these, e.g., 
>where there are a small number of different buffer sizes (coarsely 
>per-bit/byte), or ones with combined bottlenecks (there can be 
>multiple places where resources are under contention!)

[BB]: To help implementers understand how to interpret the guidance 
in a more complex cases like this we talk about both cases as 
specific examples in the text. Search for 'buffer carving' and 
'hybrid forwarding system' respectively.

>It's useful to explain what to do when there are combined 
>bottlenecks in both places (alleviating only one might not suffice). 
>I wonder what the algorithm should be in that case...

[BB]: To avoid making the draft too complicated, we fudged a bit by 
saying the following:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
    If we now imagine a hybrid forwarding system with transmission delay
    largely dependent on the byte-size of packets but buffers of one MTU
    per packet, it should strictly require a more complex algorithm to
    determine the probability of congestion.  It should be treated as two
    resources in sequence, where the sum of the byte-sizes of the packets
    within each packet buffer models congestion of the line while the
    length of the queue in packets models congestion of the queue.  Then
    the probability of congesting the forwarding buffer would be a
    conditional probability--conditional on the previously calculated
    probability of congesting the line.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

Given Fred's request for clearer advice to implementers, I would like 
to replace the above para with the following (Jukaa might be able to 
shorten it - he's better than me at brevity):

"If we now consider a hybrid forwarding system with transmission 
congestion dependent on the byte-size of packets but fixed-size 
buffers of one MTU per packet, the queue length can still simply be 
measured in bytes, based on the following reasoning.

When an interface buffers packets, we need to ask what is the root 
cause of the congestion: the line or the buffer? In this case, the 
line is the root of the congestion. Packets temporarily back-up into 
the buffer because more bits are arriving than departing. Even though 
a run of small packets would fill MTU-sized buffers more quickly, the 
line will also drain them more quickly. Therefore, the queue is 
always merely a symptom of how congested the line is. Certainly, once 
the buffer gets full, we could say the buffer has become congested 
too. But reducing the arriving bit-rate would clear congestion of the 
line, which in turn would empty the buffer.

Theoretically, congestion in this hybrid could be treated as the 
probability of congestion of the buffer conditional on congestion of 
the line, where the sum of bytes in the buffers measures congestion 
of the line while the length of the queue in packets measures 
congestion of the queue. In practice, however, just measuring the 
bytes queued gets at the root cause: congestion of the line.

If the queue were building up behind forwarding look-ups or a 
firewall, then the problem would be the number of packets headers, 
not bits. But if it is a queue into a transmission line, the problem is bits.

In summary, for these hybrid cases, whether to measure the queue 
length in bits or packets depends primarily on whether the process in 
front of the queue is bit-congestible (e.g. a transmission line) or 
packet-congestible (e.g. forwarding look-ups). The internals of the 
buffer can usefully be ignored.
"

>>one dequeues a packet that has been waiting longer than some time

Yes, you have to find something that approximates to a model of the 
sequence of congested resources. In byte-pkt we decided not to go 
deeply into this as another example. We felt the previous hybrid 
example had already covered the byte-vs-pkt issue sufficiently, and a 
wireless example might say a lot more about how to design a wireless 
AQM, but not a lot more about byte-v-pkt. Instead we referred to a 
Mobicom paper by Vasilios Siris on this 
<http://tools.ietf.org/html/draft-ietf-tsvwg-byte-pkt-congest-05#ref-ECNFixedWireless> 
(see Section 4.1.2. Congestion Measurement without a Queue).

That reference focuses on CDMA, but I chose it because it combines 
many forms of congestion you can get on an e2e fixed/wireless path. 
You can find other papers, including one on correctly signalling WiFi 
congestion here:
<http://www.ics.forth.gr/netlab/future_wireless.html>

>This is the combined example I suggested above AFAICT.

Yup.

Bob

---

From: Manner Jukka <[email-masked]>
Date: Wed, 30 Nov 2011 13:36:35 +0000

Hi, as a sort of side note, I had a chat with Fred in Taipei and gave an example of the packet-congestible resource: WLAN. (Fred had a concern/question about what is packet-congestible in the real world)

WLAN is in theory bit-congestible but WLAN also has an upper limit of packets/second and with small packets, it actually becomes packet-congestible. Yet, as a caveat, the packet-congestion is not really visible to the implementer of AQM, this was pointed out by Fred.

Jukka

On Nov 30, 2011, at 2:49 PM, Bob Briscoe wrote:

>> Bit/byte congestion:         - storage that assigns space based on the packet size
>> It is useful to note that there are some variants of these, e.g., where there are a small number of different buffer sizes (coarsely per-bit/byte), or ones with combined bottlenecks (there can be multiple places where resources are under contention!)
> 
> [BB]: To help implementers understand how to interpret the guidance in a more complex cases like this we talk about both cases as specific examples in the text. Search for 'buffer carving' and '
> hybrid
> forwarding system' respectively.
> 
> 
> 
>> It's useful to explain what to do when there are combined bottlenecks in both places (alleviating only one might not suffice). I wonder what the algorithm should be in that case...
> 
> [BB]: To avoid making the draft too complicated, we fudged a bit by saying the following:
> 
> Given Fred's request for clearer advice to implementers, I would like to replace the above para with the following (Jukaa might be able to shorten it - he's better than me at brevity):
> 
> "If we now consider a hybrid forwarding system with transmission congestion dependent on the byte-size of packets but fixed-size buffers of one MTU per packet, the queue length can still simply be measured in bytes, based on the following reasoning.
> 
> When an interface buffers packets, we need to ask what is the root cause of the congestion: the line or the buffer? In this case, the line is the root of the congestion. Packets temporarily back-up into the buffer because more bits are arriving than departing. Even though a run of small packets would fill MTU-sized buffers more quickly, the line will also drain them more quickly. Therefore, the queue is always merely a symptom of how congested the line is. Certainly, once the buffer gets full, we could say the buffer has become congested too. But reducing the arriving bit-rate would clear congestion of the line, which in turn would empty the buffer. 
> 
> Theoretically, congestion in this hybrid could be treated as the probability of congestion of the buffer conditional on congestion of the line, where the sum of bytes in the buffers measures congestion of the line while the length of the queue in packets measures congestion of the queue. In practice, however, just measuring the bytes queued gets at the root cause: congestion of the line.
> 
> If the queue were building up behind forwarding look-ups or a firewall, then the problem would be the number of packets headers, not bits. But if it is a queue into a transmission line, the problem is bits. 
> 
> In summary, for these hybrid cases, whether to measure the queue length in bits or packets depends primarily on whether the process in front of the queue is bit-congestible (e.g. a transmission line) or packet-congestible (e.g. forwarding look-ups). The internals of the buffer can usefully be ignored. 
> "
> 
> 
>> > The other is that it
> Yes, you have to find something that approximates to a model of the sequence of congested resources. In byte-pkt we decided not to go deeply into this as another example. We felt the previous hybrid example had already covered the byte-vs-pkt issue sufficiently, and a wireless example might say a lot more about how to design a wireless AQM, but not a lot more about byte-v-pkt. Instead we referred to a Mobicom paper by Vasilios Siris on this < http://tools.ietf.org/html/draft-ietf-tsvwg-byte-pkt-congest-05#ref-ECNFixedWireless > (see Section 4.1.2. Congestion Measurement without a Queue).
> 
> That reference focuses on CDMA, but I chose it because it combines many forms of congestion you can get on an e2e fixed/wireless path. You can find other papers, including one on correctly signalling WiFi congestion here:
> < http://www.ics.forth.gr/netlab/future_wireless.html>
> 
>

---

From: John Leslie <[email-masked]>
Date: Wed, 30 Nov 2011 12:22:45 -0500

Apologies up-front: I thought I was subscribed to this list and only
recently _actually_ subscribed.

   Further, searching the archives, I didn't find the answer to this
question...

   Could somebody point me at a particular post explaining why this
I-D is seeking BCP status?

--
John Leslie <[email-masked]>