Subject: [Int-area] draft-ietf-intarea-ipv4-id-update-02.txt

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Apr 2011 12:23:41 +0200

I just read draft-ietf-intarea-ipv4-id-update-02.txt.

First off, do we really need to UPDATE RFC 791 a quarter century after the fact when the IPv4 address space is all but exhausted? This work simply comes too late to be very useful.

Another big problem is that it assumes that the DF bit is always honored. There is no reason to expect that if the source sets DF bit to 1, this means the packet is never fragmented. Due to massive misuse of path MTU discovery all over the internet, people are forced to use unorthodox measures to mitigate the resulting breakage. One way to do this is to rewrite a 1 DF into a 0 DF so packets can be fragmented. This is supported on at least some routers. However, such routers don't necessarily support rewriting the ID value, so it's important that ALL IPv4 packets have a valid ID, even if the DF bit is set to 1.

BTW, does anyone know whether if, say, packet 1 with ID 100 is fragmented and a fragment is lost, and then packet 65537 with ID 100 is received, packet 2 will be used to "reassemble" packet 1? If no, then using a valid ID value in all packets is not harmful in the case where packets with DF=1 is honored.

And FYI, from RFC 4302, which is AFAIK the only place where header fields are explicitly classified as mutable or immutable in transit:

3.3.3.1.1.1.  Base Header Fields

   The IPv4 base header fields are classified as follows:

   Immutable
           Version
           Internet Header Length
           Total Length
           Identification
           Protocol (This should be the value for AH.)
           Source Address
           Destination Address (without loose or strict source routing)

   Mutable but predictable
           Destination Address (with loose or strict source routing)

   Mutable (zeroed prior to ICV calculation)
           Differentiated Services Code Point (DSCP)
              (6 bits, see RFC 2474 [NBBB98])
           Explicit Congestion Notification (ECN)
              (2 bits, see RFC 3168 [RFB01])
           Flags
           Fragment Offset
           Time to Live (TTL)
           Header Checksum

---

From: Joe Touch <[email-masked]>
Date: Tue, 05 Apr 2011 09:14:57 -0700

Hi, Iljitsch,

On 4/5/2011 3:23 AM, Iljitsch van Beijnum wrote:
> First off, do we really need to UPDATE RFC 791 a quarter century
> after the fact when the IPv4 address space is all but exhausted? This
> work simply comes too late to be very useful.

The point of the document is to explain and permit existing behavior, 
and to clarify where other existing behavior is problematic.

> Another big problem is that it assumes that the DF bit is always
> honored.

The following are already true:

- the ID is repeated within 2MSL

- the ID is repeated constantly (e.g., sent as "0" in some cellphones)

- the DF bit is ignored

This document recognizes that repeating the ID should not have 
consequences when DF is honored, and thus permits it.

We don't assume the DF bit is honored; we do explain the dangers of 
ignoring it. Those dangers are already present.

So basically, the choice in this doc is:

	A- ignore two conflicting violations of the spec

	B- declare that one violation is acceptable, but
	remind that the other is not

	C- declare that both are acceptable

The doc selects B because it is safe. We can't declare ignoring the DF 
safe; there may be cases where it 'works', but it is never safe.

> There is no reason to expect that if the source sets DF bit
> to 1, this means the packet is never fragmented. Due to massive
> misuse of path MTU discovery all over the internet, people are forced
> to use unorthodox measures to mitigate the resulting breakage. One
> way to do this is to rewrite a 1 DF into a 0 DF so packets can be
> fragmented. This is supported on at least some routers. However, such
> routers don't necessarily support rewriting the ID value, so it's
> important that ALL IPv4 packets have a valid ID, even if the DF bit
> is set to 1.

Let's assume you are correct. It's equally then important that all IPv4 
packets have a valid ID.

As you note, *some* routers and other devices ignore the DF bit.

However, *most* (if not all) devices with links over 6.4Mbps already 
violate the "valid ID" assumption you make.

This document favors the majority of hosts and their desire to have 
individual connections not limited by IP ID uniqueness. This is NOT a 
change in how anyone does things.

All this document does is shift from two "violating behaviors" to one.

> BTW, does anyone know whether if, say, packet 1 with ID 100 is
> fragmented and a fragment is lost, and then packet 65537 with ID 100
> is received, packet 2 will be used to "reassemble" packet 1?

If it arrives within the fragmentation reassembly timeout, it definitely 
will. The end host won't know it's a fragment from packet 65537.

> If no,
> then using a valid ID value in all packets is not harmful in the case
> where packets with DF=1 is honored.
>
> And FYI, from RFC 4302, which is AFAIK the only place where header
> fields are explicitly classified as mutable or immutable in transit:

RFC791 talks about only some header fields changing. It goes into detail 
only for the length of some options.

RFC1812 sec 5.2.5 says the unused bits must pass unchanged, 5.3.3.3 says 
the precedence fields must not be changed, 5.3.13 says unrecognized 
options must pass unchanged. There may be other places; I did just a 
quick check.

RFC4302 talks about DF being mutable, but that's in the sense of having 
a static set of fields over which cryptographic operations occur. Note 
that it also states - in the section below, specifically that DF can be 
set, but not that it can be reset:

--
    Flags - This field is excluded because an intermediate router might
    set the DF bit, even if the source did not select it.
--

RFC4302 does not consider the implications of setting the DF bit 
together with the Internet's path asymmetry, however.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Apr 2011 18:36:23 +0200

> The doc selects B because it is safe. We can't declare ignoring the DF safe; there may be cases where it 'works', but it is never safe.

I would call this a false trichotomy, if such a thing exists.

Whatever we do, we shouldn't let people get away with not using valid ID values.

But the real solution is not to rehash broken implementations of 25-year-old assumptions, but rather address the real problem, if there is a problem to be addressed.

> However, *most* (if not all) devices with links over 6.4Mbps already violate the "valid ID" assumption you make.

This is the idea that an ID value can't be repeated for 120 (or 240) seconds for the same 3-tuple. But why 120 seconds? Is there any reasonable way that a packet gets fragmented, and then one of the fragments is delayed so long than 65535 other packets are received in the interim? I can't think of anything. So I would favor a system where fragments are buffered for reassembly not for some fixed amout of time, but for the time it takes to receive some X number of packets, where X can be something like 100, 1000 or 10000 packets.

Yes, in a very small number of cases this will lead to an avoidable loss when the missing fragment is received after all just as the previous fragments were thrown away, but that's something I can live with, considering the alternative that is either basically an incorrect reassembly for EVERY lost fragment (and presumably if fragments happen they happen for nearly all packets so this happens for EVERY loss) which translates into an undetected data corruption once every ~ 65535 losses OR artificially limiting flows to only a few megabit/s.

Iljitsch

---

From: Joe Touch <[email-masked]>
Date: Tue, 05 Apr 2011 09:51:31 -0700

> Whatever we do, we shouldn't let people get away with not using valid ID values.

That ship sailed. Unless you're throttling all your connections to under 
6.4Mbps.

> But the real solution is not to rehash broken implementations of
> 25-year-old assumptions, but rather address the real problem, if there
> is a problem to be addressed.

That's exactly the purpose of the draft. It also helps inform other
IETF areas that IDs do repeat, and *not* to rely on the uniqueness of
the ID field, e.g., for duplicate detection and removal (as is currently
being proposed in MANET (SMF).

> This is the idea that an ID value can't be repeated for 120 (or 240)
> seconds for the same 3-tuple. But why 120 seconds? Is there any
> reasonable way that a packet gets fragmented, and then one of the
> fragments is delayed so long than 65535 other packets are received in
> the interim? I can't think of anything.

There are plenty of systems with path asymmetries with delays of tens of 
seconds. E.g., see Jim Getty's presentation this past IETF in TSVAREA.

 > So I would favor a system where
> fragments are buffered for reassembly not for some fixed amout of time,
> but for the time it takes to receive some X number of packets, where X
> can be something like 100, 1000 or 10000 packets.

That's another discussion included in the draft, and another purpose of 
the draft. As is the note that upper layers should not rely on packets 
being reassembled correctly - but instead should include upper layer 
checksums that are robust in the presence of misfragmentation.

Overall, the doc is intended to inform, and *relax* existing 
requirements where possible, and to warn where this is either not 
possible or stricter behavior (e.g., routers not changing DF=0 to DF=1) 
is safer.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 5 Apr 2011 22:44:45 +0200

> There are plenty of systems with path asymmetries with delays of tens of seconds. E.g., see Jim Getty's presentation this past IETF in TSVAREA.

Can't find that real quick.

But I'm doing some measurements currently, and what I'm seeing doesn't support that at all, for two reasons:

- reordering (= the packets must be sent down different paths) is very rare for packets that belong to the same session, think single digit percentages
- even if it happens, the timing differences between reordered packets are counted in microseconds, not even whole milliseconds

In addition to that, I don't see multiple second delays AND hundreds or thousands of packets per second happening at the same time. Slow networks could potentially do the former, fast networks the latter, but I can't think of anything that can do both.

> As is the note that upper layers should not rely on packets being reassembled correctly - but instead should include upper layer checksums that are robust in the presence of misfragmentation.

If the ID rollover time is shorter than the reassembly timeout you WILL get an incorrect reassembly for _every_ lost packet, so a checksum would have to be very strong to protect against that. The current 16-bit checksum isn't strong enough for this. But upgrading TCP and UDP to use larger checksums seems like an uphill struggle. Especially because the lower layer checksums are typically strong enough to protect against transmission errors and the weak internet checksum allows us to do stuff cheap that would otherwise be problematic, such as NAT, which would get a big performance hit if incremental checksum updates weren't possible.

---

From: Scott Brim <[email-masked]>
Date: Wed, 6 Apr 2011 11:33:32 -0400

> Can't find that real quick.

You would enjoy a visit to http://www.bufferbloat.net

Scott

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 6 Apr 2011 18:39:19 +0200

On 6 apr 2011, at 17:33, Scott Brim wrote:

> Jim Getty's presentation this past IETF in TSVAREA.

> You would enjoy a visit to http://www.bufferbloat.net

Ah, it starts at 25 minutes in on the recording for wednesday morning on channel 4, with the slides here:

http://mirrors.bufferbloat.net/Talks/PragueIETF/IETFBloat7.pdf

BTW, I'm not seeing the disasters he's describing, I only get 6 or 12 ms extra delay doing 100 Mbps transfers locally.

But... none of this has ANYTHING to do with the issue at hand, unless someone can show me how packet 50 sits in the buffer for 100 or 1000 or what have you milliseconds and then packet 51 magically flies through the queue and arrives much earlier than packet 50.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 6 Apr 2011 18:57:21 +0200 (CEST)

On Tue, 5 Apr 2011, Iljitsch van Beijnum wrote:

> - even if it happens, the timing differences between reordered packets 
> are counted in microseconds, not even whole milliseconds

I have to disagree with this. I always discourage people from using 
per-packet loadblanacing, but if done on a 2 meg connection (as an 
example), just clocking out a 1500 byte packet takes 6 milliseconds, so 
two packets can easily arrive several milliseconds in wrong order in this 
use-case.

> In addition to that, I don't see multiple second delays AND hundreds or 
> thousands of packets per second happening at the same time. Slow 
> networks could potentially do the former, fast networks the latter, but 
> I can't think of anything that can do both.

I have to agree with that, though.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 6 Apr 2011 19:19:09 +0200

> I have to disagree with this. I always discourage people from using per-packet loadblanacing, but if done on a 2 meg connection (as an example), just clocking out a 1500 byte packet takes 6 milliseconds, so two packets can easily arrive several milliseconds in wrong order in this use-case.

I've been working on 100 Mbps links, obviously this is different if you run much slower, which can happen at the edge of the network. But not really in the core, these days, I would say.

---

From: Joe Touch <[email-masked]>
Date: Wed, 06 Apr 2011 10:27:41 -0700

Hi, Iljitsch,

On 4/6/2011 9:39 AM, Iljitsch van Beijnum wrote:
> BTW, I'm not seeing the disasters he's describing, I only get 6 or 12
> ms extra delay doing 100 Mbps transfers locally.

Not everyone does. It depends on what OS you do the test from, which 
local router (DSL box, etc.) you're using, etc.

> But... none of this has ANYTHING to do with the issue at hand,
> unless someone can show me how packet 50 sits in the buffer for 100
> or 1000 or what have you milliseconds and then packet 51 magically
> flies through the queue and arrives much earlier than packet 50.

That's basically the point. The Internet is a highly variable place, and
getting more so with large delays (buffer bloat, satellite links) and
multiple paths (e.g., automatic use of WiFi vs. 3G, etc. links).

The draft doesn't change any recommendations when DF=0 *because* this 
variability persists. There should be no impact when DF=1 (except for 
misbehaving devices, for which there is already impact for existing 
devices that reuse IDs).

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 06 Apr 2011 10:33:47 -0700

On 4/5/2011 1:44 PM, Iljitsch van Beijnum wrote:
>>> This is the idea that an ID value can't be repeated for 120 (or
>>> 240) seconds for the same 3-tuple. But why 120 seconds? Is there
>>> any reasonable way that a packet gets fragmented, and then one of
>>> the fragments is delayed so long than 65535 other packets are
>>> received in the interim? I can't think of anything.
>
>> There are plenty of systems with path asymmetries with delays of
>> tens of seconds. E.g., see Jim Getty's presentation this past IETF
>> in TSVAREA.
>
> But I'm doing some measurements currently, and what I'm seeing
> doesn't support that at all, for two reasons:

The Internet is a very big place, and it's intended to work in most
places, rather than just in the places any individual currently sees.

> - reordering (= the packets must be sent down different paths) is
> very rare for packets that belong to the same session, think single
> digit percentages

As noted in my other post, that could be increasing with multiple
disparate connections becoming the norm.

> - even if it happens, the timing differences between reordered
> packets are counted in microseconds, not even whole milliseconds

You should measure 3G, and include a satellite path too. Both have
longer latencies even without buffer bloat.

> In addition to that, I don't see multiple second delays AND hundreds
> or thousands of packets per second happening at the same time. Slow

That's common over satellite paths, and also common in the buffer bloat 
case.

>> As is the note that upper layers should not rely on packets being
>> reassembled correctly - but instead should include upper layer
> If the ID rollover time is shorter than the reassembly timeout you
> WILL get an incorrect reassembly for _every_ lost packet, so a
> checksum would have to be very strong to protect against that. The
> current 16-bit checksum isn't strong enough for this. But upgrading
> TCP and UDP to use larger checksums seems like an uphill struggle.

This doc isn't proposing that. It just warns that rolling over the ID in 
the absence of stronger checksums is dangerous.

> Especially because the lower layer checksums are typically strong
> enough to protect against transmission errors and the weak internet
> checksum allows us to do stuff cheap that would otherwise be
> problematic, such as NAT, which would get a big performance hit if
> incremental checksum updates weren't possible.

Stronger object checksums can also exist at the application layer, as 
with Bittorrent, and at the session layer, such as with SSL (that's for 
crypto, but still protects against incorrect reassembly).

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 6 Apr 2011 19:46:29 +0200 (CEST)

> longer latencies even without buffer bloat.

My personal record is 180 seconds delay over GSM. However, the RAN ensures 
packet ordering, so to get re-ordering over this, they'd have to be 
per-packet loadshared between multiple connections.

> That's common over satellite paths, and also common in the buffer bloat 

I dislike the term "buffer bloat". Large buffers have a point, it's just 
that large FIFO buffers are bad. An example, there is little problem 
having single TCP session over 100 meg link between proper routers, get 
100 ms latency. This is between two linux boxes. Turn on cisco 
"fair-queue" (which does require to keep flow state, thus uses lots of 
CPU), and one flow will not give 100ms latency to another flow, instead it 
will see very little difference to an unloaded link.

Anyhow, my point is that re-ordering happens in real life networks 
(although I'd say they're badly engineered ones). I've seen it in real 
life, I've talked to application programmers who designed applications 
which relied on packets being ordered, and who had to re-design with 
jitter buffers to handle out of order packets (instead of counting skips 
in sequence numbers as drops). It happens in slow networks, it happens in 
fast networks.

It's not hugely common, but it's out there.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 6 Apr 2011 22:02:59 +0200

On 6 apr 2011, at 19:46, Mikael Abrahamsson wrote:

> I dislike the term "buffer bloat". Large buffers have a point, it's just that large FIFO buffers are bad. An example, there is little problem having single TCP session over 100 meg link between proper routers, get 100 ms latency.

I really don't understand why people defend large buffers. There is really no upside to them save for the case where the system is otherwise too slow to fill up the pipe.

In fact, in big routers they cost a lot of money and use significant energy. And still C and J put a gigabyte of buffering on some of their interfaces.

In theory some buffering will smoothe out bursts, but in practice TCPs will just build up standing queues using up all the buffering so bursts tail drop anyway.

Of course all of this applies to FIFO, but that's what's used in the vast majority of cases as there are no cheap set-and-forget first-do-no-harm AQMs AFAIK.

> Anyhow, my point is that re-ordering happens in real life networks

Only a few percent of destinations have a non flow conserving (per packet) load balancer on the path, and even then you only trigger the reordering if you send fast enough.

But even then, the timing differences between the reordered packets won't add up to 10+ seconds while the bandwidth is 100s or 1000s of packets per second. For this to happen you'd need load balancing over links with 120 second jitter. Not sure how jitter and queue length are related off the top of my head, but that probably means a standing queue that's even larger than 120 seconds. At the 500 pps that you need to fill up 6 Mbps at 1500 bytes per packet, that's an average queue length of 100k packets or 143 MB to reach a queuing delay of say 200 seconds. That translates into a utilization of 99.999%.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 7 Apr 2011 07:24:00 +0200 (CEST)

On Wed, 6 Apr 2011, Iljitsch van Beijnum wrote:

> I really don't understand why people defend large buffers. There is 
> really no upside to them save for the case where the system is otherwise 
> too slow to fill up the pipe.

Yes there is, if you have long fat pipes. It can be argued that it's much 
less needed now, when core routers have 10G and up, which is what router 
vendors are responding to, newer routers have 50ms buffer instead of 600ms 
they had 10 years ago.

TCP (file transfer) responds better to delayed packets than to lost 
packets.

> In fact, in big routers they cost a lot of money and use significant 
> energy. And still C and J put a gigabyte of buffering on some of their 
> interfaces.

At 40G speed, that's 250ms. There are applications that would rather have 
their packet delayed 100ms but get it delivered, and there are other 
applications who would rather like packet loss but almost no jitter.

The hard part is to make a network that caters well to both without having 
to keep flow state.

> In theory some buffering will smoothe out bursts, but in practice TCPs 
> will just build up standing queues using up all the buffering so bursts 
> tail drop anyway.

Depends on speed, but you're basically right.

> Of course all of this applies to FIFO, but that's what's used in the 
> vast majority of cases as there are no cheap set-and-forget 
> first-do-no-harm AQMs AFAIK.

I agree.

> Only a few percent of destinations have a non flow conserving (per 
> packet) load balancer on the path, and even then you only trigger the 
> reordering if you send fast enough.

I've seen it on 5 megabit/s video streams.

> But even then, the timing differences between the reordered packets 
> won't add up to 10+ seconds while the bandwidth is 100s or 1000s of 
> packets per second.

It's definitely a corner case, I agree.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 Apr 2011 10:13:18 +0200

Hi Joe,

> Not everyone does. It depends on what OS you do the test from, which local router (DSL box, etc.) you're using, etc.

I should have mentioned that is between two Macs, one running 10.5 and one 10.6, with an Apple base station in the middle.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 Apr 2011 10:34:40 +0200

> Yes there is, if you have long fat pipes.

They don't need big buffers. Maybe today's TCP implementations can't 95%+ throughput without them, but if that's the case, then either the hardware is underpowered or the implementation is suboptimal.

Until a few years ago, nobody really used the window scale option because the default maximum send and receive buffers in widely used OSes were very limited. In those days, it paid to have so much buffering that the maximum window size became the bottleneck in a transfer, because that way there are no drops and TCP uses the maximum bandwidth.

But today common OSes will use very large windows as the RTT increases but the larger the windows the larger the standing queues and hence the RTT. This hurts traffic that isn't unlimited file transfer competing for access and even hurts those file transfers themselves because when drops happen now, they get so severe that normal fast retransmit can't handle it, timeouts are necessary and a lot of packets have to be retransmitted.

Now compare this with the situation where the buffers are small. Drops then happen regularly. This probably means you don't get to use 99% of the bandwidth of a long RTT link/path for a single transfer because of the TCP sawtoothing, but it works much, much better for a mix of traffic. And it's still possible to reach that 99% by modifying TCP implementations so that they tune their window to the bandwidth-delay product so they avoid overlading links and depend on self clocking rather than losses to adjust their transmission rate.

> It can be argued that it's much less needed now, when core routers have 10G and up, which is what router vendors are responding to, newer routers have 50ms buffer instead of 600ms they had 10 years ago.

Still way too much.

> TCP (file transfer) responds better to delayed packets than to lost packets.

But that's optimizing for that one use case where you send a lot of data at a rate that saturizes the core of the network. Most people have a local bottleneck so they don't need buffering in the core (whether they need it locally is something they should have the opportunity to decide for themselves) and most applications aren't bulk transfers and most bulk transfers don't have higher priority than interactive traffic.

> There are applications that would rather have their packet delayed 100ms but get it delivered, and there are other applications who would rather like packet loss but almost no jitter.

> The hard part is to make a network that caters well to both without having to keep flow state.

Like I said, the cases where lots of buffering is helpful are few and far between, it's a waste of time to even consider that in general purpose networks.

>> But even then, the timing differences between the reordered packets won't add up to 10+ seconds while the bandwidth is 100s or 1000s of packets per second.

> It's definitely a corner case, I agree.

No, it's not. It's not even a loch ness monster case: nobody has even claims to have observed this, people just think "10 seconds can happen" and "reordering can happen" so "reordering at 10 second timescales can happen". Not valid logic.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 7 Apr 2011 13:13:27 +0200 (CEST)

On Thu, 7 Apr 2011, Iljitsch van Beijnum wrote:

> Like I said, the cases where lots of buffering is helpful are few and 
> far between, it's a waste of time to even consider that in general 
> purpose networks.

I have been advocating this to the people in CONEX, but they seem to think 
ECN is a worthwhile tool. I see little use for ECN without buffering and 
RED.

So "someone" needs to decide how we want the Internet to work. The 
"bufferbloat" people want small buffers generally. People who learnt about 
TCP long ago, want long-fat-pipes to work well, thus lots of buffers.

So what do we do? Right now we have both, there are seconds worth of FIFO 
buffering on some links, on other links there are 2-4ms worth of 
FIFO buffering, on some there are seconds worth of buffering, but it's 
per-flow (cisco fair-queue).

Doing a general recommendation of "never buffer more than 10ms" is a quite 
strong recommendation that needs to be backed up by, and agreed by quite a 
lot of people from different parts of the IETF.

> No, it's not. It's not even a loch ness monster case: nobody has even 
> claims to have observed this, people just think "10 seconds can happen" 
> and "reordering can happen" so "reordering at 10 second timescales can 
> happen". Not valid logic.

Well, if I load-share per-packet over two tunnels traversing a radio 
network, I can make this happen. Would I want to? No.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 Apr 2011 17:48:05 +0200

On 7 apr 2011, at 13:13, Mikael Abrahamsson wrote:

> I see little use for ECN without buffering and RED.

Agree. (Or some other AQM where packets would be dropped while buffer space is still available.)

> So "someone" needs to decide how we want the Internet to work. The "bufferbloat" people want small buffers generally. People who learnt about TCP long ago, want long-fat-pipes to work well, thus lots of buffers.

> So what do we do? Right now we have both,

We can in fact accommodate both philosophies: by creating two service classes. The only tricky part there is how to make sure that resources are evenly split between them, such that it doesn't pay for the long delay people to select the low delay service class.

More service classes could benefit the long fat pipe people a lot, because even though they are the ones that want / can stand large buffers, they are not getting their needs met today either, because they can't ramp up their transmission rate fast enough to get good utilization without being unfair to other traffic. This could be solved by moving to a low priority service class with small buffers but with aggressive transmission that ignores high packet loss.

> Doing a general recommendation of "never buffer more than 10ms" is a quite strong recommendation that needs to be backed up by, and agreed by quite a lot of people from different parts of the IETF.

I'm all for that. But a discussion about this on the HOMEGATE list a year (or two?) ago didn't go anywhere.

> Well, if I load-share per-packet over two tunnels traversing a radio network, I can make this happen. Would I want to? No.

You would need to load balance over multiple links, where one has a 120 second queue and the other doesn't. Seems to me if you have two links where one has 120 seconds more queuing delay than the other, you can probably live without that second link just fine.

---

From: Joe Touch <[email-masked]>
Date: Thu, 07 Apr 2011 09:05:58 -0700

> I should have mentioned that is between two Macs, one running 10.5 and one 10.6, with an Apple base station in the middle.

That's not the scenario that shows bloat; you need to use a constrained 
uplink (e.g., DSL or home cable router), and the goal is:

	- from a home PC, upload a large file to a server upstream of
	the bottleneck

	- from another home PC, ping somewhere upstream
	of the bottleneck

The issue doesn't come up if there's no persistent bottleneck.

Joe

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 7 Apr 2011 18:40:52 +0200

> That's not the scenario that shows bloat; you need to use a constrained uplink (e.g., DSL or home cable router),

> The issue doesn't come up if there's no persistent bottleneck.

No, Jim Gettys also talks about 100 Mbps:

http://gettys.wordpress.com/2010/11/29/home-router-puzzle-piece-one-fun-with-your-switch/

Unfortunately conciseness and easy navigation aren't his strong suits so I can't find the results he got for this test right now, but it was 200 ms unless I'm mistaken.

Of course the difference between this case and the case where the bottleneck isn't the first hop is very different: here, local buffers are a problem, which you should be able to tune locally. In the middle of the network it's a separate device that you presumably have less control over.

---

From: Joe Touch <[email-masked]>
Date: Thu, 7 Apr 2011 10:52:52 -0700

> No, Jim Gettys also talks about 100 Mbps:

He has talked about a lot of things :-). If you can, test with a home uplink. 

The bufferscan sometes be tuned (on some devices) but you need to tune all devices on the path to know upu can avoid the issue, and that's rarely feasible. 

Joe

---

From: Roland Bless <[email-masked]>
Date: Fri, 08 Apr 2011 17:07:36 +0200

Hi,

Am 07.04.2011 13:13, schrieb Mikael Abrahamsson:
> "bufferbloat" people want small buffers generally. People who learnt
> about TCP long ago, want long-fat-pipes to work well, thus lots of buffers.

I'm not sure what you exactly mean by long-fat-pipes here. Do you mean
a path with a large bandwith-delay product? In this case you need lots
of buffers, but mainly in the TCP implementation, not in the network.
For large bandwidth delay product connections the inflight packets are
mainly "stored" on the wire, not in the router buffers, aren't they?
The BDP rule-of-thumb for buffer sizes appears strange to me anyways.
IMHO, smaller buffers with AQM and ECN are probably a really good thing
since dropping fairness can be achieved and flows get once a while
packets through to probe for bandwidth and to control their sending
speed. Large buffers don't help either if you've got long-term congestion...

Regards,
 Roland

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 8 Apr 2011 22:57:22 +0200 (CEST)

> The BDP rule-of-thumb for buffer sizes appears strange to me anyways.

Well, since retransmits take so long time when you have high latency, you 
want packets buffered instead of dropped, and you want to smoothe out 
flows as they're saw-toothing. Having small buffers make this hard.

> speed. Large buffers don't help either if you've got long-term 

Small buffers make AQM and ECN useless, IMHO. ECN with RED makes sense 
with large buffers, with small buffers you might as well FIFO taildrop.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 9 Apr 2011 00:03:43 +0200

On 8 apr 2011, at 22:57, Mikael Abrahamsson wrote:

> Small buffers make AQM and ECN useless, IMHO. ECN with RED makes sense with large buffers, with small buffers you might as well FIFO taildrop.

Well, we had a guy come in and give a talk about having only a one-packet buffer last week and then there's having a gigabyte of buffering. There is middle ground between those.

At 1 Gbps you can have 500 buffers which should be more than enough for AQM and ECN and still only introduce a maximum queuing delay of 6 milliseconds. But with the AQM the buffer usually won't fill up all the way and the average packet size is only ~ 500 and not 1500, so in practice it's going to be more like 1 ms.

In the home you can do some AQM with a much smaller buffer because the number of sessions is much smaller and you can also do AQM variations that keep per session state and don't depend on having extra buffering like RED so it's not a straight choice between FIFO and large buffers there either.

---

From: Roland Bless <[email-masked]>
Date: Tue, 12 Apr 2011 13:19:32 +0200

Hi Mikael,

On 08.04.2011 22:57, Mikael Abrahamsson wrote:
> Well, since retransmits take so long time when you have high latency,
> you want packets buffered instead of dropped, and you want to smoothe
> out flows as they're saw-toothing. Having small buffers make this hard.

Yes, but as I said: if the packets are stored on the wire, there is no
need to buffer them in the router. Large buffers may reduce packet loss
and thus retransmits for TCP connections. The benefit of buffering
depends on aggregation effects due to multiplexing, flow characteristics
(bursty, synchronized behavior etc.) and so on.

>> IMHO, smaller buffers with AQM and ECN are probably a really good
>> thing since dropping fairness can be achieved and flows get once a
>> while packets through to probe for bandwidth and to control their
>> sending speed. Large buffers don't help either if you've got long-term
>> congestion...
> 

As Iljitsch also pointed out: there is some middle ground. For AQM,
e.g., RED, you probably need a larger buffer compared to small buffer
approaches using tail drop: if your threshold is small enough, you
get short queues on average while you still can compensate for transient
short-time bursts due to multiplexing etc. But even then you probably
don't need BDP sized buffers...

Regards,
 Roland