Subject: [ledbat] The pps story

From: Victor Grishchenko <[email-masked]>
Date: Wed, 3 Mar 2010 16:33:35 +0100

Hi!

I think, BitTorrent guys are already perfectly aware of the details of
the recent screw-up, so I'll post it here for archival purposes only.
uTP has a smart feature of sending smaller packets if congestion is
detected:
http://bittorrent.org/beps/bep_0029.html#packet-sizes
The intended purpose is to contribute less delay to congested/slow
links. The problem is, sometimes CPU is the reason for congestion.
That happens to Linux/BSD shapers, cheap home routers and, according
to some field reports, to some bigger hardware, in case it does some
overly intelligent stuff. Once CPU hits the ceiling once, uTP switches
to smaller packets. That further increases the router's CPU load, then
everything goes into downward spiral. In the end, the network is
effectively DDoSed with 150-byte packets (one-tenth the normal size).
Again, according to the field reports, those rogue ISPs tend to
install uTP-kill filters instead of ordering more expensive hardware.
In-depth coverage: http://forum.nag.ru/forum/index.php?showtopic=55025 (Russian)
AMS-IX load graphs show a jump in PPS, but it is unclear whether it
could be attributed to the recent uTP rollout.

That is more like an implementation bug than a general defect of LEDBAT.
So far, from my experience, the LEDBAT itself has problems on...
a) Switched networks: no queue, so no delay, LEDBAT goes into AIMD
mode. May be even more aggressive than TCP, but nobody cares.
b) Long-fat networks: cwnd is determined by the accidental drop rate.
One-way delay has no chance to grow. Not sure what happens if
accidental drop is ignored. Likely, a Bad Thing.
c) In general, when queues are shorter than the target delay.

Yours,

    Victor

---

From: Murari Sridharan <[email-masked]>
Date: Wed, 3 Mar 2010 18:34:32 +0000

Victor, thanks for your comments. 

The small packet problem you mention seems to be interesting. When you say CPU is the source of congestion which CPU are you referring to? Or are you simply saying any system level load will increase latency and will be interpreted as congestion? The framing and switching to smaller packets etc are right now not in the scope of the document although you are making an interesting point. Stas is this something you have seen? I have noticed this type of problem when dealing with delay based congestion control, usually flow control at the switches, interrupt moderation etc tend to show up as delay and that prevents the controller from further increases. However how the sources react is upto the protocol.  When it sees no delay ledbat should only be as aggressive as TCP. 

Thanks

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Victor Grishchenko
Sent: Wednesday, March 03, 2010 7:34 AM
To: [email-masked]
Subject: [ledbat] The pps story

Hi!

Yours,

    Victor
_______________________________________________
ledbat mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ledbat

---

From: Matthew Kaufman <[email-masked]>
Date: Wed, 03 Mar 2010 12:42:57 -0800

Making packets smaller almost never helps these days, in my experience. 
Among other things, far too many of the hops are over layer 2 media with 
huge per-packet overhead. Consider how many byte-length periods are used 
sending a 100 Mbps Ethernet packet of 150 bytes vs 1500 bytes, for instance.

Then inside the routers, the per-packet overhead in tracking buffers is 
also non-zero, so it doesn't help them much either.

In both cases, you'd be better off sending long packets less often (less 
link-layer overhead, less router management overhead, and fewer 
send/receives to interrupt and/or schedule)

Trying to prevent head-of-line blocking by making packets smaller made 
sense on dialup modems, but it really hasn't been useful since.

Matthew Kaufman

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 08 Mar 2010 13:54:32 +0000

Matthew,

>Making packets smaller almost never helps these days, in my experience.

I agree with what I think you meant to say, but not the exact words 
you used. To be clear, I think you mean _increasing the packet rate_ 
never helps. "Making packets smaller" is surely a valid approach if 
you still send at the same packet rate - I believe this approach is 
common for congestion control of video streams - the packets are 
still sent at the same rate, but they are smaller, thus sending less bytes.

>Among other things, far too many of the hops are over layer 2 media 
>with huge per-packet overhead. Consider how many byte-length periods 
>are used sending a 100 Mbps Ethernet packet of 150 bytes vs 1500 
>bytes, for instance.
>
>Then inside the routers, the per-packet overhead in tracking buffers 
>is also non-zero, so it doesn't help them much either.
>
>In both cases, you'd be better off sending long packets less often 
>(less link-layer overhead, less router management overhead, and 
>fewer send/receives to interrupt and/or schedule)

A more general statement of the rule to follow should be "Neither 
increase bit-rate nor packet-rate in response to congestion, because 
you don't know whether congestion is by bits or packets."

There's a draft on pkt vs byte congestion going thru tsvwg at the 
moment. It's mostly about what network equipment should and shouldn't 
do to avoid giving transports perverse incentives to send small 
packets. <http://tools.ietf.org/wg/tsvwg/draft-ietf-tsvwg-byte-pkt-congest/>
Any comments on the recommendations it gives would be gratefully 
received. It is due for a major re-write, as it has become tangled 
due to multiple updates. So if you have any requests for big changes, 
now's a good time.

Cheers

Bob

>Trying to prevent head-of-line blocking by making packets smaller 
>made sense on dialup modems, but it really hasn't been useful since.
>
>Matthew Kaufman
>

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: "Rolf Winter" <[email-masked]>
Date: Mon, 8 Mar 2010 17:43:34 +0100

Hi,

In the environment you are describing I agree. However, LEDBAT is trying to avoid delay building up as a result of the queuing delay in DSL modems in the upstream direction. Here, we today still have very low bandwidth as in DSL the loop length determines rate the modem receives. In rural areas 512kb/s and less is not uncommon (you could argue it will at some point). A single 1500 byte Ethernet frame can then already violate the 25ms delay target. The question is whether that is a scenario that LEDBAT simply does not support, i.e. LEBBAT will not be able to achieve a 25ms delay target, or should there be a mechanism such as adaptive packet sizing to let LEDBAT achieve the 25ms delay target. Such a mechanism is not trivial as flows will independently arrive at a packet size and avoiding inter-flow (un)fairness (there I said it) will be probably tough to solve.

So the question is will the LEDBAT spec include such a mechanism at some point? It seems to be implemented already...

Best,

	Rolf

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014 

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Matthew Kaufman
> Sent: Mittwoch, 3. März 2010 21:43
> To: Murari Sridharan
> Cc: [email-masked]; Victor Grishchenko
> Subject: Re: [ledbat] The pps story
> 
> with
> (less

---

From: Nicholas Weaver <[email-masked]>
Date: Mon, 8 Mar 2010 08:47:11 -0800

> So the question is will the LEDBAT spec include such a mechanism at some point? It seems to be implemented already...

Why not do a packet-pair measurement at the start, to estimate raw channel capacity, and use THAT in factoring the delay target, so the delay target is not 25ms, but the greater of 25ms or 3x the time for a MTU packet?

---

From: "Rolf Winter" <[email-masked]>
Date: Mon, 8 Mar 2010 18:30:16 +0100

Good point. We did that in some tests and a bunch of the home gateways that we tested did not do drop tail for background priority traffic (which is what we are talking about) but something different. Packet pairs did not result in correct measurements which screwed up that idea. But yes, that was our first attempt as well and would work in simple home gateways of if LEDBAT will not be classified as low priority traffic. I could show some results we have if that is of interest.

> Behalf Of Nicholas Weaver
> Sent: Montag, 8. März 2010 17:47
> To: Rolf Winter
> Cc: [email-masked]; Nicholas Weaver; Victor Grishchenko
> > In the environment you are describing I agree. However, LEDBAT is
> trying to avoid delay building up as a result of the queuing delay in
> DSL modems in the upstream direction. Here, we today still have very
> low bandwidth as in DSL the loop length determines rate the modem
> receives. In rural areas 512kb/s and less is not uncommon (you could
> argue it will at some point). A single 1500 byte Ethernet frame can
> then already violate the 25ms delay target. The question is whether
> that is a scenario that LEDBAT simply does not support, i.e. LEBBAT
> will not be able to achieve a 25ms delay target, or should there be a
> mechanism such as adaptive packet sizing to let LEDBAT achieve the 25ms
> delay target. Such a mechanism is not trivial as flows will
> independently arrive at a packet size and avoiding inter-flow
> (un)fairness (there I said it) will be probably tough to solve.
> >
> > So the question is will the LEDBAT spec include such a mechanism at
> some point? It seems to be implemented already...
> 
> Why not do a packet-pair measurement at the start, to estimate raw
> channel capacity, and use THAT in factoring the delay target, so the
> delay target is not 25ms, but the greater of 25ms or 3x the time for a
> MTU packet?
> 
>

---

From: Matthew Kaufman <[email-masked]>
Date: Mon, 08 Mar 2010 10:21:48 -0800

Bob Briscoe wrote:
> common for congestion control of video streams - the packets are still 
> sent at the same rate, but they are smaller, thus sending less bytes.
When I'm thinking LEDBAT, I'm thinking "file transfer", a case where the 
content is of fixed size and you're simply trying to move it from one 
end to the other.

And yes, the rare video scheme with congestion control sometimes works 
this way, but you get even better performance if you make the video 
packets small enough that you can put two of them in a single packet.
>> (less link-layer overhead, less router management overhead, and fewer 
That sounds like a good rule, yes. But there's also "if you are not 
sensing congestion, growing the packet size is better than growing the 
packet rate, all else being equal", at least with the technologies 
currently in use.
> packets. 
> <http://tools.ietf.org/wg/tsvwg/draft-ietf-tsvwg-byte-pkt-congest/>
> received. It is due for a major re-write, as it has become tangled due 
> to multiple updates. So if you have any requests for big changes, 
> now's a good time.
I'll give it a read.

Matthew Kaufman

---

From: Murari Sridharan <[email-masked]>
Date: Mon, 8 Mar 2010 20:05:51 +0000

The idea that the delay target is fixed and not specific to the channel capacity has been brought up in the past meetings. Several people wanted to get more clarity on these "magic" numbers as they characterized it. 

The following is text from the draft. 

   Consider a protocol that uses packets of equal size and acknowledges
   each of them separately.  (Variable-sized packets and delayed
   acknowledgements are possible and are being implemented, but
   complicate the exposition.)

I think it is worth while discussing this further and providing clarity along the lines that Bob mentioned below in terms of byte vs packet congestion and what we recommend to protocols using this controller. We will worry about where such text should go, whether it is within scope at the next meeting. 

Thanks

-----Original Message-----
From: Matthew Kaufman [mailto:[email-masked]] 
Sent: Monday, March 08, 2010 10:22 AM
To: Bob Briscoe
Cc: Murari Sridharan; [email-masked]; Victor Grishchenko
Subject: Re: [ledbat] The pps story

I'll give it a read.

Matthew Kaufman