Subject: [aqm] TCP ACK Suppression

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 6 Oct 2015 09:20:27 +0200 (CEST)

Hi,

after noticing that some TCP ACKs on my home DOCSIS connection were not 
making it to their destination, I after some interaction with cable 
Internet people, I found this:

http://www.cedmagazine.com/article/2006/12/docsis-sub-throughput-optimization

"TCP ACK Suppression (TAS)"

"TCP ACK Suppression overcomes the TRGC limitation without actually 
affecting the DOCSIS specification or involving the CMTS. It improves 
downstream TCP transmissions by taking advantage of TRGC and only sending 
the last ACK it receives when its data grant becomes active. Thus, the 
number of TCP ACKs is fewer, but the number of bytes acknowledged by each 
TCP ACK is increased."

So the DOCSIS modem basically looks at all the ACKs in the queue at the 
time of transmission (DOCSIS uses a "grant" system to tell a modem when 
it's allowed to transmit on the shared medium), and then basically deletes 
all the redundant ACKs (the ones who are just increasing linearly without 
indicating packet drop) and keeps the highest ACK only.

Now, this kind of mechanism, how should it be treated when it comes to 
AQM? This mechanism is basically done at de-queue, when a number of 
packets are emptied from the queue at one time, which is then allowed to 
fill up again until the next transmit opportunity arises.

Or is this a non-problem because it's likely that any AQM employed here 
would use the buffer fill right after a transmit opportunity has finished 
(for those that consider buffer fill as a variable), which would mean that 
most likely the TCP ACK purging had already occured so this mechanism 
doesn't influence the AQM in any significant manner anyway?

Just as a data point from my home connection, I have 250/50 (down/up) and 
when downloading at 250 megabit/s, the upstream traffic is reduced by 
approximately 20x, so instead of sending 10 megabit/s (or so) of ACKs, I 
see approximately 500 kilobits/s of ACKs.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Greg White <[email-masked]>
Date: Tue, 6 Oct 2015 16:35:22 +0000

Mikael,

Specialized upstream TCP ACK handling (which can include both
prioritization and suppression) is a recommended feature in the DOCSIS
specification.  The details of the implementation are left to the
manufacturer, but I don't expect that it is actually done at dequeue
(packet processing at dequeue is expensive in cable modems).  Rather, I
expect that devices identify ACKs at enqueue, and retain (separate from
the main service-flow queue) a single ACK for each TCP session.  Then,
upon receiving a grant, the ACK queue is flushed first, followed by
packets from the main queue.

The CM is not permitted to issue bandwidth requests for more data than it
has available to send, so bandwidth requests would need to already have
ACK suppression taken into account.  For this reason (and the above), I
doubt that the CM would include suppressed ACKs in its queue depth and
queuing latency estimation.

AQM in DOCSIS also happens at enqueue.  The spec is silent on whether the
upstream TCP ACKs are subject to AQM packet drop, but it would be
compliant for them (i.e. the one ACK per session) to be protected.

-Greg

On 10/6/15, 1:20 AM, "aqm on behalf of Mikael Abrahamsson"
<[email-masked] on behalf of [email-masked]> wrote:

>_______________________________________________
>aqm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/aqm

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 7 Oct 2015 09:23:31 +0200 (CEST)

> packets from the main queue.

Sounds reasonable. From my packet dumps it looks like (this is DOCSIS3.0) 
I most of the time get one ACK per 1-4ms, but sometimes I get several ACKs 
per 0.1 ms. It seems to me that the ACK suppression algorithm suppresses 
16 ACKs and then lets the next one through as most of the time, the number 
of bytes that each ACK is ACK:ing is 24616 which divided by 17 is 1448, 
which seems to match nicely with TCP payload size. So in my case, it seems 
it lets 1 packet through, drops 16, lets one packet through, drops 16 etc. 
Sometimes it seems to drop 32 packets though (see packet 1350 below), 
guess it works in iterations of 16 anyway.

http://swm.pp.se/acksuppression-151007.png

> queuing latency estimation.

Ok, so the ACK suppression would happen before the BW request is sent out, 
and after the request is sent out, there is no adding to it? I'm just 
thinking what the reason is that I seem to be seeing several ACKs during 
the same grant (I am guessing that multiple packets seen within 0.1ms is a 
single BW grant). Can CM ask for additional grant during the wait for a 
grant, or CM can only have a single outstanding grant at any given time?

---

From: "LAUTENSCHLAEGER, Wolfram (Wolfram)" <[email-masked]>
Date: Wed, 7 Oct 2015 07:42:01 +0000

Is this specialized upstream TCP ACK handling, particularly the
prioritization a general recommendation in all access technologies? 
Perhaps it should be, since otherwise up and downstream TCP flows interfere
in a crazy queue oscillation that is typically misinterpreted by AQMs.
Is this topic addressed in some RFC already?

Wolfram

> -----Ursprüngliche Nachricht-----
> Von: aqm [mailto:[email-masked]] Im Auftrag von Greg White
> Gesendet: Dienstag, 6. Oktober 2015 18:35
> An: Mikael Abrahamsson; [email-masked]
> Betreff: Re: [aqm] TCP ACK Suppression
> 
> Mikael,
> 
(packet
> processing at dequeue is expensive in cable modems).  Rather, I expect
that
> devices identify ACKs at enqueue, and retain (separate from the main
> service-flow queue) a single ACK for each TCP session.  Then, upon
receiving
> a grant, the ACK queue is flushed first, followed by packets from the main
> queue.
> 
> >http://www.cedmagazine.com/article/2006/12/docsis-sub-throughput-
> optimi
> >zat
> >ion
> >
> >downstream TCP transmissions by taking advantage of TRGC and only
> >sending the last ACK it receives when its data grant becomes active.
> >Thus, the number of TCP ACKs is fewer, but the number of bytes
> >acknowledged by each TCP ACK is increased."
> >
> >it's allowed to transmit on the shared medium), and then basically
> >deletes all the redundant ACKs (the ones who are just increasing
> >linearly without indicating packet drop) and keeps the highest ACK
> >only.
> >
> >packets are emptied from the queue at one time, which is then allowed
> >to fill up again until the next transmit opportunity arises.
> >
> >would use the buffer fill right after a transmit opportunity has
> >finished (for those that consider buffer fill as a variable), which
> >would mean that most likely the TCP ACK purging had already occured so
> >this mechanism doesn't influence the AQM in any significant manner
> >anyway?
> >
> >Just as a data point from my home connection, I have 250/50 (down/up)
> >and when downloading at 250 megabit/s, the upstream traffic is reduced
> >by approximately 20x, so instead of sending 10 megabit/s (or so) of
> >ACKs, I see approximately 500 kilobits/s of ACKs.
> >
> >--

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Wed, 7 Oct 2015 10:15:52 +0200

Hi,

> Is this topic addressed in some RFC already?

Oh, I hope that this is an exception. Such kind of
optimizations may cause a lot of trouble since a link
layer device is interfering with transport layer semantics.
We all know that exactly these kinds of interference
eventually end up in problems with end-to-end transparency and
deployment of new protocol options. At least it interferes with
the ACK clocking expectation of some congestion control algorithms...

Regards,
 Roland

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 7 Oct 2015 12:51:33 +0200 (CEST)

On Wed, 7 Oct 2015, Bless, Roland (TM) wrote:

> Oh, I hope that this is an exception. Such kind of optimizations may 
> cause a lot of trouble since a link layer device is interfering with 
> transport layer semantics. We all know that exactly these kinds of 
> interference eventually end up in problems with end-to-end transparency 
> and deployment of new protocol options. At least it interferes with the 
> ACK clocking expectation of some congestion control algorithms...

Personally, I think you're going to see more and more of this. There are 
mulitple shared access medium where you're allowed to send only part of 
the time, and it's someone else who tells you when you may send.

So if you're sitting there with 100 ACK packets all nicely ACKing 
increasing values indicating no packet loss, and now you're allowed to 
send, why not just kill 99 of those ACK packets?

While I agree with you on principle, these kinds of mechanisms cut the 
amount of ACK traffic down by factor 15-30, meaning I can download at 250 
megabit/s and only have a few hundred kilobit/s of upstream ACKs, instead 
of 5-10 megabit/s of ACKs. That's a huge opimization for any kind of 
asymmetric and/or time slot access medium such as ADSL, LTE, DOCSIS, PON 
and I'm sure there are others.

Btw, what is the reason for TCP doing ACKs for every 2 packets in 
steady-state, even with window sizes in the hundreds of kilobytes or even 
megabytes? I would prefer if the TCP host stack sent fewer ACKs instead of 
trying to optimize this at the access routing device.

---

From: Clark Gaylord <[email-masked]>
Date: Wed, 7 Oct 2015 07:00:58 -0400

For a good time you should see what satellite modems do! It makes sense
since they need enormous buffers, but it's kinda fun to realize the modern
next to you, even though it's not really operating at layer three, has
chosen to effectively terminate your tcp session, ACKing you, and then
buffers to the other modem, who has its own ACK protocol. Similarly on the
other end. This is why non tcp, like pptp is so bad on satellite.

I'm curious to hear the practical implication of docsis on pptp
performance. If you compare a native tcp to a pptp encapsulated tcp, do you
notice a difference?

--
Clark Gaylord
[email-masked]

... thumbed on Android
    auto-correct may have improved this message
    brevity should not be taken to imply curtness  ...

---

From: Steve Bauer <[email-masked]>
Date: Wed, 7 Oct 2015 07:40:46 -0400

> Is this topic addressed in some RFC already?

See:

RFC 3449: TCP Performance Implications of Network Path Asymmetry
https://tools.ietf.org/html/rfc3449

in particular section 5.2

     5.2 TYPE 1: Reverse Link Bandwidth Management ...................19
       5.2.1 ACK Filtering ...........................................20
       5.2.2 ACK Decimation ..........................................21

Steven Bauer
MIT

---

From: "Francini, Andrea (Andrea)" <[email-masked]>
Date: Wed, 7 Oct 2015 15:50:43 +0000

How about the effect of ACK suppression on the growth of the congestion window, for TCP sources where the trigger for window growth is the arrival of an ACK, not the number of bytes it acknowledges? 

The Appropriate Byte Counting (ABC) option (https://tools.ietf.org/html/rfc3465) was addressing a similar issue with delayed ACKs, but it looks like it is no longer available in Linux (http://www.spinics.net/lists/netdev/msg225164.html). 

Is there another workaround already available?

Andrea

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Steve Bauer
Sent: Wednesday, October 07, 2015 7:41 AM
To: LAUTENSCHLAEGER, Wolfram (Wolfram)
Cc: Greg White; [email-masked]
Subject: Re: [aqm] TCP ACK Suppression

See:

Steven Bauer
MIT

---

From: "Richard Scheffenegger" <[email-masked]>
Date: Wed, 7 Oct 2015 17:57:38 +0200

[as individual]

Hi Mikael,

ACK thinning will interfere with the ACK clock of the sender; a good recipie 
to require much more buffering at the bottlneck hop of that session (since 
the sender will start bursting out TCP data packets at it's line rate, 
rather than the bottleneck link rate).

So, if your goal is to have the sender send out TCP data segments at the 
bottleneck rate, you'd not want to ACK too many segments at once...

Of course, the drawback is your observation, that TCP (over Ethernet) 
typically requires 64 bytes of ACK per 2x 1448 bytes, or about 1:45 
uplink/downlink bandwidth).

Others have already pointed out some relevant RFCs in this thread too...

Best regards,
  Richard

----- Original Message ----- 
From: "Mikael Abrahamsson" <[email-masked]>
To: "Bless, Roland (TM)" <[email-masked]>
Cc: "LAUTENSCHLAEGER, Wolfram (Wolfram)" 
<[email-masked]>; "Greg White" 
<[email-masked]>; <[email-masked]>
Sent: Wednesday, October 07, 2015 12:51 PM
Subject: Re: [aqm] TCP ACK Suppression

---

From: Greg White <[email-masked]>
Date: Wed, 7 Oct 2015 16:48:32 +0000

I'm not sure that would be the result in this case.  Here the CM has a
choice of sending (e.g.) 16 back-to-back ACKs, each for 2 segments, or one
ACK for 32 segments.  In either case, 32 segments worth of
acknowledgements will arrive at the sender at once.

-Greg

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Wed, 7 Oct 2015 18:57:45 +0200

Hi,

Am 07.10.2015 um 17:57 schrieb Richard Scheffenegger:
> [as individual]
> 
> Hi Mikael,
> 
> ACK thinning will interfere with the ACK clock of the sender; a good
> recipie to require much more buffering at the bottlneck hop of that
> session (since the sender will start bursting out TCP data packets at
> it's line rate, rather than the bottleneck link rate).

Yes, however, if I understood Mikael correctly, the 100 ACKs are waiting
to be transmitted anyway, so the cable modem link isn't able to provide
feedback at a higher rate, i.e., there is a choice of getting 100 ACKs
transmitted as burst (lets say an ACK train) or a single ACK for the
same amount of data. The effect for congestion control is probably the
same (sender sends out a burst) if the link is the only bottleneck and
the single ACK will not  get lost (i.e., you are throwing away lots of
redundancy that may help in case of packet loss affecting the ACK.

> bottleneck rate, you'd not want to ACK too many segments at once...

Since the cable modem link will lead to clumped ACKs the difference
between sending 100 ACKs vs. 1 ACK is probably not that big...
(except w.r.t. reliability).

Regards,
 Roland

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 10:19:22 -0700 (PDT)

On Wed, 7 Oct 2015, Mikael Abrahamsson wrote:

>> Oh, I hope that this is an exception. Such kind of optimizations may cause 
>> a lot of trouble since a link layer device is interfering with transport 
>> layer semantics. We all know that exactly these kinds of interference 
>> eventually end up in problems with end-to-end transparency and deployment 
>> of new protocol options. At least it interferes with the ACK clocking 
>> expectation of some congestion control algorithms...
>
> mulitple shared access medium where you're allowed to send only part of the 
> time, and it's someone else who tells you when you may send.

it doesn't even require that someone else tells you when you may send. It can 
just be waiting for an available transmit timeslot (Wifi for example)

collapsing multiple ACKs that are going to be sent at once is almost always 
going to be a win.

David Lang

---

From: David Collier-Brown <[email-masked]>
Date: Wed, 7 Oct 2015 13:36:28 -0400

On 07/10/15 01:19 PM, David Lang wrote:
>>> interference eventually end up in problems with end-to-end 
>>> transparency and deployment of new protocol options. At least it 
>>> interferes with the ACK clocking expectation of some congestion 
>>> control algorithms...
>>
>> Personally, I think you're going to see more and more of this. There 
>> are mulitple shared access medium where you're allowed to send only 
>> part of the time, and it's someone else who tells you when you may send.
>
> it doesn't even require that someone else tells you when you may send. 
> It can just be waiting for an available transmit timeslot (Wifi for 
> example)
>
> collapsing multiple ACKs that are going to be sent at once is almost 
> always going to be a win.

I quite agree, but if there is a congestion control implementation "in 
the wild" that assumes it will get a stream of acks, that one's going to 
need some work (:-))

Anyone know if that's the case? The comment above suggest it may be...

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 7 Oct 2015 20:52:53 +0300

> On 7 Oct, 2015, at 20:36, David Collier-Brown <[email-masked]> wrote:
> 
>>>> Oh, I hope that this is an exception. Such kind of optimizations may cause a lot of trouble since a link layer device is interfering with transport layer semantics. We all know that exactly these kinds of interference eventually end up in problems with end-to-end transparency and deployment of new protocol options. At least it interferes with the ACK clocking expectation of some congestion control algorithms...
>>> 
>>> Personally, I think you're going to see more and more of this. There are mulitple shared access medium where you're allowed to send only part of the time, and it's someone else who tells you when you may send.
>> 
>> it doesn't even require that someone else tells you when you may send. It can just be waiting for an available transmit timeslot (Wifi for example)
>> 
>> collapsing multiple ACKs that are going to be sent at once is almost always going to be a win.
> 
> I quite agree, but if there is a congestion control implementation "in the wild" that assumes it will get a stream of acks, that one's going to need some work (:-))
> 

It’s not “in the wild”, but this sort of thing is a headache for ELR.  It essentially means that it won’t be able to use AccECN for its feedback (since AccECN doesn’t allow reconstructing a complex 32-packet history of ECN codepoints from a single ACK), but must introduce some other mechanism to feed back the required data to the sender.

I’m leaning towards moving the ELR-aware congestion control algorithm to the receiver, and having the feedback simply be the requested cwnd; it’s the simplest and most compact solution I can think of, and would work for any congestion control scheme, not just ELR.  The sender may of course maintain its own idea of cwnd in the conventional way, and conservatively take the minimum of the two as effective; this would protect against naively greedy receivers.

I note that CUBIC doesn’t care how many acks it gets, since its scaling curve depends only on time so long as the data flow is not application-limited.  That’s good news for Linux, given the lack of ABC.  Do the BSDs and Windows use ABC?

 - Jonathan Morton

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 11:02:34 -0700 (PDT)

On Wed, 7 Oct 2015, Jonathan Morton wrote:

>>> it doesn't even require that someone else tells you when you may send. It 
>>> can just be waiting for an available transmit timeslot (Wifi for example)
>>> 
>> I quite agree, but if there is a congestion control implementation "in the 
>> wild" that assumes it will get a stream of acks, that one's going to need 
>> some work (:-))
>> 
> It’s not “in the wild”, but this sort of thing is a headache for ELR.  It 
> essentially means that it won’t be able to use AccECN for its feedback (since 
> AccECN doesn’t allow reconstructing a complex 32-packet history of ECN 
> codepoints from a single ACK), but must introduce some other mechanism to feed 
> back the required data to the sender.

but if you are getting all 32 acks at the same time, with no other packets 
flowing, are you going to be able to do anything sane? do you mark every one of 
the acks with ECN (sending a super-strong backoff message), or only mark one of 
the acks (sending a weak backoff message)

David Lang

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 11:05:41 -0700 (PDT)

On Wed, 7 Oct 2015, David Lang wrote:

>> essentially means that it won’t be able to use AccECN for its feedback 
>> (since AccECN doesn’t allow reconstructing a complex 32-packet history of 
>> ECN codepoints from a single ACK), but must introduce some other mechanism 
>> to feed back the required data to the sender.
>
> flowing, are you going to be able to do anything sane? do you mark every one 
> of the acks with ECN (sending a super-strong backoff message), or only mark 
> one of the acks (sending a weak backoff message)

Also, how can you tell how many acks you 'should' get? what happens if the 1500 
byte packets get combined into 9000 byte jumbo packets for the final hop? how 
many acks 'should' you get?

David Lang

---

From: "Richard Scheffenegger" <[email-masked]>
Date: Wed, 7 Oct 2015 21:02:06 +0200

[as one of the AccECN authors]

Hi David,

I believe you are convoluting a few things in the assumptions behind your 
comment...

First, as of now, ACKs are non-ECT; and even if they were marked as ECT by 
the receiver, and subsequently marked CE by the network device when they get 
bundled and thinned (which, by the way, may be the signal required for AckCC 
RFC5690), that would not affect the rate of the data sent to the receiver...

The issue with ACK thinning has most to do with causing a burst of data 
segments that get sent all at once, requiring increased buffering later in 
the network.

And how the ECN-mark fraction or exact sequencing of the data segments is 
conveyed back to the sender by the receiver, can be independent of ACK 
thinning (you may want to read through one suggestion to have better ACK 
loss protection while still being able to fully reconstruct the exact 
sequencing of ECN markings in revision 
https://www.ietf.org/archive/id/draft-kuehlewind-tcpm-accurate-ecn-03.txt 
(section 3.3.4).

So, you have two issues - one of timing (ACK clock eliciting new data to 
send), and one of loss of signal fidelity to deal with, when devices bundle 
up ACKs, or thin out ACKs...

Best regards,
   Richard

----- Original Message ----- 
From: "David Lang" <[email-masked]>
To: "Jonathan Morton" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Wednesday, October 07, 2015 8:02 PM
Subject: Re: [aqm] TCP ACK Suppression

[...]

> (since

but if you are getting all 32 acks at the same time, with no other packets
flowing, are you going to be able to do anything sane? do you mark every one 
of
the acks with ECN (sending a super-strong backoff message), or only mark one 
of
the acks (sending a weak backoff message)

David Lang

--------------------------------------------------------------------------------

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 7 Oct 2015 12:29:27 -0700

> Is there another workaround already available?
Yes. Linux removed abc support b/c it simply does not need that. For
congestion control, Linux always pays attention to how many actual
(mss-sized) data packets are acked or sacked (i.e., delivered), not
how many acks received.

Similarly, it does not count "3" dupacks to trigger fast recovery (in
SACK). It counts how many packets are delivered out of order. So
literally you can compress ack w/ SACK options too.

Having said that, big stretch acks introduce big data bursts in a
ack-clocking sliding window protocol. Hence fq/pacing really helps on
DOCSIS links.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 7 Oct 2015 22:58:15 +0300

Having just re-read that section of AccECN, it is as I remembered - it is
limited in the quantity of information it can reconstruct, and would be
overwhelmed by an ack covering as many as 32 packets.  IIRC, this was also
a part of AccECN that was felt to be too complex to implement widely.

- Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 7 Oct 2015 23:08:03 +0300

> On 7 Oct, 2015, at 21:05, David Lang <[email-masked]> wrote:
> 
>> but if you are getting all 32 acks at the same time, with no other packets flowing, are you going to be able to do anything sane? do you mark every one of the acks with ECN (sending a super-strong backoff message), or only mark one of the acks (sending a weak backoff message)
> 
> Also, how can you tell how many acks you 'should' get? what happens if the 1500 byte packets get combined into 9000 byte jumbo packets for the final hop? how many acks 'should' you get?

Let me turn that around:

What should a node which aggregates 1500-byte packets into jumbo frames set the ECN field to on the resulting jumbo packet, if the ECN fields of the original packets had different values?  I’m having some trouble imagining a scenario where such destructive aggregation is permitted, except for TCP proxies which should act as endpoints anyway.  GSO, in particular, requires the headers to be identical in that respect, ensuring that the original packets can be reconstructed (assuming they are MSS sized).

 - Jonathan Morton

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 14:22:42 -0700 (PDT)

On Wed, 7 Oct 2015, Jonathan Morton wrote:

>> Also, how can you tell how many acks you 'should' get? what happens if the 
>> 1500 byte packets get combined into 9000 byte jumbo packets for the final 
>> hop? how many acks 'should' you get?
>
> Let me turn that around:
>
> What should a node which aggregates 1500-byte packets into jumbo frames set 
> the ECN field to on the resulting jumbo packet, if the ECN fields of the 
> original packets had different values?  I’m having some trouble imagining a 
> scenario where such destructive aggregation is permitted, except for TCP 
> proxies which should act as endpoints anyway.  GSO, in particular, requires 
> the headers to be identical in that respect, ensuring that the original 
> packets can be reconstructed (assuming they are MSS sized).

TCP data is supposed to be logically a continuous stream, it should not matter 
to the endpoints how it is packetized. So a conversion to/from jumbo frames 
should be acceptable.

David Lang

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 14:28:25 -0700 (PDT)

> RFC5690), that would not affect the rate of the data sent to the receiver...

fair enough.

> segments that get sent all at once, requiring increased buffering later in 

The point here is that the ACKs are going to all arrive at the same time anyway, 
so it doesn't matter if 32 ack packets arrive at wire speed, or just one packet 
arrives that acks all the data, in either case the data that is held up waiting 
for acks is going to be sent at the same time (if anything, avoiding the delay 
in receiving and processing the extra acks would decrease latency)

> thinning (you may want to read through one suggestion to have better ACK loss 
> protection while still being able to fully reconstruct the exact sequencing 
> of ECN markings in revision 

The issue here isn't that anything delays acks in order to bundle or thin them 
out, but that other criteria result in the acks sitting in one queue at some 
point in time. If they are already together like that, it makes sense to thin 
them.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 14:15:05 -0700

On 10/7/2015 12:42 AM, LAUTENSCHLAEGER, Wolfram (Wolfram) wrote:
...
> Is this topic addressed in some RFC already?

It's a direct violation of RFC793, which expects one ACK for every two
segments:

4.2 Generating Acknowledgments

   The delayed ACK algorithm specified in [Bra89] SHOULD be used by a
   TCP receiver.  When used, a TCP receiver MUST NOT excessively delay
   acknowledgments.  Specifically, an ACK SHOULD be generated for at
   least every second full-sized segment, and MUST be generated within
   500 ms of the arrival of the first unacknowledged packet.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 14:27:27 -0700

CC'ing TCPM on my responses to this thread...

On 10/7/2015 8:50 AM, Francini, Andrea (Andrea) wrote:
> How about the effect of ACK suppression on the growth of the
> congestion window, for TCP sources where the trigger for window growth
> is the arrival of an ACK, not the number of bytes it acknowledges?
> 
> The Appropriate Byte Counting (ABC) option
> (https://tools.ietf.org/html/rfc3465) was addressing a similar issue
> with delayed ACKs, but it looks like it is no longer available in Linux
> (http://www.spinics.net/lists/netdev/msg225164.html).

Even with ABC, killing off some of the ACKs would result in sender
bursts because you're interfering with ACK clocking -- unless ABC were
coupled with some sort of rate pacing.

Note also that when you kill off intermediate ACKs you're increasing the
round-trip delay. The sender can't start sending what you ACK until the
ACK arrives; if you kill off earlier ACKs, then the later ACK will (by
definition) arrive later.

This is bad on many levels.

Joe

---

From: Greg White <[email-masked]>
Date: Thu, 8 Oct 2015 21:30:50 +0000

>definition) arrive later.

As discussed previously in this thread, the specific case where this has
been deployed does not suffer from these problems.

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 14:31:57 -0700 (PDT)

>   500 ms of the arrival of the first unacknowledged packet.

actually, this is only a violation of the SHOULD section, not the MUST section.

And if the Ack packets are going to arrive at wire-speed anyway (due to other 
causes), is there really an advantage to having 32 ack packets arriving one 
after the other instead of making it so that the first ack packet (which arrives 
at the same time) can ack everything?

And if there is such an advantage, does it outweight the disadvantages that the 
extra ack packets cause by causing highly asymmetric links to be overloaded and 
drop packets?

David Lang

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 14:46:21 -0700

> actually, this is only a violation of the SHOULD section, not the MUST

When you violate a SHOULD, you need to have a good reason that applies
in a limited subset of cases.

"it benefits me" isn't one of them, otherwise the SHOULD would *always*
apply.

> And if the Ack packets are going to arrive at wire-speed anyway (due to
> other causes), is there really an advantage to having 32 ack packets
> arriving one after the other instead of making it so that the first ack
> packet (which arrives at the same time) can ack everything?

If the first ACK confirms everything, you're giving the endpoint a false
sense of how fast the data was received. This is valid only if the
*last* ACK is the only one you retain, but then you'll increase delay.

Unless you know that the endpoint supports ABC and pacing, yes, there's
a very distinct advantage to getting 32 ACKs rather than 1. It also
helps with better accuracy on the RTT calculation, which is based on
sampling (and you've killed 97% of the samples).

> And if there is such an advantage, does it outweight the disadvantages
> that the extra ack packets cause by causing highly asymmetric links to
> be overloaded and drop packets?

Why is it so bad to drop packets? Cumulative ACKs will still work
properly in the presence of the losses. If you make reasonable progress,
you have to determine whether the lost ACKs should make you slow down.

AFAICT, this isn't horrible if you KNOW something specific about the
other end (ABC + pacing), but how can you know? In a closed deployment,
maybe. But this IMO is over-optimizing TCP for a very specific environment.

TCP isn't supposed to be the most efficient in EVERY corner case. It's
supposed to *always work* in EVERY corner case.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 8 Oct 2015 14:52:05 -0700

> packet (which arrives at the same time) can ack everything?

I would like to add that GRO also cause stretched ACKs (acking up to 64KB),
and GRO is crucial to reduce cycles for +10Gbps transfers on some
architectures.

> that the extra ack packets cause by causing highly asymmetric links to be
> overloaded and drop packets?
>
> David Lang
>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 15:10:42 -0700

The fact that you need to do something to go fast does not make it safe
for everyone else.

Again, TCP is intended to work everywhere, not to work well anywhere.

Joe

On 10/8/2015 2:52 PM, Yuchung Cheng wrote:
> 
> 
> On Thu, Oct 8, 2015 at 2:31 PM, David Lang <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     And if the Ack packets are going to arrive at wire-speed anyway (due
>     to other causes), is there really an advantage to having 32 ack
>     packets arriving one after the other instead of making it so that
>     the first ack packet (which arrives at the same time) can ack
>     everything?
> 
> 
> I would like to add that GRO also cause stretched ACKs (acking up to
> 64KB), and GRO is crucial to reduce cycles for +10Gbps transfers on some
> architectures.
> 
> 
>     And if there is such an advantage, does it outweight the
>     disadvantages that the extra ack packets cause by causing highly
>     asymmetric links to be overloaded and drop packets?
> 
>     David Lang
> 
> 
>     [email-masked] <mailto:[email-masked]>

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 15:29:44 -0700 (PDT)

> *last* ACK is the only one you retain, but then you'll increase delay.

why does it give the server a false sense of how fast the data was received? the 
packets don't have timestamps that the server can trust, they are just packets 
arriving. And if the server concludes something different from 32 packets 
arriving, each acking 2 packet, but all arriving one after the other at it's 
wire speed (let's say it's a slow network, only Gig-E) compared to a single 
packet arriving that acks 64 packets of data at once, it's doing something very 
strange and making assumptions about how the network works that are invalid.

> sampling (and you've killed 97% of the samples).

the 97% of the samples that I've killed would be producing invalid data for your 
calculation because they were delayed in returning. The one packet I'd deliver 
would be accurate for the last data it acks, the others would all give 
misleading info.

> Why is it so bad to drop packets?

because forcing packets for other services to be dropped to make room for acks 
degrades those other services.

>  Cumulative ACKs will still work

the problem isn't if acks are dropped, it's when the acks fill the available 
bandwidth so that other services get dropped.

> supposed to *always work* in EVERY corner case.

I don't see how it fails to work in this case. As people have pointed out, some 
cable routers have been doing this for 15 years and the Internet has not 
imploded from it yet, so the drawbacks of dropping these already-delayed and 
redundant ack packets cannot be the end-of-the-internet that you are painting it 
to be.

We are talking about only doing this in one specific case, the case where other 
things have already caused some of the acks to be delayed to the point where 
later acks have 'caught up' with them on the network and both early and late 
acks are sitting in the same queue on the same device waiting to be sent at the 
same time.

At this point there are three possiblilities

1. all the acks get sent back-to-back, wasting bandwith with their redundancy

2. send only the newest ack, trashing all the ones that would be redundant

3. the total of the acks that are queued exceeds the next transmit window, so 
only some of the acks get sent, the newest one doesn't and gets delayed further.

we know that #2 doesn't break the Internet, it's within the range of responses 
permitted by the RFC SHOULD. It decreases load on congested links.

But you keep insisting that it's a horrible thing to consider doing.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 16:23:38 -0700

On 10/8/2015 3:29 PM, David Lang wrote:
> why does it give the server a false sense of how fast the data was
> received? the packets don't have timestamps that the server can trust,
> they are just packets arriving.

Well, the only reason we can no longer trust them is that an
intermediate device has tampered with them.

See, this is the problem - the DOCSIS modem wants to do what *it* wants,
assuming everyone else plays by the rules, but it doesn't care whether
it violates the assumptions other parties are making.

That's an example of "tragedy of the commons".

> And if the server concludes something
> different from 32 packets arriving, each acking 2 packet, but all
> arriving one after the other at it's wire speed (let's say it's a slow
> network, only Gig-E) compared to a single packet arriving that acks 64
> packets of data at once, it's doing something very strange and making
> assumptions about how the network works that are invalid.

Says who? The RFCs say that this assumption SHOULD be reasonable.

> the 97% of the samples that I've killed would be producing invalid data
> for your calculation because they were delayed in returning.

Why do you think that is invalid data? That's an accurate measure of the
return path of the ACK stream.

...
> because forcing packets for other services to be dropped to make room
> for acks degrades those other services.

Sure, but remember that we're not here to support the cable company's
business model. They deployed networks that had severely
underprovisioned backchannels so they could use shared channels rather
than routers one step lower in the hierarchy. Now they pull this stunt
so they can fix what's broken with their provisioning model.

The trouble is that it has effects for others in the network, not just
the cable company.

...
> I don't see how it fails to work in this case. As people have pointed
> out, some cable routers have been doing this for 15 years and the
> Internet has not imploded from it yet, so the drawbacks of dropping
> these already-delayed and redundant ack packets cannot be the
> end-of-the-internet that you are painting it to be

Oh, right. That argument. We haven't seen it break anything, so it
*must* be safe.

What would you see if it were broken? Maybe hosts that burst into the
net and caused router buffers to overload? Hmmm.

> We are talking about only doing this in one specific case, the case
> where other things have already caused some of the acks to be delayed to
> the point where later acks have 'caught up' with them on the network and
> both early and late acks are sitting in the same queue on the same
> device waiting to be sent at the same time.

They're in a queue. That means the early ones go out before the late
ones. You have two choices if you coalesce their information:

a) delete the early ACKs

	Oh, but you wouldn't do *that* because it would hit *your*
	customers with a higher delay.

b) delete the late ACKs and alter the early ones

	Giving your customers a false sense of how fast their
	data was getting there. Roadrunner pulled stunts like this
	in the early 90's too. It's not exactly news.

> 1. all the acks get sent back-to-back, wasting bandwith with their

That's not a waste; that's information.

> 2. send only the newest ack, trashing all the ones that would be redundant

If you wait to send it last, maybe... but then you're still encouraging
the receiver to burst its next transmissions. We already know that sort
of bursting causes problems (even if *we* don't see them, someone does).

> 3. the total of the acks that are queued exceeds the next transmit
> window, so only some of the acks get sent, the newest one doesn't and
> gets delayed further.
> 
> 
> we know that #2 doesn't break the Internet, 

No, you really don't. What you know is that #2 is cheap and benefits
you. Everyone continually doing that *will* break the Internet.

> it's within the range of
> responses permitted by the RFC SHOULD.

SHOULD means that breaking it needs to be done for a reason. I've long
argued that the SHOULD should never be there in the first place without
explaining why it isn't a MUST or a MAY and the conditions under which
it might be appropriate to violate it. RFC793 doesn't have that context,
unfortunately, but it doesn't mean that any - and every - SHOULD is
intended to be willfully ignored at all times.

> It decreases load on congested links.
                       ^^^^^^^^^
severely underprovisioned

> But you keep insisting that it's a horrible thing to consider doing.

The tragedy of the commons is a horrible thing. Just because something
doesn't hurt you or you can't see how it hurts others doesn't mean there
isn't a problem.

I've outlined the reasons why this is bad - basically it works only
under the assumption that DOCSIS modems get to play by their own rules
and every one else plays fair. If that's not a bad idea, I don't know
what is.

Joe

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 17:04:48 -0700 (PDT)

> intermediate device has tampered with them.

no, you could not trust any timestamps in the packets even if nothing changes 
the packets between endpoints.

> return path of the ACK stream.

so how do you sanely conclude anything from 32 ack packets arriving at wire 
speed back-to-back?

> The trouble is that it has effects for others in the network, not just

It's not just cable companies. the same sort of thing will happen with 
half-duplex wifi links where acks will accumulate while data packets are flowing 
in the other direction.

stop trying to say that this is the fault of one subset of industry and 
recognize that there are lots of legitimate reasons for this.

Highly asymmetric links are not just 'cable companies underprovisioning their 
networks'. DSL lines are highly asymmetric due to the difference in the cost of 
the transmitters on each end of the link. As are Satellite IP systems, etc.

> net and caused router buffers to overload? Hmmm.

that happens without this, so you can't blame it on the missing acks.

> 	customers with a higher delay.

but by not having to transmit the early acks, the later ack goes out faster, so 
the customers get less of a delay in getting the data they have received acked.

If the only thing in the queue is acks, then the last ack in the queue goes out 
as fast as the first ack would. By doing this you transmit less, which can speed 
up the network overall as the next station can transmit it's data faster 
(thinking of wifi as an example)

If there are other packets in the queue, you still can delete all the acks 
except the last one that will fit into the burst with no degredation in how fast 
data is acknoleged (and you increase the amount of usable data that is sent in 
that timeslot instead of wasting it on redundant ack packets)

> 	in the early 90's too. It's not exactly news.

unless there are other packets in the flow that the ack is jumping, I see no 
problem with this. You aren't sending out an ack before the data is arrived, you 
just aren't delaying the last ack unneccessarily.

> That's not a waste; that's information.

very low value information at best.

> of bursting causes problems (even if *we* don't see them, someone does).

how would the burst be any different if the server gets 32 acks back to back vs 
1 ack that covers everything. The additional acks aren't going to get there any 
faster than the single one would.

> you. Everyone continually doing that *will* break the Internet.

you keep stating that, but you are short on details about why a string of acks 
at wire speed is better than a single ack covering the same data 'because I say 
so' doesn't cut it.

> severely underprovisioned

no, merely congested for some reason. Any shared media will have the same 
situation, when another station is transmitting, a queue builds.

> and every one else plays fair. If that's not a bad idea, I don't know

you say that it breaks timing assumptions and calculations, but then don't 
explain how the train of acks arriving at wire speed would let your calculations 
be any more accurate.

David Lang

---

From: Christian Huitema <[email-masked]>
Date: Fri, 9 Oct 2015 00:21:52 +0000

All that discussion is fun, but we should be careful to stay within the respective group charters.

As far as AQM is concerned, the question is whether the group wants to standardize some kind of special handling of TCP ACK as part of queue management. As far as the IETF rules are concerned, the answer is clearly NO -- we cannot create an IETF recommendation that breaks other IETF recommendations. Besides, such rules would not be very effective if the transport protocol is encrypted, as for example QUIC, or SCTP over DTLS. AQM should certainly not depend on end-systems not using encryption.

As for TCP and other protocols, the question is whether they should pay more attention to the volume of ACK and other control packets. The deployment of queue management systems like FQ-CODEL actually creates an incentive to do that, because a transport protocol that creates congestion on its uplink will be automatically penalized. But that discussion belongs in TCPM and other transport working groups, not AQM.

-- Christian Huitema

---

From: Rick Jones <[email-masked]>
Date: Thu, 8 Oct 2015 17:36:31 -0700

> 64KB), and GRO is crucial to reduce cycles for +10Gbps transfers on some

And they go back even further than GRO (or LRO) appearing in Linux. 
They go all the way back to the mid/late 1990s when HP-UX 11 and Solaris 
2 started shipping their cousin (Mentat origin) stacks with ACK 
avoidance heuristics in them.  Perhaps even before, but that is the 
earliest I know of.

GRO, TSO, CKO etc have come into being from the fields made fertile by 
the combination of ever increasing link speeds, never increasing de jure 
Ethernet MTUs, no longer increasing processor core frequency, and paying 
customer demand for performance.

happy benchmarking,

rick jones

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 17:42:33 -0700 (PDT)

On Fri, 9 Oct 2015, Christian Huitema wrote:

> As far as AQM is concerned, the question is whether the group wants to 
> standardize some kind of special handling of TCP ACK as part of queue 
> management. As far as the IETF rules are concerned, the answer is clearly NO 
> -- we cannot create an IETF recommendation that breaks other IETF 
> recommendations. Besides, such rules would not be very effective if the 
> transport protocol is encrypted, as for example QUIC, or SCTP over DTLS. AQM 
> should certainly not depend on end-systems not using encryption.

AQM should not depend on end-systems not using encryption.

But that doesn't mean that AQM should not take advantage of additional data that 
it can get from non-encrypted sessions.

For example, in the fq_codel/cake development, we're finding that there are some 
transports that bundle very large numbers of packets together to send at one 
time in order to maximize the transport bandwidth. (for example, 4x4 wifi sends 
a LOT of data in one transmit timeslot). Treating that large aggregate as a 
single packet seriously hurts fairness and latency on the next hop. So 'pulling 
apart' this aggregate into the individual packets/streams and making decisions 
based on the pieces ends up being a serious win in fairness and latency.

> As for TCP and other protocols, the question is whether they should pay more 
> attention to the volume of ACK and other control packets. The deployment of 
> queue management systems like FQ-CODEL actually creates an incentive to do 
> that, because a transport protocol that creates congestion on its uplink will 
> be automatically penalized. But that discussion belongs in TCPM and other 
> transport working groups, not AQM.

I can't say where the discussion needs to take place, but people need to realize 
that it's not just hurting itself. In fact it's probably not going to hurt 
itself the most.

most AQM systems tend to (at least slightly) prioritize acks, DNS responses, and 
other small packets because letting them get delayed has such a disproportionate 
impact on the overall throughput and user experience. As a result, the overload 
of ACKs is probably going to end up impacting other uses of the link instead.

Suggesting that the queues that build up produce a special enough case to 
consider thinning out the duplicate acks is a far cry from 'making a 
recommendation that breaks other recommendations'

David Lang

---

From: Christian Huitema <[email-masked]>
Date: Fri, 9 Oct 2015 02:08:37 +0000

On Thursday, October 8, 2015 5:43 PM, David Lang wrote:
> ...
> For example, in the fq_codel/cake development, we're finding that there are
> some transports that bundle very large numbers of packets together to send
> at one time in order to maximize the transport bandwidth. (for example, 4x4
> wifi sends a LOT of data in one transmit timeslot). Treating that large
> aggregate as a single packet seriously hurts fairness and latency on the next
> hop. So 'pulling apart' this aggregate into the individual packets/streams and
> making decisions based on the pieces ends up being a serious win in fairness
> and latency.

Define "bundle" please. If they are making sure that several IP packets are sent back to back in a single Wi-Fi slot, then it is of course perfectly fine for AQM to handle the IP packets one by one. Does 4x4 Wi-Fi do something else?

> recommendation that breaks other recommendations'

That definitely contradicts the TCP specs. So it is very much in "don't go there" territory...

-- Christian Huitema

---

From: David Collier-Brown <[email-masked]>
Date: Thu, 8 Oct 2015 22:09:25 -0400

One short interjection: send at least two acks, separated by a time 
based on available packets and/or average noise-burst durations on links 
with noise problems...

On 08/10/15 08:04 PM, David Lang wrote:
>>>>> actually, this is only a violation of the SHOULD section, not the 
>>>>> MUST
>>>>> section.
>>>>
> no, you could not trust any timestamps in the packets even if nothing 
> changes the packets between endpoints.
>
> so how do you sanely conclude anything from 32 ack packets arriving at 
> wire speed back-to-back?
>
>> ...
>>>>> disadvantages
> half-duplex wifi links where acks will accumulate while data packets 
> are flowing in the other direction.
>
> stop trying to say that this is the fault of one subset of industry 
> and recognize that there are lots of legitimate reasons for this.
>
> Highly asymmetric links are not just 'cable companies 
> underprovisioning their networks'. DSL lines are highly asymmetric due 
> to the difference in the cost of the transmitters on each end of the 
> link. As are Satellite IP systems, etc.
>
> but by not having to transmit the early acks, the later ack goes out 
> faster, so the customers get less of a delay in getting the data they 
> have received acked.
>
> If the only thing in the queue is acks, then the last ack in the queue 
> goes out as fast as the first ack would. By doing this you transmit 
> less, which can speed up the network overall as the next station can 
> transmit it's data faster (thinking of wifi as an example)
>
> If there are other packets in the queue, you still can delete all the 
> acks except the last one that will fit into the burst with no 
> degredation in how fast data is acknoleged (and you increase the 
> amount of usable data that is sent in that timeslot instead of wasting 
> it on redundant ack packets)
>
> unless there are other packets in the flow that the ack is jumping, I 
> see no problem with this. You aren't sending out an ack before the 
> data is arrived, you just aren't delaying the last ack unneccessarily.
>
> how would the burst be any different if the server gets 32 acks back 
> to back vs 1 ack that covers everything. The additional acks aren't 
> going to get there any faster than the single one would.
>
> you keep stating that, but you are short on details about why a string 
> of acks at wire speed is better than a single ack covering the same 
> data 'because I say so' doesn't cut it.
>
>>> it's within the range of
> no, merely congested for some reason. Any shared media will have the 
> same situation, when another station is transmitting, a queue builds.
>
> you say that it breaks timing assumptions and calculations, but then 
> don't explain how the train of acks arriving at wire speed would let 
> your calculations be any more accurate.
>
> David Lang
>

---

From: Simon Barber <[email-masked]>
Date: Thu, 08 Oct 2015 19:26:45 -0700

I like this suggestion - but it's only needed in the case where there 
are dup acks.

Simon

On 10/8/2015 7:09 PM, David Collier-Brown wrote:
> based on available packets and/or average noise-burst durations on 
> links with noise problems...
>
>
>> so how do you sanely conclude anything from 32 ack packets arriving 
>> at wire speed back-to-back?
>>
>>> ...
>> It's not just cable companies. the same sort of thing will happen 
>> with half-duplex wifi links where acks will accumulate while data 
>> packets are flowing in the other direction.
>>
>> underprovisioning their networks'. DSL lines are highly asymmetric 
>> due to the difference in the cost of the transmitters on each end of 
>> the link. As are Satellite IP systems, etc.
>>
>> If the only thing in the queue is acks, then the last ack in the 
>> queue goes out as fast as the first ack would. By doing this you 
>> transmit less, which can speed up the network overall as the next 
>> station can transmit it's data faster (thinking of wifi as an example)
>>
>> amount of usable data that is sent in that timeslot instead of 
>> wasting it on redundant ack packets)
>>
>> you keep stating that, but you are short on details about why a 
>> string of acks at wire speed is better than a single ack covering the 
>> same data 'because I say so' doesn't cut it.
>>
>>>> it's within the range of

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 21:03:00 -0700 (PDT)

On Fri, 9 Oct 2015, Christian Huitema wrote:

> Define "bundle" please. If they are making sure that several IP packets are 
> sent back to back in a single Wi-Fi slot, then it is of course perfectly fine 
> for AQM to handle the IP packets one by one. Does 4x4 Wi-Fi do something else?

I don't remember the details from the discussion, but the combined bundle 
required extra steps to pull apart to get at the individual packets. IIRC, not 
doing so ended up with multi-MB chunks of data to be delivered, which blocked 
all other traffic while it was being delivered.

> That definitely contradicts the TCP specs. So it is very much in "don't go there" territory...

By 'not going there' you are crippling people's networks for the sake of 
following a spec. Rather than following the letter of the old spec, we should be 
looking at the reasons for it, and reasons to make exceptions. There is a long 
history of introducing new things that break the old way of doing things, from 
breaking "classful" network routing to Anycast, there are lots of things that 
"broke" the old way of doing things.

In this case there is more than a decade of people doing exactly what shouldn't 
even be considered.

I'll ask yet again, if acks have already been delayed so that they will be 
delivered at the same time as later acks, how much value do they actually 
provide? We need to compare whatever value this is against the cost of the 
misinformation that they provide, and the impact on other traffic.

David Lang

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 21:06:38 -0700 (PDT)

On Thu, 8 Oct 2015, David Collier-Brown wrote:

> One short interjection: send at least two acks, separated by a time based on 
> available packets and/or average noise-burst durations on links with noise 
> problems...

When you say send two acks, are you talking about doing so in case one gets lost 
on the network?

If so, that seems like a reasonable thing to do, if you have a bunch of acks in 
the queue, collapse them and put a copy of the resulting ack at the beginning of 
the non-transmitted queue. If there is no other data, it will be sent in the 
next available slot. If there is other traffic, it can get absorbed into 
additional acks that arrive before the next transmit slot and vanish (with the 
new ack serving the purpose)

David Lang

>> half-duplex wifi links where acks will accumulate while data packets are 
>> flowing in the other direction.
>> 
>> Highly asymmetric links are not just 'cable companies underprovisioning 
>> their networks'. DSL lines are highly asymmetric due to the difference in 
>> the cost of the transmitters on each end of the link. As are Satellite IP 
>> systems, etc.
>> 
>> faster, so the customers get less of a delay in getting the data they have 
>> received acked.
>> 
>> If the only thing in the queue is acks, then the last ack in the queue goes 
>> out as fast as the first ack would. By doing this you transmit less, which 
>> can speed up the network overall as the next station can transmit it's data 
>> faster (thinking of wifi as an example)
>> 
>> except the last one that will fit into the burst with no degredation in how 
>> fast data is acknoleged (and you increase the amount of usable data that is 
>> sent in that timeslot instead of wasting it on redundant ack packets)
>> 
>> unless there are other packets in the flow that the ack is jumping, I see 
>> no problem with this. You aren't sending out an ack before the data is 
>> arrived, you just aren't delaying the last ack unneccessarily.
>> 
>> how would the burst be any different if the server gets 32 acks back to 
>> back vs 1 ack that covers everything. The additional acks aren't going to 
>> get there any faster than the single one would.
>> 
>> you keep stating that, but you are short on details about why a string of 
>> acks at wire speed is better than a single ack covering the same data 
>> 'because I say so' doesn't cut it.
>> 
>>>> it's within the range of
>> explain how the train of acks arriving at wire speed would let your 
>> calculations be any more accurate.
>> 
>> David Lang
>>

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 21:08:53 -0700 (PDT)

Well, the only situation we are talking about is when an AQM function finds that 
there are multiple acks for one TCP session in it's queue. The dispute is if 
these acks are can be considered 'duplicates' (since the later ones will ack 
data for the earlier ones) and so can be 'deduped' or if there is a significant 
amount of value in sending every ack through anyway.

David Lang

On Thu, 8 Oct 2015, Simon Barber wrote:

> I like this suggestion - but it's only needed in the case where there are dup 
> acks.
>
> Simon
>
>> One short interjection: send at least two acks, separated by a time based 
>> on available packets and/or average noise-burst durations on links with 
>> noise problems...
>> 
>> 
>>>>>>> to
>>> goes out as fast as the first ack would. By doing this you transmit less, 
>>> which can speed up the network overall as the next station can transmit 
>>> it's data faster (thinking of wifi as an example)
>>> 
>>> except the last one that will fit into the burst with no degredation in 
>>> how fast data is acknoleged (and you increase the amount of usable data 
>>> that is sent in that timeslot instead of wasting it on redundant ack 
>>> packets)
>>>

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 9 Oct 2015 10:04:19 -0700

On Thu, Oct 8, 2015 at 9:03 PM, David Lang <[email-masked]> wrote:

>> Define "bundle" please. If they are making sure that several IP packets
>> are sent back to back in a single Wi-Fi slot, then it is of course
>> perfectly fine for AQM to handle the IP packets one by one. Does 4x4 Wi-Fi
>> do something else?
>>
>
> required extra steps to pull apart to get at the individual packets. IIRC,
> not doing so ended up with multi-MB chunks of data to be delivered, which
> blocked all other traffic while it was being delivered.
>
> following a spec. Rather than following the letter of the old spec, we
> should be looking at the reasons for it, and reasons to make exceptions.
> There is a long history of introducing new things that break the old way of
> doing things, from breaking "classful" network routing to Anycast, there
> are lots of things that "broke" the old way of doing things.
>
> In this case there is more than a decade of people doing exactly what
> shouldn't even be considered.
>

Does DOCSIS suppress/filter ACK with SACK blocks and/or ACKs with
timestamps?

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 10:13:24 -0700

On 10/9/2015 10:04 AM, Yuchung Cheng wrote:
> There is a long history of introducing new things that break the old way
> of doing things, from breaking "classful" network routing to Anycast,
> there are lots of things that "broke" the old way of doing things.

There's a useful less on here:

Van Jacobson, in an unpublished appendix to his Sigcomm 1988 paper on
congestion control, discussed how TCP should restart after idle, and
that this would require a "time since last sent" timer. A footnote to
that text noted that there was already a "time since last received"
timer, and that - 'since TCP traffic is typically symmetric', it would
be reasonable to use that timer for slow-start restart after idle.

A few years later, when developing HTTP, Tim Berners-Lee said that he
didn't want to use FTP for file transfers because it would open two
connections for one file from one location (one for control, one for the
file). He said that was heavyweight, and that (paraphrasing), 'most of
the time, we'll only get one file from a location'. So we ended up with TCP.

When the web became more sophisticated, users wanted many files from one
server for each page visited (images, formatting, etc.). What we ended
up with was burst after idle for HTTP traffic.

I.e., there are a lot of things there for a reason. Yes, nothing is
immutable, but simply claiming "I need it now" and "nobody has
complained" isn't considered a reasonable way to deploy changes in the wild.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 10:58:50 -0700

Dave in particular wanted some specific reasons this is a bad idea as
presented. Here is my summary.

The rest of this discussion should happen on TCPM.

Joe

---

1) *you* shouldn't be using a mechanism that destroys information for others

	whether the timestamps or ACK stream spacing has any
	meaning is for the receiver - not you - to decide

2) *you* don't know where your mechanism will have an impact

	those clumped ACKs might be spaced out further
	downstream

	even if you have a rule of "I'll only gather 3 ACKs",
	you can't know if another box - including yours -
	downstream might gather those composite ACKs further

3) you claim this might be safe *if* AQM is widely deployed

	but *you* don't appear to be making that determination
	*before* deploying your approach

	also, AQM is in the opposite direction, and unless
	you deploy this with enough state to track the fact
	that your box is seeing traffic in both directions,
	you shouldn't be turning it on at all

As others have noted, the SHOULD of ACK requirements can be exceeded,
but only with careful consideration of the potential impact. That
careful consideration does not consist of "I turned it on and I didn't
hear anyone scream".

---

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 11:16:15 -0700

> customer demand for performance.

Right, and we're just starting to see places where they interfere with
TCP. I've been doing tests on a new TCP option to extend the space
available for options.

We've already seen cases where these extensions aggregate segments that
have different unknown options (which should clearly never happen), or
even when they aggregate segments with unknown options that match (which
I believe should never happen either).

Again, just because nobody's screamed to date doesn't mean these are
known safe.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 11:45:54 -0700 (PDT)

> Does DOCSIS suppress/filter ACK with SACK blocks and/or ACKs with

I don't know, but this discussion is not limited to DOCIS.

also, see the other thread covering the RFC discussion. There's a lot more 
details there and in the RFC

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 11:48:27 -0700 (PDT)

> complained" isn't considered a reasonable way to deploy changes in the wild.

similarly, it's not right to assume that the conditions haven't changed.

In this case, we have a changed situation. The ACKs are _not_ going to flow 
at their 'natural' rate. That assumption has already failed. What we are 
discussion is the impact of these changes in the face of this broken assumption.

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 11:52:04 -0700 (PDT)

> 	meaning is for the receiver - not you - to decide

The temporal spacing is already lost.

> 	you shouldn't be turning it on at all

this discussion started in the context of AQM. AQM is needed in both directions, 
it's not a one-direction only thing.

> careful consideration does not consist of "I turned it on and I didn't

that's not the only consideration here. There's also the consideration that the 
ACK stream is already badly distorted.

As it turns out (as discussed on the other thread), the RFC actually recommends 
what we was proposed (with some additional nuances)

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 11:56:02 -0700

On 10/9/2015 11:48 AM, David Lang wrote:
> In this case, we have a changed situation. The ACKs are _not_ going to
> flow at their 'natural' rate. That assumption has already failed. What
> we are discussion is the impact of these changes in the face of this
> broken assumption.

Discuss first, deploy later.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 11:58:39 -0700

> The temporal spacing is already lost.

You don't know that won't be restored downstream.

> this discussion started in the context of AQM. AQM is needed in both
> directions, it's not a one-direction only thing.

Needed, but you don't *know* it's deployed on both path directions. The
reverse path could be taking a different route.

> that's not the only consideration here. There's also the consideration
> that the ACK stream is already badly distorted.

It's not distorted at all - it's queued up and waiting its turn.

> As it turns out (as discussed on the other thread), the RFC actually
> recommends what we was proposed (with some additional nuances)

Assuming that you know what you can't know and assuming that you don't
care whether your actions affect what others have a right to assume.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 12:29:53 -0700 (PDT)

> Discuss first, deploy later.

Rough Concensus and Working Code

How can you tell if it actually works if you don't try it :-)

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 12:48:39 -0700 (PDT)

> You don't know that won't be restored downstream.

it can only be restored by further delaying the arrival of the ACKs (you can't 
send an 'fixed' ACK before you receive the one that was delayed (I'm trying to 
avoid the phrase 'Delayed ACK' to avoid confusion)

This will have further detrimental effects on your throughput.

> reverse path could be taking a different route.

AQM is needed on any bottleneck link. If the link isn't a bottleneck, the data 
rate isn't too high.

> It's not distorted at all - it's queued up and waiting its turn.

the temporal spacing of the ACKs is lost. They won't be sent out at spaced 
intervals, they will be sent out in a burst at line speed (whatever that is).

> care whether your actions affect what others have a right to assume.

Actually, the router does know that the ACK stream is being quantitized. It's 
doing the quatitization (wifi aggregation,  DOCSIS on cable, satellite transmit 
slots, LTO Cell networks, or AQM queue selection)

What it can't know is if anything else would try to slow traffic further by 
trying to delay ACKs more to restore spacing (RFC3449 talks about the problems 
of doing this and recommends against it).

So the only thing remaining is what assumptions other things have the right to 
make. Given how common wireless (Wifi and Cell) networks are, anything that 
makes an assumption that ACKs are flowing back as the traditional wired networks 
handled them is broken today.

If some RFC needs to be updated to make everyone recognize this, then the RFC 
needs to be updated.

But people actually working with servers and the real Internet are seeing these 
effects today.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 13:07:20 -0700

On 10/9/2015 1:04 PM, Mark Allman wrote:
> tl;dr summary: myopia is why we can't have nice things

Actually, it's "be conservative in what you do, be liberal in what you
accept".

Joe

---

From: Greg White <[email-masked]>
Date: Fri, 9 Oct 2015 22:02:24 +0000

>tl;dr summary: myopia is why we can't have nice things

Too true.  DOCSIS would have been much cleaner if we didn't have to deal
with the fallout from the myopic TCP designers.  :-P

>
>allman
>
>
>--
>http://www.icir.org/mallman/
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:10:17 -0700

> with the fallout from the myopic TCP designers.  :-P

Wouldn't it have been cleaner with more appropriate network provisioning?

Why are you blaming TCP?

A DOCSIS router ought to route. If it left the packets alone, IMO we'd
all be better off.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 15:16:41 -0700 (PDT)

> Wouldn't it have been cleaner with more appropriate network provisioning?

"more appropriate network provisioning" is not always going to result in more 
bandwidth the way you want it to.

If there is established infrastructure that can handle X in one direction and 
100X in the other direction, but "appropriate network provisioning" requires 
that the ratio never be more than 3x, it's not going to magically increase 
bandwidth in one direction, all it can do is cap bandwidth in the other 
direction (throwing away capacity)

> Why are you blaming TCP?

first off, didn't you see the smiley?

But treating your question as serious, it's because as RFC3449 shows, highly 
assymetric networks or networks with other traffic on them don't work with TCP 
the way you want it to work. So the option is to modify TCP or tinker with the 
packets to leverage less commonly used features of TCP (or features that are 
implicit in the design rather than explicit)

> A DOCSIS router ought to route. If it left the packets alone, IMO we'd

So you would rather have people getting slower downloads than have network 
equipment modify packets?

Those are the real-world choices.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:24:19 -0700

On 10/9/2015 3:16 PM, David Lang wrote:
...
> "more appropriate network provisioning" is not always going to result in
> more bandwidth the way you want it to.
> 
> If there is established infrastructure that can handle X in one
> direction and 100X in the other direction, but "appropriate network
> provisioning" requires that the ratio never be more than 3x, it's not
> going to magically increase bandwidth in one direction, all it can do is
> cap bandwidth in the other direction (throwing away capacity)

Sure, but we're dealing with a problem that arises when the ratio can't
support 40:1. That's quite an asymmetry except in extreme cases where we
already know extreme measures are required (e.g., satcom with telco
backchannels).

> But treating your question as serious, it's because as RFC3449 shows,
> highly assymetric networks or networks with other traffic on them don't
> work with TCP the way you want it to work. So the option is to modify
> TCP or tinker with the packets to leverage less commonly used features
> of TCP (or features that are implicit in the design rather than explicit)

The former is always on the table - that's what TCPM is for.

The latter has always been problematic, exactly because it's subject to
tragedy-of-the-commons. TCP is a contract between the endpoints, and
whenever a middlebox starts modifying those packets, IMO it becomes a
party to that contract that needs to be taken very seriously (smiley
faces aside ;-)

> So you would rather have people getting slower downloads than have
> network equipment modify packets?

I would rather people get slower downloads than break TCP in ways we
won't see for 5 years or inhibit the extension of TCP in new ways that
might be even more beneficial.

> Those are the real-world choices.

There were other cable modem companies that tried similar "tricks" in
the past and got in a lot of hot water for making a box that went a LOT
faster for *their* customers.

TCP works when everyone plays by the same conservative rules. There has
always been a choice to break those rules for individual gain, and there
has always been a reason we try very hard to avoid that.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 15:31:14 -0700 (PDT)

> already know extreme measures are required (e.g., satcom with telco

that's not the only situation we're talking about here.

Also, keep in mind that your 40:1 ratio assumes that there is no traffic going 
the other direction.

If you have one person trying to watch streaming video while another person is 
uploading pictures to facebook, you can run into trouble at much more even 
ratios.

<skipping the rest since it seems we are talking past each other, each restating 
the same things>

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:39:10 -0700

On 10/9/2015 3:31 PM, David Lang wrote:
> Also, keep in mind that your 40:1 ratio assumes that there is no traffic
> going the other direction.

FWIW, I wasn't - I was assuming that 40:1 is fine if what you intend to
support is browsing only.

> If you have one person trying to watch streaming video while another
> person is uploading pictures to facebook, you can run into trouble at
> much more even ratios.

Restated, you run into trouble because you sold a service you didn't
provision for ;-)

(or, more to the point, you took a gamble that you could sell a useful
service with a particular assumption about traffic ratios, and that
assumption no longer holds)

Again, IMO TCP isn't to be tampered with merely to support a business
model, and that's all I see so far.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 16:31:55 -0700 (PDT)

> model, and that's all I see so far.

Where did you get the idea that I work for an ISP, let alone a cable ISP or had 
any influence in designing or building such networks. I don't.

You are assigning motives here that are clearly false.

As I keep re-stating, this is not a cable/DOCSIS issue, it applies to all sorts 
of things, with Wifi and Cellular being the newest (and arguably biggest) 
sources of the network flow getting quantomized.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 17:01:19 -0700

FWIW, I'm using "you" metaphorically, both here and in other posts.

I.e., whomever is pushing this solution is supporting a business model
that's broken. I don't much care whether the proponents are employed by
that business or not.

There is always tension between commercial interests (and proponents)
and the model of the Internet. The Internet presents a dangerous
opportunity for incredible gain without responsibility.

The tragedy of the commons is very real, and this is a good example of
it. It doesn't matter whether the solutions are being pushed by those
profiting or by those who see only the local, short-term benefit rather
then the potential long term impact.

As has been discussed on TCPM, there may be ways to reduce the impact of
ACKs on the net without having an intermediate device interfere. Let's
have that discussion there and work together on a solution that benefits
everyone - avoiding unnecessary load on the net AND preserving the E2E
semantics of TCP so it can continue to evolve for future capabilities.

Joe

On 10/9/2015 4:31 PM, David Lang wrote:
> Where did you get the idea that I work for an ISP, let alone a cable ISP
> or had any influence in designing or building such networks. I don't.
> 
> As I keep re-stating, this is not a cable/DOCSIS issue, it applies to
> all sorts of things, with Wifi and Cellular being the newest (and
> arguably biggest) sources of the network flow getting quantomized.
> 
> David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 17:22:39 -0700 (PDT)

accusing people of discussing in bad faith because of their business model, 
short sightedness, or other failing is not a good approach to take.

This discussion started from AQM and the smart things that can be done on the 
data in the queues. If TCP is to remain purely "end to end" as you want it to 
be, AQM should not exist, because it re-orders packets.

You don't want to acknowlege it, but TCP is broken in the face of excessive 
buffering. TCPM isn't fixing that, grassroots efforts are developing the fixes 
and AQM is formalizing the results.

TCP is also broken in the face of current and within-the-next-year future wifi 
technologies. AQM helps some here, and there is again outside efforts to address 
the problem. TCPM hasn't done anything that I've heard of other than possibly 
say that the network technology is broken and shouldn't be used.

Guess what, all existing Wifi and Cell networks are not going to be scrapped 
just because you don't like them. Especially when people are demonstrating ways 
to significantly improve performance there.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 17:39:32 -0700

On 10/9/2015 5:22 PM, David Lang wrote:
> You don't want to acknowlege it, but TCP is broken in the face of
> excessive buffering. 

Arguably, buffering was broken and failed to provide the feedback to TCP
(see next paragraph)

> TCPM isn't fixing that, grassroots efforts are
> developing the fixes and AQM is formalizing the results.

TCPM fixed it in 2001 by providing the flags for ECN, which was enabled
by default in Windows since 2012. ALTQ support for ECN has been around
for nearly that long.

What's changed? Not the TCP reaction (except in extreme cases such as
for datacenters) but the router algs. And getting the router algs into
routers - esp. home devices.

> TCP is also broken in the face of current and within-the-next-year
> future wifi technologies. AQM helps some here, and there is again
> outside efforts to address the problem. TCPM hasn't done anything that
> I've heard of other than possibly say that the network technology is
> broken and shouldn't be used.

You might start tracking the TCPM list. We've been talking today about
how to reduce the ACKs by direct action of the source.

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 19:14:45 -0700 (PDT)

> routers - esp. home devices.

if ECN solved the problem, then bufferbloat would have been a non-issue. A large 
percentage of the home routers are running OpenWRT or similar with ECN enabled. 
That didn't solve the problem.

> how to reduce the ACKs by direct action of the source.

There are only so many lists I can track (linux kernel takes a lot of time for 
example :-), I'll consider it.

The problem with proposals like that is just like RFC3449 says, the source can't 
know what the network looks like to decide if there is a benefit to reducing 
ACKs.

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 20:29:55 -0700 (PDT)

On Fri, 9 Oct 2015, David Lang wrote:

> if ECN solved the problem, then bufferbloat would have been a non-issue. A 
> large percentage of the home routers are running OpenWRT or similar with ECN 
> enabled. That didn't solve the problem.

when the investigation into bufferbloat started, we were assuming that the fix 
was just to reduce the size of the buffers and/or make sure that ECN was 
enabled. Experimentation showed that there was no buffer size that both enabled 
full utilization of the links and avoided excessive delays. Switching buffers 
from X packets to X bytes helped a lot, but not enough.

Part of what's changed is the usage patterns, but a lot of it is just that 
things that worked when a fast network was 10Mb don't always scale across 
several orders of magnatude.

David Lang

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 20:32:58 -0700 (PDT)

On Fri, 9 Oct 2015, David Lang wrote:

> buffering. TCPM isn't fixing that, grassroots efforts are developing the 
> fixes and AQM is formalizing the results.

Apologies if this came off as an attack on the TCPM group. I don't know what 
they have been working on, I just know what is visible to the outside world. It 
was intended as an observation of apparent results, not an attack.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 21:19:15 -0700

On 10/9/2015 7:14 PM, David Lang wrote:
> The problem with proposals like that is just like RFC3449 says, the
> source can't know what the network looks like to decide if there is a
> benefit to reducing ACKs.

To understand your position, is your conclusion that if it benefits a
single place in the network, then that's enough of a view to know that
it's safe and beneficial throughout the network?

Joe

---

From: David Lang <[email-masked]>
Date: Fri, 9 Oct 2015 22:58:34 -0700 (PDT)

> it's safe and beneficial throughout the network?

No.

But when something shows clear beneifts when deployed, clear problems need to be 
identified to counter those benefits, or alturnative methods of solveing the 
problem need to be shown.

I didn't know about it initially (I was working from experience and logic), 
RFC3449 has explored many of the problems that cause and result from fewer than 
1 ACK per 2 data packets, including worrying about the timing of the ACK 
packets.

As much as you seem to want to make this discussion about cable modems and 
highly assymetric links, this discussiona ctually started  from the position of 
packets in a particular flow being sent in bursts due to non-endpoint related 
reasons. It started with AQM queues, but radio networks (Wifi and Cell) have 
similar behavior (packets queue until a transmit window is available, tehn a 
bunch are sent at once)

Cable modems were introduced to the discussion to counter the thought that 
sending fewer ACKs would destroy the Intenet.

RFC3449 doesn't completely address the current situation, but it provides a very 
good place to start, and it seems to me that the solitions it explores to 
address the conerns that it (and you) raise are actually being addressed pretty 
completely. There are still some areas to talk about (ECN interaction for 
example) and we wshould be talking about those issues rather than arguing that 
the proposal violates holy writ.

you want a network where packets are just forwarded with no modification and no 
delays. Unfortunantly such a network does not match the real world any longer 
(and it only approximated the real world in the first place) Shared media 
networks have existed since the earliest networking days. What is changing is 
theratio between the data speed when transmittingand the time available to 
transmit. Given the same number of stations in an area, I'll bet that old 
thinnet ethernet at 10Mb/s spent a significantly higher percentage of the time 
transmitting data than current Gb/s and immediate future 10Gb/s wireless 
networks. Just like wired network speeds are climbing with the MTU staying the 
ame, wireless network data rates are climbing but the time available for a given 
station is remaining the same (or shrinking), so the number of packets that get 
delayed until the next transmit slot are only going to keep climbing, no matter 
what anyone wants.

The asymmetrical nature of the networks is just addng insult to injury, not the 
cause of the issues.

Wireless networks are gaining the ability to transmit to multiple stations at 
once. But the nature of usage patterns and the physical geometry of the wired vs 
mobile ends of the link make the reality that only the wired end can effectively 
make use of this capability. this makes the availble downstream bandwidth on a 
network fabric grow much faster than the available upstream bandwidth. To some 
extent this will put less stress on the upstream side of the wired network it's 
attached to, but it will mean that the demand for mobile station transmit slots 
is only going to multiply. Just reducing the number of timeslots the base 
station uses isn't the answer because that will add latency to all the data 
downloads.

IIRC, I saw a story today where one of the contenders for 5G cell networks is 
already at a ratio of 24:1 or worse.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 23:24:10 -0700

On 10/9/2015 10:58 PM, David Lang wrote:
> RFC3449 doesn't completely address the current situation, but it
> provides a very good place to start, and it seems to me that the
> solitions it explores to address the conerns that it (and you) raise are
> actually being addressed pretty completely. There are still some areas
> to talk about (ECN interaction for example) and we wshould be talking
> about those issues rather than arguing that the proposal violates holy
> writ.

I'm OK with the recommendations for the endpoints to modify their ACK
behavior, but still am concerned that stripping out ACKs in the middle
of the network is problematic for a number of reasons, only some of
which are addressed in RFC3449 (which also recommends against deploying
them in the general Internet, so if that's your justification then it's
very clearly in direct opposition).

Joe

---

From: Simon Barber <[email-masked]>
Date: Sat, 10 Oct 2015 13:46:43 -0700

I'm talking specifically about duplicate ACKs - same ACK, same contents 
- signalling that the receiver has seen a hole. These should be 
communicated as quickly as possible to reduce head of line blocking and 
application latency - hence the TCP spec that you should stop delaying 
your ACKs in this state.

Simon

On 10/8/2015 9:08 PM, David Lang wrote:
> Well, the only situation we are talking about is when an AQM function 
> finds that there are multiple acks for one TCP session in it's queue. 
> The dispute is if these acks are can be considered 'duplicates' (since 
> the later ones will ack data for the earlier ones) and so can be 
> 'deduped' or if there is a significant amount of value in sending 
> every ack through anyway.
>
> David Lang
>
>>>>>>>> actually, this is only a violation of the SHOULD section, not 
>>>>>>>> the MUST
>>>>>>>> section.
>>>>>>>
>>>> no, you could not trust any timestamps in the packets even if 
>>>> nothing changes the packets between endpoints.
>>>>
>>>> due to the difference in the cost of the transmitters on each end 
>>>> of the link. As are Satellite IP systems, etc.
>>>>
>>>> but by not having to transmit the early acks, the later ack goes 
>>>> out faster, so the customers get less of a delay in getting the 
>>>> data they have received acked.
>>>>
>>>> If there are other packets in the queue, you still can delete all 
>>>> the acks except the last one that will fit into the burst with no 
>>>> unless there are other packets in the flow that the ack is jumping, 
>>>> I see no problem with this. You aren't sending out an ack before 
>>>> the data is arrived, you just aren't delaying the last ack 
>>>> unneccessarily.
>>>>
>>>> how would the burst be any different if the server gets 32 acks 
>>>> back to back vs 1 ack that covers everything. The additional acks 
>>>> aren't going to get there any faster than the single one would.
>>>>
>>>> string of acks at wire speed is better than a single ack covering 
>>>> the same data 'because I say so' doesn't cut it.
>>>>
>>>>>> it's within the range of
>>>> no, merely congested for some reason. Any shared media will have 
>>>> the same situation, when another station is transmitting, a queue 
>>>> builds.
>>>>
>>>> you say that it breaks timing assumptions and calculations, but 
>>>> then don't explain how the train of acks arriving at wire speed 
>>>> would let your calculations be any more accurate.
>>>>
>>>> David Lang
>>>>

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 14:15:28 -0700 (PDT)

Ok, I think we hve a couple different situations here. RFC3449 section 5.2.1 
covers several special signals that multile acks cover that should not be 
touched.

Then what I was understanding from the comment below was a way to mitigate the 
impact of the consolodated ack getting dropped by the network (since one lost 
ack out of a train of them has very little impact, but if they are combined into 
one stretch ack and that ack gets dropped, it's the equivalent of the entire 
trin of acks getting lost)

So here's some thoughts for people.

with the existing signalling (double or triple acks), if the string of acks 
happens to land exactly on the edge of a transmit window, it may get split 
across multiple windows, at which point the signal can be lost. This cannot be 
any worse than if one of the ack packets ws dropped by the network anyway, but 
how bad is it?

Also, if the stretched ack packet gets duplicated into the next transmit window 
to cover the possibility of it getting lost by the network and it both doesn't 
get lost and there is no furhter ack traffic before the next transmit window, 
this then will generate a exact duplicate ack. What harm cn this do? I believe 
that there are already conditions where various failure modes can cause packets 
to be duplicated already, so its not an impossible situation today. I also think 
that it would be fairly rare to trigger this as it would requie that there be 
multiple acks in the queue one transmit window and none for the next transmit 
window, but overall this would probably increase the odds of duplicate ack 
packets arriving.

David Lang

On Sat, 10 Oct 2015, Simon Barber wrote:

> I'm talking specifically about duplicate ACKs - same ACK, same contents - 
> signalling that the receiver has seen a hole. These should be communicated as 
> quickly as possible to reduce head of line blocking and application latency - 
> hence the TCP spec that you should stop delaying your ACKs in this state.
>
> Simon
>
>> Well, the only situation we are talking about is when an AQM function finds 
>> that there are multiple acks for one TCP session in it's queue. The dispute 
>> is if these acks are can be considered 'duplicates' (since the later ones 
>> will ack data for the earlier ones) and so can be 'deduped' or if there is 
>> a significant amount of value in sending every ack through anyway.
>> 
>> David Lang
>> 
>>> I like this suggestion - but it's only needed in the case where there are 
>>> dup acks.
>>> 
>>> Simon
>>> 
>>>>> their networks'. DSL lines are highly asymmetric due to the difference 
>>>>> in the cost of the transmitters on each end of the link. As are 
>>>>> Satellite IP systems, etc.
>>>>> 
>>>>> degredation in how fast data is acknoleged (and you increase the amount 
>>>>> of usable data that is sent in that timeslot instead of wasting it on 
>>>>> redundant ack packets)
>>>>> 
>>>>> see no problem with this. You aren't sending out an ack before the data 
>>>>> is arrived, you just aren't delaying the last ack unneccessarily.
>>>>> 
>>>>> back vs 1 ack that covers everything. The additional acks aren't going 
>>>>> to get there any faster than the single one would.
>>>>> 
>>>>> of acks at wire speed is better than a single ack covering the same data

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 15:02:51 -0700 (PDT)

> very clearly in direct opposition).

Remember that this can only take place when there is enough of a bottleneck that 
enough of a queue has built up that multiple ack packets are sitting in the 
queue waiting to be transmitted. Such a queue is only going to build up if the 
link is a bottleneck [1]. As I undertsand the current Industry Best Practice, 
unless you have ISPs deliberatly not upgrading (ala the Netflix fights), the 
core systems almost never queue and so such algorithms would never kick in.

So even if I could wave a magic wand and deploy this on every router, it would 
only have an effect in a few places. Endpoint links where the available 
bandwidth drastically changes are not the exclusive list of palces that this 
would effect, but I think it's pretty close.

David Lang

[1] the exception to this is when you have a hop where the transport quanomizes 
the data the way Wifi and Cell networks do, So if you have a half-duplex link 
like normal Wifi or a time-slot MAC like a Cell network in the middle of the 
path, this sort of thing could trigger there, but I'm not convinced that it 
would be a bad thing there. Full duplex Wifi links that can transmit 
continuously don't need this

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 15:28:25 -0700 (PDT)

On Sat, 10 Oct 2015, David Lang wrote:

> Remember that this can only take place when there is enough of a bottleneck 
> that enough of a queue has built up that multiple ack packets are sitting in 
> the queue waiting to be transmitted. Such a queue is only going to build up 
> if the link is a bottleneck [1]. As I undertsand the current Industry Best 
> Practice, unless you have ISPs deliberatly not upgrading (ala the Netflix 
> fights), the core systems almost never queue and so such algorithms would 
> never kick in.
>
> So even if I could wave a magic wand and deploy this on every router, it 
> would only have an effect in a few places. Endpoint links where the available 

Also, since it costs CPU to go through the queue looking for ACK packets to try 
and combine, it is a feature unlikly to be enbled on high speed devices. It 
would probably only be enabled when the savings in the reduced number of ACKs 
that would be transmitted is significant enough to be worth the effort. Endpoint 
devices for home users where there is probably also a highly asymmetric link are 
a very clear win. Core routers are almost never going to be a win.

David Lang

> David Lang
>
> [1] the exception to this is when you have a hop where the transport 
> quanomizes the data the way Wifi and Cell networks do, So if you have a 
> half-duplex link like normal Wifi or a time-slot MAC like a Cell network in 
> the middle of the path, this sort of thing could trigger there, but I'm not 
> convinced that it would be a bad thing there. Full duplex Wifi links that can 
> transmit continuously don't need this
>

---

From: Dave Taht <[email-masked]>
Date: Sat, 10 Oct 2015 16:06:33 -0700

I have been busy with other matters and unable to keep up on this thread.

I just wanted to say that I'm with dlang here... aggregated and
asymmetric transmits in wifi, cell, and cable, are here to stay. Deal
with it.

I would certainly have preferred a wifi world that instead of all
stations and APs highly contending for bursty access to a single
320mhz channel, we had 160 dedicated, low latency, 5mhz channels, but
that is not what the IEEE has handed us.

On Fri, Oct 9, 2015 at 10:58 PM, David Lang <[email-masked]> wrote:
> But when something shows clear beneifts when deployed, clear problems need
> to be identified to counter those benefits, or alturnative methods of
> solveing the problem need to be shown.
>
> RFC3449 has explored many of the problems that cause and result from fewer
> than 1 ACK per 2 data packets, including worrying about the timing of the
> ACK packets.
>
> highly assymetric links, this discussiona ctually started  from the position
> of packets in a particular flow being sent in bursts due to non-endpoint
> related reasons. It started with AQM queues, but radio networks (Wifi and
> Cell) have similar behavior (packets queue until a transmit window is
> available, tehn a bunch are sent at once)
>
> RFC3449 doesn't completely address the current situation, but it provides a
> very good place to start, and it seems to me that the solitions it explores
> to address the conerns that it (and you) raise are actually being addressed
> pretty completely. There are still some areas to talk about (ECN interaction
> for example) and we wshould be talking about those issues rather than
> arguing that the proposal violates holy writ.
>
> you want a network where packets are just forwarded with no modification and
> no delays. Unfortunantly such a network does not match the real world any
> longer (and it only approximated the real world in the first place) Shared
> media networks have existed since the earliest networking days. What is
> changing is theratio between the data speed when transmittingand the time
> available to transmit. Given the same number of stations in an area, I'll
> bet that old thinnet ethernet at 10Mb/s spent a significantly higher
> percentage of the time transmitting data than current Gb/s and immediate
> future 10Gb/s wireless networks. Just like wired network speeds are climbing
> with the MTU staying the ame, wireless network data rates are climbing but
> the time available for a given station is remaining the same (or shrinking),
> so the number of packets that get delayed until the next transmit slot are
> only going to keep climbing, no matter what anyone wants.
>
> The asymmetrical nature of the networks is just addng insult to injury, not
> the cause of the issues.
>
> Wireless networks are gaining the ability to transmit to multiple stations
> at once. But the nature of usage patterns and the physical geometry of the
> wired vs mobile ends of the link make the reality that only the wired end
> can effectively make use of this capability. this makes the availble
> downstream bandwidth on a network fabric grow much faster than the available
> upstream bandwidth. To some extent this will put less stress on the upstream
> side of the wired network it's attached to, but it will mean that the demand
> for mobile station transmit slots is only going to multiply. Just reducing
> the number of timeslots the base station uses isn't the answer because that
> will add latency to all the data downloads.
>
> IIRC, I saw a story today where one of the contenders for 5G cell networks
> is already at a ratio of 24:1 or worse.
>
> David Lang
>
>

-- 
Dave Täht
Do you want faster, better, wifi? https://www.patreon.com/dtaht

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 11 Oct 2015 02:08:07 +0300

> On 11 Oct, 2015, at 01:28, David Lang <[email-masked]> wrote:
> 
>> So even if I could wave a magic wand and deploy this on every router, it would only have an effect in a few places. Endpoint links where the available bandwidth drastically changes are not the exclusive list of palces that this would effect, but I think it's pretty close.
> 
> Also, since it costs CPU to go through the queue looking for ACK packets to try and combine, it is a feature unlikly to be enbled on high speed devices. It would probably only be enabled when the savings in the reduced number of ACKs that would be transmitted is significant enough to be worth the effort. Endpoint devices for home users where there is probably also a highly asymmetric link are a very clear win. Core routers are almost never going to be a win.

In the case of wifi, aggregation and huge per-TXOP overheads mean that the cost of sending multiple acks instead of just one is negligible.  It’s simply not worth performing any sort of compression or consolidation except for (losslessly reversible) aggregation, because the gains pale into insignificance next to that unavoidable overhead.

The question is more arguable in the case of DOCSIS, where the cost of setting up a TXOP is in latency and temporal quantisation rather than channel overheads.  However, acks are individually really small compared to data packets, so the benefit of dropping each one is correspondingly small.  I do think it’s important to avoid dropping dupacks that might be associated with congestion or loss signalling, because that slows down the sender’s response to these relatively infrequent but significant events.

Both of the above situations are independent of reverse-channel congestion.

One of our many bufferbloat tests involves a 10:1 asymmetric link carrying a single downstream TCP flow, fifty upstream TCP flows, and a small number of latency-measuring (UDP and ICMP) flows.  Different queue types cope with that workload in very different ways:

- Dumb FIFOs and “pure” flow isolators (like SFQ) don’t drop many acks; usually they drop data packets instead, since they’re bigger and thus more likely to overflow the upstream queue.  This results in the downstream flow being starved via ack-clocking (flow isolators being more predictable in this respect).

- “Pure” AQMs without flow isolation tend not to care whether the packets they’re shooting at are acks or data packets.  Dropped acks  save much less queue space than dropped data packets, so the AQM tends to tune its drop rate so that the upstream data flows back off sufficiently to allow almost all of the acks through.  This permits full downstream bandwidth, but the acks consume a significant proportion of the uplink, which is then unavailable for the upstream flows.

- Flow-isolating AQMs (like fq_codel and cake) give the ack flow an equal share of the bandwidth with each of the uplink data flows, and are able to give each flow an independent drop rate.  In practice, the (responsive) data flows get a modest drop rate, while the ack flow (being antiresponsive, since thinning the acks increases the downstream rate and thus the incoming ack rate) ends up with a very high drop rate.  This demonstrably works better with more perfect flow isolation (fq_codel suffers from significant hash collisions in this scenario, cake does not); it also exercises an edge case in codel when the drop rate rises to a particularly high value (without care, it can wrap around, resulting in a wildly fluctuating downstream rate).

Cake takes care to handle this extreme scenario sensibly.  It effectively performs ack thinning when the uplink is congested, without actually being aware of what an ack *is* - it’s just another packet.  DupACKs are not a problem, since it doesn’t spread the surviving packets between distinct TXOPs as DOCSIS modems apparently can; indeed, due to acks’ small size compared to the DRR quantum, the requisite three DupACKs will quite likely still go out back-to-back, even if they weren’t consecutively originated.  None of this required special handling - it just falls out of a well-engineered, protocol-agnostic AQM scheme.

The important point, however, is that Cake *only* performs ack thinning when the link is congested.  If there’s sufficient bandwidth to send all the acks, it sends them all.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 11 Oct 2015 02:12:49 +0300

> On 11 Oct, 2015, at 02:06, Dave Taht <[email-masked]> wrote:
> 

Dual radios would be a useful compromise - one 20-40 MHz channel to the AP, one from it.  Stations would still need to negotiate for the channel in the uplink direction, but the AP would be able to transmit on the downlink more freely.

 - Jonathan Morton

---

From: Simon Barber <[email-masked]>
Date: Sat, 10 Oct 2015 16:29:14 -0700

The next generation of WiFi is likely to include full duplex - it's 
about the only physical layer thing left!

Simon

---

From: Joe Touch <[email-masked]>
Date: Sat, 10 Oct 2015 17:56:30 -0700

> asymmetric transmits in wifi, cell, and cable, are here to stay. Deal

Right until they're encrypted, then we'll be back to square 1.

Time to deal with the case where the net isn't involved in tampering
with E2E connections, IMO.

Joe

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 20:45:55 -0700 (PDT)

On Sun, 11 Oct 2015, Jonathan Morton wrote:

>>> would only have an effect in a few places. Endpoint links where the 
>>> available bandwidth drastically changes are not the exclusive list of palces 
>>> that this would effect, but I think it's pretty close.
>>
>> Also, since it costs CPU to go through the queue looking for ACK packets to 
>> try and combine, it is a feature unlikly to be enbled on high speed devices. 
>> It would probably only be enabled when the savings in the reduced number of 
>> ACKs that would be transmitted is significant enough to be worth the effort. 
>> Endpoint devices for home users where there is probably also a highly 
>> asymmetric link are a very clear win. Core routers are almost never going to 
>> be a win.
>
> In the case of wifi, aggregation and huge per-TXOP overheads mean that the 
> cost of sending multiple acks instead of just one is negligible.  It’s simply 
> not worth performing any sort of compression or consolidation except for 
> (losslessly reversible) aggregation, because the gains pale into 
> insignificance next to that unavoidable overhead.

the cost in that txop of sending multiple acks instead of one is minor, but does 
sending those multiple acks mean that there are data packets that have to wait 
for the next txop?

But I actually view this as a tragedy of the commons situation. Sending the 
extra packets doesn't hurt you, but it does mean that you are using more airtime 
than you need, and that hurts everyone (including, eventually, you)

Anything that we can routinely do to minimize the length of the transmission 
should be done.

David Lang

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 20:55:14 -0700 (PDT)

On Sun, 11 Oct 2015, Jonathan Morton wrote:

> Dual radios would be a useful compromise - one 20-40 MHz channel to the AP, 
> one from it.  Stations would still need to negotiate for the channel in the 
> uplink direction, but the AP would be able to transmit on the downlink more 
> freely.

That helps less than you would think because there isn't just one AP in the 
area. There are multiple APs around on the same channel. Even in the best 
managed and coordinated wifi networks there are only so many channels and you 
_are_ going to end up re-using a channel in such a way that stations are going 
to hear more than one AP.

I run the wireless network for the Scale conference in Los Angeles, and I do a 
really good job with it (It's not just me bragging, see 
https://www.youtube.com/watch?v=UXvGbEYeWp0&t=4795 for a segment from the Linux 
Action Show from this years conference where they outright rant about how good 
things were compared to other places they've been). But even with RF sanity 
being my primary focus, and using 5GHz equipment, there is still going to be 
overlapping coverage. And on 2.4GHz, especially outside a carefully engineered 
environment, it's horrific.

The focus of thte IEEE and engineers on per-station speed in an isolated 
evironment is pushing things in the wrong direction. With 802.11ac (and cell 
LTE networks) we are gaining the ability for the fixed stations to transmit to 
more than one mobile station at the same time on a single channel, but that only 
adds to the bandwidth distortion and means that there will be even more acks 
being generated in response to a single AP txop.

David Lang

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 21:02:56 -0700 (PDT)

full duplex RF communication is HARD, and completely incompatible with existing 
stuff. Full duplex also doesn't really help when the problem is more than two 
stations communicating. And with the mu-mimo mode of operation where the AP is 
transmitting to multiple receivers at once, 'full duplex' between all the 
stations is orders of magnatude harder.

The biggest problem with the latest standards is that the per transmit overhead 
is not shrinking at all. Even as the data bitrates skyrocket, the management 
bitrates, quiet time requirements, etc remain static.

There would actually be far more benefits from scrapping these timings and 
re-setting them to match current high-speed capabilities than in going to 
full-duplex with the current timings in place. Unfortuantly, unless the new 
systems are on a newly opened frequency band, legacy requirements (existing 
networks continuing to operate, not just legacy equipment talking to new 
networks) make this hard. The fact that Wifi networks are not on a dedicated, 
regulated band and are instead in the 'free for all' bands means that it's not 
even possible to outlaw old stuff to make room for new stuff.

David Lang

On Sat, 10 Oct 2015, Simon Barber wrote:

> The next generation of WiFi is likely to include full duplex - it's about the 
> only physical layer thing left!
>
> Simon
>

---

From: David Lang <[email-masked]>
Date: Sat, 10 Oct 2015 21:16:08 -0700 (PDT)

> Right until they're encrypted, then we'll be back to square 1.

Don't just take the simpleminded approach of taking existing data and encrypting 
it. Develop new encryption friendly protocols that aren't excessivly chatty to 
begin with.

> with E2E connections, IMO.

just like IPv6 is going to eliminate all firewalls and NAT and restore the 
pristine end-to-end Internet that was originally designed?

sorry, but I think those days are past. The Internet isn't A International 
Network, it's a Network of Networks. While I agree that the core that connects 
the edge networks where the servers and users live should be data agnostic, the 
points where the networks join is not going to be transparent as the owners of 
the edge networks have not only a right, but a duty to manage their network. The 
interesting discussion is how you define core vs edge (I count ISP networks as 
core, even if they have some islands inside them that count as edge)

David Lang

---

From: Joe Touch <[email-masked]>
Date: Sat, 10 Oct 2015 22:13:46 -0700

On 10/10/2015 8:45 PM, David Lang wrote:
> But I actually view this as a tragedy of the commons situation. Sending
> the extra packets doesn't hurt you, but it does mean that you are using
> more airtime than you need, and that hurts everyone (including,
> eventually, you)

So, basically, you appreciate the tragedy of the commons when it's your
tragedy and their commons, but not the other way around?

(using 'sed')

s/sending the extra/altering the transport/

s/using more airtime/altering E2E semantics/

Joe

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 14:18:28 -0700 (PDT)

> s/using more airtime/altering E2E semantics/

I actually disagree that there is a significnt difference in the actual E2E 
semantics between sending a single stretched ACK and sending the full number of 
ACKs back-to-back. The "transmit burst" issue will happen in either case.

But you will notice that in both cases, I am in favor of reducing the number of 
packets on the wire. Packets not send can't interfere with other traffic.

If ACK packets are delayed to the point that additional ACK packets have caught 
up with them, they are not providing the rate signalling signal that you are 
looking for.

Question: Does anyone know if current server TCP stacks implement the TCP byte 
counting that RFC3449 talks about?

David Lang

---

From: Joe Touch <[email-masked]>
Date: Sun, 11 Oct 2015 14:28:27 -0700

On 10/11/2015 2:18 PM, David Lang wrote:
> But you will notice that in both cases, I am in favor of reducing the
> number of packets on the wire. Packets not send can't interfere with
> other traffic.

The degenerate case of this approach is a circuit. Take a look at
Morris's 1997 ICNP paper to see what happens when you end up with one -
or sometimes less than one - packet per round trip time.

One of the reasons we use packets is to provide more timely,
fine-grained feedback between endpoints. Aggregating them at the source
or in the network (rather than at the receiver) can amplify reaction to
packet loss (when the aggregate ACK is lost) and increases
discretization effects in the TCP algorithms.

Joe

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 15:49:25 -0700 (PDT)

On Sun, 11 Oct 2015, Joe Touch wrote:

> packet loss (when the aggregate ACK is lost)

the issue of amplified reaction to packet loss is a valid one, but packet loss 
is still pretty rare, so the proposal of putting a duplicate stretched ack into 
the queue for the next transmit window when you aggregate packets in the current 
transmit window should largely mitigate this.

In addition, Since cablemodems have been doing this for 15 years, we should be 
able to get people to look at real-world traffic and tell us how much of a 
problem this is.

> and increases

This is where I disagree with you. The ACK packets are already collapsed 
time-wise. Whatever timing they had when they were generated, the fact that they 
are all sitting in the same queue at the same time means taht that timing has 
been lost, Nobody is adding latency by trying to recreate the time spacing, so 
the discretization (or as I've been calling it quantitization) of the traffic is 
happening even if every ACK packet is sent.

My contention is that since this is already happening, consolidating the ACK 
packets into stretched ACKs doesn't make this any worse, and it saves network 
bandwidth (and decreases latency to the extent that data is acked faster than 
waiting for the entire chain or original ACKs to get through, especially if that 
would take multiple transmit windows). As a result, thinning the ACKs is kinder 
to the network.

I am in no way advocating delaying any packet that could be sent, just combining 
those that have already been delayed.

We're talking about ACK packets here, but any protocol that sends short TCP 
packets is potentially fair game (TCP/SSH for example). If the packets are all 
sitting in the same queue, they should be able to be combined. This happens on 
the sending server if multiple writes to the socket happen before the packet has 
been sent, it should be able to happen on the network as well (further reducing 
overhead by sending the same data with fewer packets)

David Lang

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 12 Oct 2015 02:34:45 +0300

> My contention is that since this is already happening, consolidating the
ACK packets into stretched ACKs doesn't make this any worse, and it saves
network bandwidth (and decreases latency to the extent that data is acked
faster than waiting for the entire chain or original ACKs to get through,
especially if that would take multiple transmit windows). As a result,
thinning the ACKs is kinder to the network.

I agree, *iff* they are not DupACKs signalling packet loss.  Do existing
and future cable modems take that subtle distinction into account?

- Jonathan Morton

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 17:10:37 -0700 (PDT)

> and future cable modems take that subtle distinction into account?

I don't know, and RFC3449 section 5.2.1 lists a couple other conditions to 
watch out for. But this discussion started with what AQM algorithms can/should 
do when they see multiple ACKs in their queue (and I added wifi as part of 
make-wifi-fast). So we are talking about what should be put in future code. The 
cable modems are just an example we can draw from of a past implementation of 
similar ideas.

hopefully future cable modems will use a more generic AQM algorithm instead of 
the hard-coded optimizations they've had in the past. It's almost always a win 
when a generic solution can replace a specific solution :-)

David Lang

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 12 Oct 2015 16:13:45 +0200 (CEST)

> with the fallout from the myopic TCP designers.  :-P

So I agree that most likely, it's beneficial to have fewer ACKs.

What I think people arguing against this practice are these kinds of 
issues:

http://blog.dan.drown.org/sb6183-dropping-ipv6-traffic/

I don't think there is a solution that we all can agree on, all approaches 
have their benefits and drawbacks. I think the above article just shows 
how things can go wrong in very subtle ways.

---

From: David Lang <[email-masked]>
Date: Mon, 12 Oct 2015 09:09:04 -0700 (PDT)

On Mon, 12 Oct 2015, Mikael Abrahamsson wrote:

> What I think people arguing against this practice are these kinds of issues:
>
> have their benefits and drawbacks. I think the above article just shows how 
> things can go wrong in very subtle ways.

no question that things can go wrong in subtle ways. If network protocol design 
was easy, we would have had a couple protocols designed back in the early days 
of computing and there wouldn't be the recent rash of new protocol designs :-)

But too much fear that "something may go wrong somewhere" can prevent any 
progress.

David Lang

---

From: Greg White <[email-masked]>
Date: Mon, 12 Oct 2015 23:08:41 +0000

> My contention is that since this is already happening, consolidating the ACK packets into stretched ACKs doesn't make this any worse, and it saves network bandwidth (and decreases latency to the extent that data is acked faster than waiting for the entire chain or original ACKs to get through, especially if that would take multiple transmit windows). As a result, thinning the ACKs is kinder to the network.

I agree, *iff* they are not DupACKs signalling packet loss.  Do existing and future cable modems take that subtle distinction into account?

I presume that they are not discarding DupACKs or corrupting SACK in some way (the whole point of specialized ACK handling is to ensure good TCP performance after all), but since the algorithms are proprietary, for existing cable modems I don't know.  I will contact the developers as well as run some tests on a few modems to get a better idea (and maybe Mikael already has some data from his recent testing he could share?).   For future modems, I can add an explicit reference to RFC3449 in the DOCSIS 3.1 spec just in case implementers aren't aware for some reason.

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Oct 2015 07:54:56 -0700

David,

On 10/11/2015 3:49 PM, David Lang wrote:
> the issue of amplified reaction to packet loss is a valid one, but
> packet loss is still pretty rare,

But you claim to need to drop the ACKs to avoid dropping other packets.
What's to say that 1) the ACKs won't end up getting dropped in your
system or 2) the ACKs won't get dropped further downstream?

Again, *you* don't know.

...
> time-wise.

Time is only one element of the discretization. The other is fate
sharing - dropping one aggregate ACK has the effect of dropping the
entire set of un-aggregated ones. The stream of unaggregated ACKs gives
TCP a robustness to drops, but you're now forcing a large burst of drops
for a specific packet type. That's the problem.

We've gone round this issue for a lot of messages, but the key point is
that this isn't a new idea, nor are the numerous considerations and
concerns.

Ultimately, if you have too many packets from a protocol that reacts to
congestion, the correct solution is to provide that feedback and let the
ends figure out how to respond. That's the only solution that will work
when (not if) new messages or encrypted headers enter the picture.

These kinds of in-network hacks are what's really killing the ability of
the Internet to evolve.

Joe

---

From: David Lang <[email-masked]>
Date: Tue, 13 Oct 2015 12:09:12 -0700 (PDT)

> system or 2) the ACKs won't get dropped further downstream?

combining ACKs further downstream should not hurt anything (as long as the 
RFC3449 criteria are covered)

_dropping_ ACKs without combinging them is packet loss, pure and simple. With 
all the implications of that. The aggregated ACK is more significant that one of 
the other ACKs that were combined into it, but reducing the number of packets 
reduces the chances of needing to drop a packet. And the suggestion that was 
made to dup the aggregated ACK into the next transmit slot will greatly reduce 
the impact of the aggregated ACK being lost.

> for a specific packet type. That's the problem.

this is why the suggestion to dup the aggregated ACK into the next transmit slot 
is such a great idea (I'd have to go back through this thread and dig up who 
suggested it). It almost entirely mitigates the problem of the ACK packet 
getting dropped.

> These kinds of in-network hacks are what's really killing the ability of

If people are faced with existing protocols not working well, or making local 
modifications that make things work, but limit the ability for the "Internet to 
eveolve" the local modifications are going to win every time. This is the 
Internet evolving, just not in the way that you want.

David Lang

---

From: David Lang <[email-masked]>
Date: Tue, 13 Oct 2015 12:36:01 -0700 (PDT)

On Mon, 12 Oct 2015, Greg White wrote:

>> waiting for the entire chain or original ACKs to get through, especially if 
>> that would take multiple transmit windows). As a result, thinning the ACKs is 
>> kinder to the network.
>
> I agree, *iff* they are not DupACKs signalling packet loss.  Do existing and 
> future cable modems take that subtle distinction into account?
>
> I presume that they are not discarding DupACKs or corrupting SACK in some way 
> (the whole point of specialized ACK handling is to ensure good TCP performance 
> after all), but since the algorithms are proprietary, for existing cable 
> modems I don't know.  I will contact the developers as well as run some tests 
> on a few modems to get a better idea (and maybe Mikael already has some data 
> from his recent testing he could share?).  For future modems, I can add an 
> explicit reference to RFC3449 in the DOCSIS 3.1 spec just in case implementers 
> aren't aware for some reason.

It should also say that AQM is needed on the downstream side to protect against 
transmit bursts.

The suggestion was posted early in this thread (but not in someting I can 
quickly find) that to harden the connection against the aggregate ACK being 
lost, it should be sent again in a separate transmit window.

I latched onto that and think the right thing to do is to duplicate the 
aggregate ACK that's created and put it at the beginning of the queue that 
remains after the current transmit window.

If there are more ACKs in the next transmit window, it will get aggregated into 
them and not cause any additional traffic.

If there are not any more ACKs in the next transmit window, and the first ACK 
was lost, it will replace it with the minimum of delay

If there are not any more ACKs in the next transmit window, and the first ACK 
was not lost, we are now transmitting a duplicate ACK, which may be interpreted 
as a signal to speed up tranmission (depending on the exact timing of the 
arrivials of these two ACKs). While this is not good, we already need to be 
protected from transmit bursts, so this is just another possible cause for such 
bursts.

David Lang