Subject: Re: [aqm] ACK Suppression

From: [email-masked]
Date: Wed, 7 Oct 2015 15:52:18 -0400 (EDT)

I presume this is not a serious focus of the AQM group. So maybe I'm wasting my time by writing about this.
 
Nonetheless I am troubled by the very fact of the discussion taking place, for two reasons:
 
1) TCP ACKs are TCP's business only.  It's not a gateway or router's job to get involved in or to understand end-to-end protocols, *even if* the router thinks it knows exactly what the endpoints' goals are.  And the router cannot know that for every protocol, not even the many higher level protocols on top of TCP, which use TCP quite differently.  The idea that routers can be omniscient, merely by looking at packets and taking the designers' personal prejudices into account, seems ridiculous. TCP endpoints on both ends of a connection can reduce the number of ACKs they send if they want. If ACKs are filling up buffers in intermediate routers, just drop them or mark them to notify that they are contributing to congestion.  The endpoints have to slow down something, and they can slow down ACKs by mutual agreement.
 
2) The hypothetical that there will be a sufficiently long sequence of ACKs for a single end-to-end flow buffered in a single router queue may seem plausible, *until it becomes clear that in the big picture, having so many packets in a queue means that the network is extremely congested by that point*.  In other words, in order for this "optimization" to apply, you would have to operate the network at an unacceptable operating point! It's like saying that when a highway has slowed to a crawl, we can load all the cars going to a particular destination onto  single "car carrier" to save gas. Far better to insure that queues are not built up!  The purpose of queueing is to absorb bursts that can't be anticipated, not to build up congestion in order to have enough data to perform a dubious optimization that can best be done at the source of traffic in cooperation with the destination.
 
It's said that in committees the amount of time spent on trivialities far exceeds the time spent on important issues.  That seems to be true as I watch the discussion on this list.
 
There are real congestion problems in the real Internet.  Real engineers would be working on solving them with practical solutions that actually work.  I see none of that here.  It's very, very sad.  We still have serious "bufferbloat" in most access networks nationwide, and lots of gear sold by the biggest brand name vendors of both high-speed fiber/copper access and packet wireless data access that is allowing queueing of between 1/2 second to 4 seconds to build up and sustain itself.
 
But this group dithers.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 7 Oct 2015 22:18:17 +0200 (CEST)

On Wed, 7 Oct 2015, [email-masked] wrote:

> not built up!  The purpose of queueing is to absorb bursts that can't be 
> anticipated, not to build up congestion in order to have enough data to 
> perform a dubious optimization that can best be done at the source of 
> traffic in cooperation with the destination.

There is no congestion when the ACK suppression kicks in and is useful.

If I download at 250 megabit/s, at 1500 byte packets, this is around 
21kpps. That means I am sending around 10kpps worth of ACKs.

If my media access layer only gives me access to send packets every 1-3 ms 
(but I can send many packets at a time each time), that means I basically 
will have 10-30 ACKs in queue each time I get access to the medium.

For each transmit opportunity, you will see a "train of ACKs" with a 
duration of 0.1-0.5ms, then there will be nothing for 1-3ms, then you will 
see a train of ACks again.

So providing that the router in question actually knows what it's doing, 
this is a useful optimization. Actually, I'd say I don't really understand 
why the TCP stack is sending 10kpps anyhow, is this very fine grained 
resolution used for anything anyhow? Even considering that this "train of 
ACKs" will arrive back-to-back, is the difference that big compared to 
just sending fewer ACKs (from the host, so the router doesn't have to 
optimize).

My ISP has a very common service with is 250 megabit/s down and 10 
megabit/s up. If this TCP ACK suppression doesn't exist, then much of the 
customer upstream capacity is used for ACKing the download packets (for no 
great use for the user as far as I can tell).

Wouldn't it make sense to for TCP to limit ACKs to once per millisecond or 
1/10th RTT, whatever is lower? Or something along these lines, perhaps the 
1/10th of RTT is not low enough, so perhaps 1/50th of RTT.

For instance for my test here with around 10ms RTT, what would limit the 
ACKs from 10kpps to 2kpps in the host stack itself (with 1/50th of RTT 
being lower in this case).

It would at least make sense when using a time based media access layer 
such as LTE, DOCSIS etc.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Agarwal, Anil" <[email-masked]>
Date: Wed, 7 Oct 2015 20:40:54 +0000

Bless, Roland (TM) <[email-masked]>
Wrote -

> Since the cable modem link will lead to clumped ACKs the difference between sending 100 ACKs vs. 1 ACK is probably not that big...
> (except w.r.t. reliability).

The difference may not be big in the spacing of new packets that a sender will send, unless the sender implements some sort of pacing or if the return link is very thin.

But with ABC, there will be a difference in the amount of cwnd increase at the sender.
With 100 ACKs, each ACK will result in cwnd increase by 2 * MSS (assuming delayed Acks) during slow start and 
2 * MSS / cwnd during congestion avoidance.
With a single ACK covering 200 segments,  cwnd will increase by L*MSS, during slow start and 
L * MSS / cwnd during congestion avoidance; L =2, as stated in RFC 3465. Hence, cwnd increase will be much slower.
Unless L is set to a larger value.

Anil

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Mikael Abrahamsson
Sent: Wednesday, October 07, 2015 4:18 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [aqm] ACK Suppression

On Wed, 7 Oct 2015, [email-masked] wrote:

> not built up!  The purpose of queueing is to absorb bursts that can't 
> be anticipated, not to build up congestion in order to have enough 
> data to perform a dubious optimization that can best be done at the 
> source of traffic in cooperation with the destination.

If I download at 250 megabit/s, at 1500 byte packets, this is around 21kpps. That means I am sending around 10kpps worth of ACKs.

If my media access layer only gives me access to send packets every 1-3 ms (but I can send many packets at a time each time), that means I basically will have 10-30 ACKs in queue each time I get access to the medium.

For each transmit opportunity, you will see a "train of ACKs" with a duration of 0.1-0.5ms, then there will be nothing for 1-3ms, then you will see a train of ACks again.

So providing that the router in question actually knows what it's doing, this is a useful optimization. Actually, I'd say I don't really understand why the TCP stack is sending 10kpps anyhow, is this very fine grained resolution used for anything anyhow? Even considering that this "train of ACKs" will arrive back-to-back, is the difference that big compared to just sending fewer ACKs (from the host, so the router doesn't have to optimize).

My ISP has a very common service with is 250 megabit/s down and 10 megabit/s up. If this TCP ACK suppression doesn't exist, then much of the customer upstream capacity is used for ACKing the download packets (for no great use for the user as far as I can tell).

Wouldn't it make sense to for TCP to limit ACKs to once per millisecond or 1/10th RTT, whatever is lower? Or something along these lines, perhaps the 1/10th of RTT is not low enough, so perhaps 1/50th of RTT.

For instance for my test here with around 10ms RTT, what would limit the ACKs from 10kpps to 2kpps in the host stack itself (with 1/50th of RTT being lower in this case).

It would at least make sense when using a time based media access layer such as LTE, DOCSIS etc.

_______________________________________________
aqm mailing list
[email-masked]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_aqm&d=BQICAg&c=jcv3orpCsv7C4ly8-ubDob57ycZ4jvhoYZNDBA06fPk&r=FyvaklKYrHaSCPjbBTdviWIW9uSbnxdNSheSGz1Jvq4&m=O9YyDhG9aYPKoP_qfqzx2qr1WyA9s7n8zHybjDGhnkM&s=KQjsKwtKdO7fKEDJ2iWN7o0p-LVDX5-7it_eroz_ew0&e=

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 7 Oct 2015 23:57:19 +0300

> On 7 Oct, 2015, at 23:40, Agarwal, Anil <[email-masked]> wrote:
> 

> But with ABC, there will be a difference in the amount of cwnd increase at the sender.

There is also a potential difference for detecting packet loss in the forward direction.  It’s entirely possible that thinning would cause a DupAck condition to be recognised only after three MAC grants in the reverse direction have elapsed, rather than one.  Receivers are REQUIRED to send an ack for every received packet under these conditions, but that would be subverted by the modem.  AckCC would not induce this effect, because the receiver would still produce the extra acks as required.

Packet loss causes head-of-line blocking at the application level, which is perceived as latency and jerkiness by the end-user, until the lost packet is retransmitted and actually arrives.  Hence the addition of two MAC grant delays (60ms?) may make the difference between an imperceptible problem and a noticeable one.

 - Jonathan Morton

---

From: [email-masked]
Date: Wed, 7 Oct 2015 16:59:56 -0400 (EDT)

> There is no congestion when the ACK suppression kicks in and is useful.
We may be defining congestion *differently*.  If a backlog of packets from one endpoint to another builds up in an outbound queue, that is congestion.  That it might be caused by scheduling the "channel" is not relevant, really - at least in my definition of congestion.
 
So let's look at what TCP can do (or any other end-to-end protocol can do) to avoid such congestion:
 
1) knowing that the packets will be "hung up" in the middle (which it can observe by noticing that acks do get hung up), it can delay sending acks, since it will make no difference to send a cumulative ack rather than multiple acks. That's assuming that the link in question is the "bottleneck" link on the path.
 
2) the ideal TCP behavior here would be to bundle packets and to bundle ACKs, given that it can see that the bottleneck link has intermittent traffic openings with long gaps between open packet slots.
 
Let's look at what the MAC layer of the shared link can do:
 
1) It can bundle IP packets being sent from a station (from an arbitrary number of sources) into a cumulative bundle (concatenating them).  This is far more general than ACK consolidation, and covers protocols that have small packets, no ACKs at all, etc.  Obviously this is dependent on the specific MAC protocol, and the MAC protocol should maintain some degree of "fair sharing", because in essence there is one queue waiting for airtime, it is just distributed across many transmitters.  Thus the amount of bundling needs to be bounded.  Since arbitration time for channel access is wasted, the critical parameter is the arbitration overhead, as a percentage of maximum load - this controls the desired bundle size.
 
2) to the extent that packets will be bundled, there is a certain degree of compressability of the data.  For example, simple adaptive compression algorithms like Lempel-Ziv encoding are protocol independent.  If the frequent case is a series of ACKs, for example, Lempel-Ziv encoding the bundle will reduce the air time occupied by the bundle.
 
3) note that the critical parameter here is the arbitration overhead.  For the normal mode of the WiFi MAC, that is huge.  The preambles, etc. needed to deal with unequal received signal strength are relative large (multiple microseconds), plus the cost of a collision (due to lack of RTS/CTS usage that limits the cost of a collision).   Reducing the arbitration overhead is much more effective than ACK thinning.  On the principle of working on the big payoffs first, the idea of ACK thinning just doesn't seem to make sense outside a hypothetical that disallows much more significant improvements that don't require specialization to a particular version of TCP at a specific point in time.
 
In general, if you have a network that is NOT an Internet, which has a fixed set of protocols and a fixed and unchanging workload, of course you can tune every part of the system to that single case and get some benefit.
 
But this is the *Internet* Engineering Task Force.  The whole point of the Internet is to enable evolution.  That's why it is not a telephone voice network optimized for a 3 KHz voice coded in 8-bit digitized form with globally synchronized clocking.  And it doesn't have pre-allocation of fixed bit-rate flows.
 
That's the real engineering challenge.  Modularity is your friend when engineering large systems with very broad requirements.  Focusing on a very narrow issue, especially one that assumes way too much about what should be allowed and what traffic "must" look like, misses the point.

---

From: [email-masked]
Date: Wed, 7 Oct 2015 17:02:31 -0400 (EDT)

You are thinking that the point of a router is to be totally focused on TCP as it is at the moment.  cwnd is a TCP-only concept.  And in fact, you are focused on a "long-hold time" FTP-like TCP flow with maximal bitrate.
 
TCP in the Internet is not like that.  In ns2 experiments that are used to generate benchmarks, maybe.  But those benchmarks are rarely realistic.

On Wednesday, October 7, 2015 4:40pm, "Agarwal, Anil" <[email-masked]> said:

> > Since the cable modem link will lead to clumped ACKs the difference between
> sending 100 ACKs vs. 1 ACK is probably not that big...
> The difference may not be big in the spacing of new packets that a sender will
> send, unless the sender implements some sort of pacing or if the return link is
> very thin.
> 
> If I download at 250 megabit/s, at 1500 byte packets, this is around 21kpps. That
> means I am sending around 10kpps worth of ACKs.
> 
> If my media access layer only gives me access to send packets every 1-3 ms (but I
> can send many packets at a time each time), that means I basically will have 10-30
> ACKs in queue each time I get access to the medium.
> 
> For each transmit opportunity, you will see a "train of ACKs" with a duration of
> 0.1-0.5ms, then there will be nothing for 1-3ms, then you will see a train of ACks
> again.
> 
> So providing that the router in question actually knows what it's doing, this is a
> useful optimization. Actually, I'd say I don't really understand why the TCP stack
> is sending 10kpps anyhow, is this very fine grained resolution used for anything
> anyhow? Even considering that this "train of ACKs" will arrive back-to-back, is
> the difference that big compared to just sending fewer ACKs (from the host, so the
> router doesn't have to optimize).
> 
> My ISP has a very common service with is 250 megabit/s down and 10 megabit/s up.
> If this TCP ACK suppression doesn't exist, then much of the customer upstream
> capacity is used for ACKing the download packets (for no great use for the user as
> far as I can tell).
> 
> Wouldn't it make sense to for TCP to limit ACKs to once per millisecond or 1/10th
> RTT, whatever is lower? Or something along these lines, perhaps the 1/10th of RTT
> is not low enough, so perhaps 1/50th of RTT.
> 
> For instance for my test here with around 10ms RTT, what would limit the ACKs from
> 10kpps to 2kpps in the host stack itself (with 1/50th of RTT being lower in this
> case).
>

---

From: [email-masked]
Date: Wed, 7 Oct 2015 17:08:51 -0400 (EDT)

The IETF used to define itself by rough consensus and working code.
 
So we can observe some of these things.  In the end of the day, the problem is to make a platform on which many applications, with many design approaches, and whose goals we cannot appreciate at the "network layer", work well in the real world.
 
Protocol designers should actually spend some time working on large scale systems, writing code in JavaScript and server-side frameworks, or developing conference systems or MMORPG's.  These are the real issues, and congestion matters.  But the problems cannot be "defined away" by focusing on a single TCP flow on a simplified network where everything else is assumed to be gaussian or poisson "noise" that can be ignored.

On Wednesday, October 7, 2015 4:57pm, "Jonathan Morton" <[email-masked]> said:

> >> Since the cable modem link will lead to clumped ACKs the difference
> between sending 100 ACKs vs. 1 ACK is probably not that big...
> > The difference may not be big in the spacing of new packets that a sender
> will send, unless the sender implements some sort of pacing or if the return link
> is very thin.
> 
> There is also a potential difference for detecting packet loss in the forward
> direction. It’s entirely possible that thinning would cause a DupAck
> condition to be recognised only after three MAC grants in the reverse direction
> have elapsed, rather than one. Receivers are REQUIRED to send an ack for every
> received packet under these conditions, but that would be subverted by the modem. 
> AckCC would not induce this effect, because the receiver would still produce the
> extra acks as required.
> 
> Packet loss causes head-of-line blocking at the application level, which is
> perceived as latency and jerkiness by the end-user, until the lost packet is
> retransmitted and actually arrives. Hence the addition of two MAC grant delays
> (60ms?) may make the difference between an imperceptible problem and a noticeable
> one.
> 
> - Jonathan Morton
> 
>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 7 Oct 2015 21:12:27 +0000

On Wednesday, October 7, 2015 2:00 PM, [email-masked] wrote:

> That's the real engineering challenge.  Modularity is your friend when engineering 
> large systems with very broad requirements.  Focusing on a very narrow issue, especially 
> one that assumes way too much about what should be allowed and what traffic "must" 
> look like, misses the point.

This ACK thinning "optimization" obviously does not work with encrypted transports like QUIC. The network may see that some packets are short and some are long, but it cannot tell whether the short
Packets are ACK or some other management data. Quite often, they will be both. 

According to our friends at Google, a fairly large part of the traffic from and to the Chrome browser now uses QUIC. If the uplink in DOCSIS networks is so limited that it would not work without ACK thinning, that would be a problem for QUIC -- and very much an illustration of Dave's "real engineering challenge." I wonder whether QUIC telemetry shows specific issues caused by congestions of the DOCSIS uplinks.

-- Christian Huitema

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 14:13:09 -0700 (PDT)

On Wed, 7 Oct 2015, Jonathan Morton wrote:

>>> sending 100 ACKs vs. 1 ACK is probably not that big... (except w.r.t. 
>>> reliability).
>> 
>> will send, unless the sender implements some sort of pacing or if the return 
>> link is very thin.
>
> condition to be recognised only after three MAC grants in the reverse 
> direction have elapsed, rather than one.  Receivers are REQUIRED to send an 
> ack for every received packet under these conditions, but that would be 
> subverted by the modem.  AckCC would not induce this effect, because the 
> receiver would still produce the extra acks as required.
>
> retransmitted and actually arrives.  Hence the addition of two MAC grant 
> delays (60ms?) may make the difference between an imperceptible problem and a 
> noticeable one.

and excessive ack traffic causes congestion and results in packet loss on 
real-world hightly asymmetric links.

So things that reduce the flow of acks can result in very real benefits to 
users.

David Lang

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 14:39:36 -0700 (PDT)

On Wed, 7 Oct 2015, [email-masked] wrote:

> We may be defining congestion *differently*.  If a backlog of packets from one 
> endpoint to another builds up in an outbound queue, that is congestion.  That 
> it might be caused by scheduling the "channel" is not relevant, really - at 
> least in my definition of congestion.
> 
> 1) knowing that the packets will be "hung up" in the middle (which it can 
> observe by noticing that acks do get hung up), it can delay sending acks, 
> since it will make no difference to send a cumulative ack rather than multiple 
> acks. That's assuming that the link in question is the "bottleneck" link on 
> the path.
> 
> 2) the ideal TCP behavior here would be to bundle packets and to bundle ACKs, 
> given that it can see that the bottleneck link has intermittent traffic 
> openings with long gaps between open packet slots.

the endpoints may be sufficiently isolated from the bottleneck that they can't 
tell the timeing of packets precisely enough to detect this traffic pattern

> 1) It can bundle IP packets being sent from a station (from an arbitrary 
> number of sources) into a cumulative bundle (concatenating them).  This is far 
> more general than ACK consolidation, and covers protocols that have small 
> packets, no ACKs at all, etc.  Obviously this is dependent on the specific MAC 
> protocol, and the MAC protocol should maintain some degree of "fair sharing", 
> because in essence there is one queue waiting for airtime, it is just 
> distributed across many transmitters.  Thus the amount of bundling needs to be 
> bounded.  Since arbitration time for channel access is wasted, the critical 
> parameter is the arbitration overhead, as a percentage of maximum load - this 
> controls the desired bundle size.
> 
> 2) to the extent that packets will be bundled, there is a certain degree of 
> compressability of the data.  For example, simple adaptive compression 
> algorithms like Lempel-Ziv encoding are protocol independent.  If the frequent 
> case is a series of ACKs, for example, Lempel-Ziv encoding the bundle will 
> reduce the air time occupied by the bundle.
> 
> 3) note that the critical parameter here is the arbitration overhead.  For the 
> normal mode of the WiFi MAC, that is huge.  The preambles, etc. needed to deal 
> with unequal received signal strength are relative large (multiple 
> microseconds), plus the cost of a collision (due to lack of RTS/CTS usage that 
> limits the cost of a collision).  Reducing the arbitration overhead is much 
> more effective than ACK thinning.  On the principle of working on the big 
> payoffs first, the idea of ACK thinning just doesn't seem to make sense 
> outside a hypothetical that disallows much more significant improvements that 
> don't require specialization to a particular version of TCP at a specific 
> point in time.

packets that don't have to be sent beat packets that are bundled and/or 
compressed by a large margin :-)

compression takes a lot of cpu, and in many cases the devices have enough 
trouble shoving packets around, doing compression on them is going to end up 
with a net loss in throughput.

David Lang

_______________________________________________
aqm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/aqm

---

From: David Lang <[email-masked]>
Date: Wed, 7 Oct 2015 14:50:33 -0700 (PDT)

On Wed, 7 Oct 2015, [email-masked] wrote:

> Date: Wed, 7 Oct 2015 15:52:18 -0400 (EDT)
> From: [email-masked]
> To: [email-masked]
> 1) TCP ACKs are TCP's business only.  It's not a gateway or router's job to 
> get involved in or to understand end-to-end protocols, *even if* the router 
> thinks it knows exactly what the endpoints' goals are.  And the router cannot 
> know that for every protocol, not even the many higher level protocols on top 
> of TCP, which use TCP quite differently.  The idea that routers can be 
> omniscient, merely by looking at packets and taking the designers' personal 
> prejudices into account, seems ridiculous. TCP endpoints on both ends of a 
> connection can reduce the number of ACKs they send if they want. If ACKs are 
> filling up buffers in intermediate routers, just drop them or mark them to 
> notify that they are contributing to congestion.  The endpoints have to slow 
> down something, and they can slow down ACKs by mutual agreement.
> 
> 2) The hypothetical that there will be a sufficiently long sequence of ACKs 
> for a single end-to-end flow buffered in a single router queue may seem 
> plausible, *until it becomes clear that in the big picture, having so many 
> packets in a queue means that the network is extremely congested by that 
> point*.  In other words, in order for this "optimization" to apply, you would 
> have to operate the network at an unacceptable operating point! It's like 
> saying that when a highway has slowed to a crawl, we can load all the cars 
> going to a particular destination onto single "car carrier" to save gas. Far 
> better to insure that queues are not built up!  The purpose of queueing is to 
> absorb bursts that can't be anticipated, not to build up congestion in order 
> to have enough data to perform a dubious optimization that can best be done at 
> the source of traffic in cooperation with the destination.

Look at Wifi, you have close to 1Gb/s data rates, but it's still half-duplex. So 
while a station/AP is receiving, there can be a substantial number of ack 
packets that queue up while recieving even a 'small' bundle of data packets.

This is normal operation, not 'operate the network at an unacceptable operating 
point'

David Lang

---

From: Greg White <[email-masked]>
Date: Wed, 7 Oct 2015 22:49:13 +0000

Jonathan, 

You are making some assumptions about modem behavior that may or may not
be true.  RFC3449 does mention that DupACKs should be treated with care,
but I honestly don't know what implementations do in this regard.  (and
btw, two MAC grant delays in DOCSIS would be more like 2-10ms)

Others,

Just so everyone is clear, DOCSIS modems have been doing these sorts of
optimizations for close to 15 years.  So, if your analysis concludes that
the Internet will catch fire as a result, you may want to double check
your math.  I am interested in knowing about real issues, and whether we
need to make more specific guidelines (or requirements) on what vendors do
in this space, but I think they need to be pragmatic guidelines (and not
assume that transports are going to become hyper L2-aware, or that modems
are going to become hyper transport-aware).

I like Mikael's suggestion that the TCP default behavior should be to try
to send ACKs one per millisecond or 1/10th RTT (whichever is lower).
Perhaps QUIC can raise the bar here....

-Greg

On 10/7/15, 2:57 PM, "aqm on behalf of Jonathan Morton"
<[email-masked] on behalf of [email-masked]> wrote:

>> The difference may not be big in the spacing of new packets that a
>>sender will send, unless the sender implements some sort of pacing or if
>>the return link is very thin.
>
>There is also a potential difference for detecting packet loss in the
>forward direction.  It¹s entirely possible that thinning would cause a
>DupAck condition to be recognised only after three MAC grants in the
>reverse direction have elapsed, rather than one.  Receivers are REQUIRED
>to send an ack for every received packet under these conditions, but that
>would be subverted by the modem.  AckCC would not induce this effect,
>because the receiver would still produce the extra acks as required.
>
>Packet loss causes head-of-line blocking at the application level, which
>is perceived as latency and jerkiness by the end-user, until the lost
>packet is retransmitted and actually arrives.  Hence the addition of two
>MAC grant delays (60ms?) may make the difference between an imperceptible
>problem and a noticeable one.
>
> - Jonathan Morton
>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 7 Oct 2015 15:56:27 -0700

I'll be quick: we haven't actively looked at ack loss as an issue, since
QUIC's mechanisms are largely independent of ack loss. It is entirely
possible that we're building up a queue at the uplink, but this really
shouldn't be a big issue in the common case.

That said, we'd like to solve for reducing ack packets in QUIC, and we are
working on it as an e2e problem since reducing signaling redundancy seems
like a sensible thing to do. Fortunately for us, middleboxes have no idea
(yet) of what QUIC ack packets look like since they are encrypted, so we
should be able to clearly measure issues/benefits with and without
ack-decimation.

We definitely see in-network ack-decimation happening for TCP.

On Wed, Oct 7, 2015 at 2:12 PM, Christian Huitema <[email-masked]>
wrote:

> This ACK thinning "optimization" obviously does not work with encrypted
> transports like QUIC. The network may see that some packets are short and
> some are long, but it cannot tell whether the short
> According to our friends at Google, a fairly large part of the traffic
> from and to the Chrome browser now uses QUIC. If the uplink in DOCSIS
> networks is so limited that it would not work without ACK thinning, that
> would be a problem for QUIC -- and very much an illustration of Dave's
> "real engineering challenge." I wonder whether QUIC telemetry shows
> specific issues caused by congestions of the DOCSIS uplinks.
>
> -- Christian Huitema
>
>
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 8 Oct 2015 08:52:14 +0000

This discussion affects a number of different WGs, and, indeed, it may be well homed on the AQM list.

But I'd suggest to move any detailed discussion on TCP modifications (i.e., reducing the ACK frequency, ACk/reverse congestion control, etc.) to the TCPM list. Not all TCPM contributors may closely follow the AQM list ... including the chairs.

Michael

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Greg White
Sent: Thursday, October 08, 2015 12:49 AM
To: Jonathan Morton; Agarwal, Anil
Cc: [email-masked]; [email-masked]; Mikael Abrahamsson
Subject: Re: [aqm] ACK Suppression

Jonathan, 

You are making some assumptions about modem behavior that may or may not be true.  RFC3449 does mention that DupACKs should be treated with care, but I honestly don't know what implementations do in this regard.  (and btw, two MAC grant delays in DOCSIS would be more like 2-10ms)

Others,

Just so everyone is clear, DOCSIS modems have been doing these sorts of optimizations for close to 15 years.  So, if your analysis concludes that the Internet will catch fire as a result, you may want to double check your math.  I am interested in knowing about real issues, and whether we need to make more specific guidelines (or requirements) on what vendors do in this space, but I think they need to be pragmatic guidelines (and not assume that transports are going to become hyper L2-aware, or that modems are going to become hyper transport-aware).

I like Mikael's suggestion that the TCP default behavior should be to try to send ACKs one per millisecond or 1/10th RTT (whichever is lower).
Perhaps QUIC can raise the bar here....

-Greg

On 10/7/15, 2:57 PM, "aqm on behalf of Jonathan Morton"
<[email-masked] on behalf of [email-masked]> wrote:

>>sender will send, unless the sender implements some sort of pacing or 
>>if the return link is very thin.
>
>reverse direction have elapsed, rather than one.  Receivers are 
>REQUIRED to send an ack for every received packet under these 
>conditions, but that would be subverted by the modem.  AckCC would not 
>induce this effect, because the receiver would still produce the extra acks as required.
>
>Packet loss causes head-of-line blocking at the application level, 
>which is perceived as latency and jerkiness by the end-user, until the 
>lost packet is retransmitted and actually arrives.  Hence the addition 
>of two MAC grant delays (60ms?) may make the difference between an 
>imperceptible problem and a noticeable one.
>
> - Jonathan Morton
>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 8 Oct 2015 11:44:54 +0200

Hi David,

Am 07.10.2015 um 21:52 schrieb [email-masked]:
> 1) TCP ACKs are TCP's business only.  It's not a gateway or router's job
> to get involved in or to understand end-to-end protocols, *even if* the
> router thinks it knows exactly what the endpoints' goals are.  And the
> router cannot know that for every protocol, not even the many higher
> level protocols on top of TCP, which use TCP quite differently.  The
> idea that routers can be omniscient, merely by looking at packets and
> taking the designers' personal prejudices into account, seems
> ridiculous. TCP endpoints on both ends of a connection can reduce the
> number of ACKs they send if they want. If ACKs are filling up buffers in
> intermediate routers, just drop them or mark them to notify that they
> are contributing to congestion.  The endpoints have to slow down
> something, and they can slow down ACKs by mutual agreement.

+1

> 2) The hypothetical that there will be a sufficiently long sequence of
> ACKs for a single end-to-end flow buffered in a single router queue may
> seem plausible, *until it becomes clear that in the big picture, having
> so many packets in a queue means that the network is extremely congested
> by that point*.  In other words, in order for this "optimization" to
> apply, you would have to operate the network at an unacceptable
> operating point! It's like saying that when a highway has slowed to a
> crawl, we can load all the cars going to a particular destination onto
>  single "car carrier" to save gas. Far better to insure that queues are

+1
Maybe an incentive for some people to think about alternative link
layer access schemes that will be better suited for such kind of
Internet traffic. As already pointed out the specific optimization will
be useless for newer transport protocols as well as for tunneled or
encrypted traffic or advanced TCP features, including ECN feedback.

> It's said that in committees the amount of time spent on trivialities
> far exceeds the time spent on important issues.  That seems to be true
> as I watch the discussion on this list.

I think the "discussion" seems to be necessary since Mikael's original
question was:
>> Now, this kind of mechanism, how should it be treated when it comes
>> to AQM? This mechanism is basically done at de-queue, when a number of
>> packets are emptied from the queue at one time, which is then allowed to
>> fill up again until the next transmit opportunity arises.

I really hate it if the IETF tries to work around "broken"
(short-sighted, cross-layer optimized, and so on) behavior
of middleboxes or other devices. So I don't think that the AQM
WG should take into account this particular optimization
of specific link layer boxes. Otherwise, we'll make the situation
only worse for transport protocol evolution. We've got enough
examples for ossification due to non-farseeing implementations
of middlebox stuff. It's quite perverted if we start to design
mechanisms according to such kind of "special/broken" behavior.

Regards,
 Roland

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 8 Oct 2015 18:22:52 +0000

I'm not sure why this discussion is happening on aqm@ instead of tcpm@... I have added cpm to the cc line, and would recommend that anyone responding to this thread do the same and remove aqm@.

On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked]> wrote:
> So things that reduce the flow of acks can result in very real benefits to users.

Dumb question of the month. What would it take to see wide deployment of RFC 5690? That would result in the data/ack ratio being reduced, on average, to whatever amount had been negotiated.

Summary for those that haven't read it - TCP implementations today generally ack every other packet, with caveats for isolated or final data packets. This proposal allows consenting adults to change that ratio, acking every third or fourth packet, or every tenth.

If ack reductions are so very valuable, what's the chance of doing that on an end to end basis instead of in the network?

> On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked]> wrote:
>>>> Since the cable modem link will lead to clumped ACKs the difference between sending 100 ACKs vs. 1 ACK is probably not that big... (except w.r.t. reliability).
> and excessive ack traffic causes congestion and results in packet loss on real-world hightly asymmetric links.
> 
> So things that reduce the flow of acks can result in very real benefits to users.
> 
> David Lang_______________________________________________
> aqm mailing list
> [email-masked]

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 11:34:21 -0700 (PDT)

On Thu, 8 Oct 2015, Fred Baker (fred) wrote:

> I'm not sure why this discussion is happening on aqm@ instead of tcpm@... I 
> have added cpm to the cc line, and would recommend that anyone responding to 
> this thread do the same and remove aqm@.

I don't know why it started here, but not everyone on the aqm list is on the 
tcpm list, so removing aqm@ will drop some people from the conversation.

> Dumb question of the month. What would it take to see wide deployment of RFC 
> 5690? That would result in the data/ack ratio being reduced, on average, to 
> whatever amount had been negotiated.

well, given that people are still running Windows XP, it may get out to some 
things but it's not going to get out to everything. Remember that VMware 
responded to the EoL of XP with a campaign telling people they could keep 
running XP in a VM.

in particular streaming media devices (smart TVs, etc) are not getting many, if 
any, OS upgrades from the vendors, and they are a big part of the problem.

> Summary for those that haven't read it - TCP implementations today generally 
> ack every other packet, with caveats for isolated or final data packets. This 
> proposal allows consenting adults to change that ratio, acking every third or 
> fourth packet, or every tenth.
>
> If ack reductions are so very valuable, what's the chance of doing that on an 
> end to end basis instead of in the network?

a little less than the chance of  shutting off IPv4  :-)

requiring changes to all the software on each end to enable this is wishful 
thinking. The major servers could get the update pretty promtly, but updating 
the client side?? not for a long time.

But in any case, that's addressing a slightly different issue. This discussion 
started based on the fact that when there are queues, and the queues have 
multiple acks for the same flow, it can save queue space, bandwidth, and reduce 
latency by eliminating all but the last ack. If this can be done at a low cpu 
cost (so it can be implemented on consumer endpoint hardware that usually has 
the first bottleneck), it could be a win for the Internet as a whole.

David Lang

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 8 Oct 2015 19:22:16 +0000

> On Oct 8, 2015, at 11:34 AM, David Lang <[email-masked]> wrote:
> 

Hey, we can dream, can't we? :-)

> requiring changes to all the software on each end to enable this is wishful thinking. The major servers could get the update pretty promtly, but updating the client side?? not for a long time.

Yes, of course. That said, consider trends like

http://www.zdnet.com/article/latest-os-share-data-shows-windows-still-dominating-in-pcs/
http://www.netmarketshare.com/operating-system-market-share.aspx?qprid=11&qpcustomb=0
https://www.pinterest.com/pin/137148751123435826/

What that basically says is that corporations tend to pick a version and install every update after testing it. The folks still on XP are very likely people who installed a (potentially pirated) release in 2003 and have probably never updated it. If we can get this into the next MacOSX 10.11 and Windows 10 update cycle, roughly 50% of computers will have it within six months, maybe three. If we can get it into common Linux distributions, we can probably cover the servers as well. 100% deployment - that might take until the XP computers actually all fry. Getting it significantly deployed - I don't see the cause of the pessimism.

You're correct that one has to update set-top-boxes and smart TVs. They get updates too, or at least mine do. It doesn't require buying a new TV, it requires getting the updated software.

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 12:27:16 -0700 (PDT)

On Thu, 8 Oct 2015, Fred Baker (fred) wrote:

> What that basically says is that corporations tend to pick a version and 
> install every update after testing it. The folks still on XP are very likely 
> people who installed a (potentially pirated) release in 2003 and have probably 
> never updated it. If we can get this into the next MacOSX 10.11 and Windows 10 
> update cycle, roughly 50% of computers will have it within six months, maybe 
> three. If we can get it into common Linux distributions, we can probably cover 
> the servers as well. 100% deployment - that might take until the XP computers 
> actually all fry. Getting it significantly deployed - I don't see the cause of 
> the pessimism.

Yes, you can get it to most desktops and servers in a few years (but keep in 
mind the 5+ year supported lifetime of Enterprise Linux distros)

But it's the non desktop clients that are going to be the problem

> You're correct that one has to update set-top-boxes and smart TVs. They get 
> updates too, or at least mine do. It doesn't require buying a new TV, it 
> requires getting the updated software.

when the vendors essentially abandon the devices after manufacture, getting them 
to update the OS is going to be hard.

David Lang

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 14:27:14 -0700

On 10/7/2015 2:12 PM, Christian Huitema wrote:
> This ACK thinning "optimization" obviously does not work with encrypted transports like QUIC.

Nor would it work with TCP-AO.

Joe

---

From: [email-masked]
Date: Fri, 9 Oct 2015 04:03:41 +0100

RFC3449 pointed to some of the reasons & problems with manipulating TCP
ACKs, and could provide a useful background if people needed to get up to
speed on the history...

Gorry

> I'm not sure why this discussion is happening on aqm@ instead of tcpm@...
> I have added cpm to the cc line, and would recommend that anyone
> responding to this thread do the same and remove aqm@.
>
> Dumb question of the month. What would it take to see wide deployment of
> RFC 5690? That would result in the data/ack ratio being reduced, on
> average, to whatever amount had been negotiated.
>
> Summary for those that haven't read it - TCP implementations today
> generally ack every other packet, with caveats for isolated or final data
> packets. This proposal allows consenting adults to change that ratio,
> acking every third or fourth packet, or every tenth.
>
> If ack reductions are so very valuable, what's the chance of doing that on
> an end to end basis instead of in the network?
>
>>> forward direction.  It’s entirely possible that thinning would cause
>>> a DupAck condition to be recognised only after three MAC grants in the
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: David Lang <[email-masked]>
Date: Thu, 8 Oct 2015 22:02:08 -0700 (PDT)

Ok, so it turns out that what we are talking about here is talked about 
explicitly in section 5.2.1 of RFC3449. It warns of increased sender burst 
sizes, but I really question if that's a valid concern in the cases that we are 
talking about here. The choice in these cases isn't to send more acks spaced 
out, but rather to send a lot of acks back to back in one transmit slot vs one 
stretch ack per transmit slot.

RFC3449 doesn't address this directly, but it does address it indirectly by 
proposing several mechanisms to recreate the missing acks. These all talk about 
the need to space out the re-created ack messages to prevetnt large transmit 
bursts. This implies that if the re-created acks were not spaced out, the 
various problems would still happen. [1]

In the case we are talking about, the choice is between spending bandwidth on 
sending the acks back to back (the equivalent of re-creating the acks and not 
spacing them) or not just sending one ack. This looks to me like they have the 
equivalent effect.

Section 5.3.3 talking about ways to mitigate excessive transmit burst sizes 
sounds like a primitive version of running AQM on the downstream side. It looks 
to me like any of the currently considered strong AQM options would satisfy this 
requiement.

The final recommendations section says about 5.3.1 "Use in the Internet is 
dependent on a scheme for preventing excessive TCP transmission bursts"

Since AQM on the downstream side does exactly that, it looks like thinning out 
duplicate acks in the AQM queue is actually following the recommendations of 
RFC3449, as long as AQM in also in place on the downstream side

[1] A major problem with any of the options that try to reconstruct the lost 
acks is that in order to space out the acks appropriately, it can only do so by 
delaying the ack that it actually received.

David Lang

On Fri, 9 Oct 2015, [email-masked] wrote:

>>>> forward direction.  Itâ??s entirely possible that thinning would cause

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Oct 2015 08:43:44 +0200 (CEST)

On Thu, 8 Oct 2015, David Lang wrote:

> explicitly in section 5.2.1 of RFC3449. It warns of increased sender 
> burst sizes, but I really question if that's a valid concern in the 
> cases that we are talking about here. The choice in these cases isn't to 
> send more acks spaced out, but rather to send a lot of acks back to back 
> in one transmit slot vs one stretch ack per transmit slot.

Exactly. And in my specific case, we're talking 20kPPS of data down, and 
10kPPS of ACKs, over a medium that allows the modem to send a bunch of 
packets only once every few milliseconds.

So there are two alternatives:

You get approx 10-60 ACKs as a "train of ACKs" within 0.3 milliseconds, 
every 1-4 milliseconds.

You get 1-3 ACKs within 0.1ms, every 2-3 milliseconds. These ACKs will 
acknowledge 16-32 packets at a time.

By the nature of the beast, I'd say that this kind of medium implicitly 
will be able to handle the "bursts" we're talking about here, if you ACK 
24-48kilobyte of data per ACK (when a middle box does ACK Suppression to 
remove intermediate ACKs), it's implicit that the medium designers know 
that they can handle a wirespeed burst of data that is the result of the 
ACKing of these fairly large number of bytes.

But again, I question the wisdom in ACKing every 2 packets when the 
congestion window is in the hundreds of kilobytes and you're sending tens 
of thousands of packets per second.

I suggested in another part of the thread that perhaps the host itself 
should do ACK suppression and don't send ACKs more closely together than 
once per millisecond or 1/10th RTT, whatever is lower. Perhaps the RTT 
value needs to be even lower, 1/25 of RTT or something.

In my specific test, using once per millisecond or 1/25 of RTT would cut 
down the number of ACKs to around 2kpps instead of the 10kpps ACKs I see 
now, but it wouldn't cut it down to the around 500-1000 ACKs per second 
that the DOCSIS modem is cutting it down to...

I think that as bandwidths go up, there are changes to TCP that can be 
done to make it work better, and reducing the frequency of ACKs sent 
depending on RTT and different window sizes might be a way forward. I 
fully believe there are cases where ACKing every 2 packets is exactly the 
right way to go, but when you're sending 1 gigabit/s over 100ms RTT and 
the medium itself clumps the packets together because that's what it does, 
then these ACKs don't carry the same information anymore and some of them 
could be optimized away (preferrably by the host stack) to spare upstream 
bandwidth and transmit opportunities.

---

From: Greg White <[email-masked]>
Date: Mon, 12 Oct 2015 22:34:06 +0000

From: aqm <[email-masked]<mailto:[email-masked]>> on behalf of Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, October 7, 2015 at 4:56 PM
To: Christian Huitema <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [aqm] ACK Suppression

I'll be quick: we haven't actively looked at ack loss as an issue, since QUIC's mechanisms are largely independent of ack loss. It is entirely possible that we're building up a queue at the uplink, but this really shouldn't be a big issue in the common case.

That said, we'd like to solve for reducing ack packets in QUIC, and we are working on it as an e2e problem since reducing signaling redundancy seems like a sensible thing to do. Fortunately for us, middleboxes have no idea (yet) of what QUIC ack packets look like since they are encrypted, so we should be able to clearly measure issues/benefits with and without ack-decimation.

But, unfortunately for you, middleboxes have no idea (yet) of what QUIC ack packets look like, so they can't prioritize them.   Because of bufferbloat, downstream TCP throughput would really suffer if upstream ACKs got stuck at the tail of a 3000 millisecond upstream queue.  This won't be a problem for QUIC in the future when AQM is deployed, but on existing cable equipment you should expect to see the full effects of upstream bufferbloat (whereas TCP won't).

On Wed, Oct 7, 2015 at 2:12 PM, Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

-- Christian Huitema

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 13 Oct 2015 01:39:56 +0300

> On 13 Oct, 2015, at 01:34, Greg White <[email-masked]> wrote:
> 
> But, unfortunately for you, middleboxes have no idea (yet) of what QUIC ack packets look like, so they can't prioritize them.

If they are small, then at least some boxes will take that as a sufficient hint.

 - Jonathan Morton

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 12 Oct 2015 16:23:55 -0700

> > But, unfortunately for you, middleboxes have no idea (yet) of what QUIC
> ack packets look like, so they can't prioritize them.
>

Fair point, but that's not unfortunate, it's by design.  We'd like to try
doing things e2e before requiring middleboxes to help, even in this case.
QUIC has the ability to measure delay increases on the forward path, so our
congestion controller can (doesn't yet, but will) filter out delay
variations on the reverse path.

---

From: Simon Barber <[email-masked]>
Date: Mon, 12 Oct 2015 19:57:24 -0700

The issue here is that the hosts don't know how the medium is going to 
do the clumping, so thinning the ACKs at the host may add additional delay.

 From what I understand from Greg's earlier message the cable modem must 
request uplink bandwidth in advance, so if it had to send multiple ACKs, 
it would send the first one, then request bandwidth the the additional 
ones that had arrived since the first one (please correct me if I'm 
wrong) - in this case collapsing/thinning the ACKs will reduce latency 
by one grant cycle.

WiFi is quite different - there is no need to request bandwidth in 
advance, so there is less pressure to collapse the ACKs - just send them 
all and they will be burst out when the medium becomes available.

Simon

On 10/8/2015 11:43 PM, Mikael Abrahamsson wrote:
>> Ok, so it turns out that what we are talking about here is talked 
>> about explicitly in section 5.2.1 of RFC3449. It warns of increased 
>> sender burst sizes, but I really question if that's a valid concern 
>> in the cases that we are talking about here. The choice in these 
>> cases isn't to send more acks spaced out, but rather to send a lot of 
>> acks back to back in one transmit slot vs one stretch ack per 
>> transmit slot.
>
> Exactly. And in my specific case, we're talking 20kPPS of data down, 
> and 10kPPS of ACKs, over a medium that allows the modem to send a 
> bunch of packets only once every few milliseconds.
>
> You get approx 10-60 ACKs as a "train of ACKs" within 0.3 
> milliseconds, every 1-4 milliseconds.
>
> By the nature of the beast, I'd say that this kind of medium 
> implicitly will be able to handle the "bursts" we're talking about 
> here, if you ACK 24-48kilobyte of data per ACK (when a middle box does 
> ACK Suppression to remove intermediate ACKs), it's implicit that the 
> medium designers know that they can handle a wirespeed burst of data 
> that is the result of the ACKing of these fairly large number of bytes.
>
> congestion window is in the hundreds of kilobytes and you're sending 
> tens of thousands of packets per second.
>
> should do ACK suppression and don't send ACKs more closely together 
> than once per millisecond or 1/10th RTT, whatever is lower. Perhaps 
> the RTT value needs to be even lower, 1/25 of RTT or something.
>
> In my specific test, using once per millisecond or 1/25 of RTT would 
> cut down the number of ACKs to around 2kpps instead of the 10kpps ACKs 
> I see now, but it wouldn't cut it down to the around 500-1000 ACKs per 
> second that the DOCSIS modem is cutting it down to...
>
> fully believe there are cases where ACKing every 2 packets is exactly 
> the right way to go, but when you're sending 1 gigabit/s over 100ms 
> RTT and the medium itself clumps the packets together because that's 
> what it does, then these ACKs don't carry the same information anymore 
> and some of them could be optimized away (preferrably by the host 
> stack) to spare upstream bandwidth and transmit opportunities.
>

---

From: Simon Barber <[email-masked]>
Date: Mon, 12 Oct 2015 20:00:41 -0700

The problem is that many of these link layers are working around 
physical limitations, and cannot avoid clumping packets together or 
delaying them in some way.

Simon

---

From: David Lang <[email-masked]>
Date: Mon, 12 Oct 2015 20:26:09 -0700 (PDT)

True, assuming that they will all fit in the next burst.

but sending all of them lengthens the transmission compared to only sending the 
minimum, which consumes airtime that that or some other station could be using 
for data. With congestion of the airwaves such a significant problem, being able 
to not use some of it is a win overall, even if it doesn't affect a benchmarkk 
for one station in isolation.

David Lang

On Mon, 12 Oct 2015, Simon Barber wrote:

> The issue here is that the hosts don't know how the medium is going to do the 
> clumping, so thinning the ACKs at the host may add additional delay.
>
> request uplink bandwidth in advance, so if it had to send multiple ACKs, it 
> would send the first one, then request bandwidth the the additional ones that 
> had arrived since the first one (please correct me if I'm wrong) - in this 
> case collapsing/thinning the ACKs will reduce latency by one grant cycle.
>
> WiFi is quite different - there is no need to request bandwidth in advance, 
> so there is less pressure to collapse the ACKs - just send them all and they 
> will be burst out when the medium becomes available.
>
> Simon
>
>>> sizes, but I really question if that's a valid concern in the cases that 
>>> we are talking about here. The choice in these cases isn't to send more 
>>> acks spaced out, but rather to send a lot of acks back to back in one 
>>> transmit slot vs one stretch ack per transmit slot.
>> 
>> I think that as bandwidths go up, there are changes to TCP that can be done 
>> to make it work better, and reducing the frequency of ACKs sent depending 
>> on RTT and different window sizes might be a way forward. I fully believe 
>> there are cases where ACKing every 2 packets is exactly the right way to 
>> go, but when you're sending 1 gigabit/s over 100ms RTT and the medium 
>> itself clumps the packets together because that's what it does, then these 
>> ACKs don't carry the same information anymore and some of them could be 
>> optimized away (preferrably by the host stack) to spare upstream bandwidth 
>> and transmit opportunities.
>> 
>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 13 Oct 2015 11:43:31 +0200

Hi,

> delaying them in some way.

Sure, but that wasn't my main point in the mail below.
I can't tell whether alternative link layer access
schemes could have led to smaller request slot times,
which would have been beneficial for smoother ACK feedback.
Not sure whether https://tools.ietf.org/html/rfc3819 is well know
to link layer designers...

Regards,
 Roland

---

From: Greg White <[email-masked]>
Date: Tue, 13 Oct 2015 18:34:32 +0000

>wrong) - in this case collapsing/thinning the ACKs will reduce latency

You've got it right. Also, consider cases where there is other upstream
traffic.  AFAIK ACK thinning is only implemented along with ACK
prioritization.  One could imagine only doing prioritization, without
thinning.  In that case, if other upstream traffic were present, you may
be able to get some of the latency reduction benefits by allowing the
newly arrived ACKs to be sent as a burst in a grant that  had been sized
to carry some other packets, but then you are delaying the other traffic
by (at least) one grant cycle. Thinning in combination with prioritization
reduces ACK latency with less impact on other applications.

---

From: [email-masked]
Date: Thu, 15 Oct 2015 11:08:32 -0400 (EDT)

Small packets shouldn't be supressed just because they are small... no ssh connections would work.

On Monday, October 12, 2015 6:39pm, "Jonathan Morton" <[email-masked]> said:

> > But, unfortunately for you, middleboxes have no idea (yet) of what QUIC ack
> packets look like, so they can't prioritize them.
>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 15 Oct 2015 18:13:23 +0300

> On 15 Oct, 2015, at 18:08, [email-masked] wrote:
> 

Of course not, that would be stupid.

But some network nodes do *prioritise* packets based solely on size, since (as a heuristic) latency-sensitive packets do tend to be small.  That’s what I was pointing out.

 - Jonathan Morton