Subject: Re: [tcpm] More motivating scenarios for tcpm-ack-pull

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 4 Dec 2019 15:18:38 +0100

Carles,

I just found, that as a performance improvement measure, linux has
recently added a patch to trigger immediate ACKs when a packet with CWR
is received...

While the underlying issue was found in the dctcp context, the patch
appears to be against the generic rfc3168 ECN handling code.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fd2123a3d7527d4c7092633d55e877c0cc1d84a3

While that performance improvement has obvious benefits, RFC3168 is
silent on this particular issue.

But there are use cases where it is certainly valuable for the sender to
be able to elicit an immediate ACK from the receiver.

I don't know if it warrants a dedicated bit though - or if, for example,
the combination of CWR/PSH could perhaps obtain the ACK PULL semantics
as an overload of of their normal semantics...

Best regards,
    Richard

Am 18.11.2019 um 08:21 schrieb Jon Crowcroft:
> Mark Handley said a lot of the middlebox corner case avoidance stuff is
> in the tcpmp RFC...have got time to look just yet...
>
> On Mon, 18 Nov 2019, 04:46 Carles Gomez Montenegro,
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi Bob,
>
>     Once again, thanks a lot for all the very useful feedback!
>
>     Please find below some inline responses.
>
>      > Carles, Jon, (re-sending, this time without accidentally omitting
>     tcpm,
>      > also see couple of addenda tagged [Bob adds:])
>      >
>      > I'm generally supportive of a mechanism to suppress delayed ACKs from
>      > the receiver (in any transport protocol including TCP). So thank you.
>      >
>      > You might want to borrow at least some of the text for additional
>      > motivating scenarios for such a facility from the first two posts
>     on a
>      > thread for a similar facility in QuiC:
>      > https://github.com/quicwg/base-drafts/issues/1978
>      >
>      > Also pasted at the end.
>      >
>      > Some is not relevant to your specific ack-pull scheme, 'cos it's
>      > proposing a more general facility in QUIC for the sender to the
>     receiver
>      > alters its ACK ratio. (theoretically there are more bits available in
>      > QUIC, but this particular request has been put on hold while someone
>      > goes and finds them ;)
>
>     Thanks a lot. This is indeed very useful!
>
>     As you explain, the more general facility for QUIC allows to suppress
>     Delayed ACKs and also allows to modify the ACK ratio in a more general
>     way.
>
>     Regarding the Delayed ACK suppression discussion, the additional
>     motivation is definitely important.
>
>      >       Another motivating example (rather niche)
>      >
>      > Coincidentally, Just this morning I published a tech report that
>     had to
>      > use the hack of overlapping a byte from the previous segment to
>     force a
>      > quick ack:
>      > TCP Prague Fall-back on Detection of a Classic ECN AQM
>      > <https://arxiv.org/pdf/1911.00710.pdf#page9> (the link takes you
>     to Fig
>      > 2, which illustrates the hack)
>      > I'd rather not have to do these sorts of hacks - it really messes
>     with
>      > the segmentation code.
>
>     This is very useful feedback as well, and a nice coincidence!
>
>     Regarding the hack, sending an old byte may be even worse when there is
>     not a next data segment readily available for sending, which would
>     lead to
>     sending a whole packet just to carry an old byte.
>
>      >       Middlebox traversal of bit 6
>      >
>      > I recall that someone did a study of traversal of each of the
>     reserved
>      > flags (Marcelo Bagnulo maybe?). I don't think it was that good [Bob
>      > adds: I mean traversal - not the study!].
>      > A good search engine might find it ;)
>
>     I wasn't able to find the study, but this is an important
>     consideration..
>
>     One might think that if new functionality is defined and standardized,
>     related middlebox traversal of it should improve over time. Perhaps that
>     might be optimistic...
>
>      >       More alternative approaches:
>      >
>      > 1/ At one stage, we tried to include a bit for Delayed Ack control in
>      > another protocol called Accurate ECN. We were persuaded to take
>     it out,
>      > because it was mission creep (outside the scope of the protocol
>     we were
>      > working on):
>      >
>     https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#appendix-B.4
>
>     I was unaware of this, thanks!
>
>     This is actually very aligned with what we are proposing...
>
>      > 2/ Have you considered using the Urgent Pointer in a novel way?
>     I.e. a
>      > non-zero Urgent Pointer field, even tho the URG flag is zero.
>      > [Bob adds:] From memory, traversal was pretty good - much better than
>      > bits 4-6. But I think Windows treated it as a potential attack. I
>      > believe Fernando Gont did a study on how 'invalid' TCP header
>     fields are
>      > handled.
>      >
>      > You could assign, say, 3 bits of the urgent pointer for a variable
>      > ack_exp that is log base 2 of the ack ratio the sender would
>     like. Then
>      > the sender can request the receiver uses ack_ratio` = 2^ack_exp,
>     as in
>      > the QUIC proposal below,
>      > With ack_exp = 0, you get ack_ratio = 2^0 = 1, which has the same
>     effect
>      > as ack-pull. But you also have a more general facility for high speed
>      > machines to widen out the ack ratio.
>      >
>      > If this works, you could open up a registry for the other 13 bits. So
>      > instead of using up a precious bit, you generate 13 more ;)
>
>     Nice! :)
>
>     Actually, we had not considered such an idea.
>
>     In your proposal above, if a segment has the URG flag set to 1,
>     would then
>     the 3 ack_exp bits still be used as ack_exp bits, thus reducing the size
>     of the urgent pointer in practice to 13 bits?
>
>      > See:
>      > https://tools.ietf.org/agenda/90/slides/slides-90-tcpm-10.pdf#page=5
>      >
>      > I think there were problems with some implementations and middleboxes
>      > (bound to be). I've checked the minutes of the IETF meeting where
>     that
>      > slide was presented
>      > <https://www.ietf.org/proceedings/90/minutes/minutes-90-tcpm>, but
>      > there's no push-back in there. I suggest you delve back into the
>      > archives of the tcpm mailing list around the time of that meeting to
>      > find out if there were any show-stoppers.
>
>     Thanks for the pointers!
>
>     Sure, we'll check the tcpm mailing list archive.
>
>     Cheers,
>
>     Carles
>
>
>      > Cheers
>      >
>      >
>      >
>      > Bob
>      >
>      >
>      > Paste from QUIC thread:
>      >
>     /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
>      >
>      >
>      >
>      >       Problem/goals:
>      >
>      >  1. Getting up to speed fast without inducing much queue is one
>     of the
>      >     main areas where latency reductions are needed (for short and
>     long
>      >     flows). The sender needs frequent ACKing during this phase in
>     many
>      >     approaches (hystart, etc). A Linux TCP receiver starts with
>      >     ack_ratio 1 and uses heuristics to identify the end of the
>     sender's
>      >     slow-start (or a re-start after idle). This has ossified a
>     certain
>      >     slow-start behaviour into all TCP senders (whether Linux or
>     not). We
>      >     are trying to new sender behaviours (paced chirping being an
>      >     example), but we need a way for the sender to suppress
>     delayed ACKs.
>      >  2. At the other end of the scale, for long-running flows, many
>     current
>      >     CC approaches (e.g. BBR) use pacing not ACK-clocking so they
>     can use
>      >     very few ACKs per RTT. And fewer means less cache ejections
>     for GRO.
>      >     But the receiver doesn't know what CC the sender is using, so it
>      >     doesn't know how many or few is too many or too few.
>      >  3. Some middleboxes and the majority of link technologies (e.g.
>     DOCSIS,
>      >     LTE, Satellite) thin TCP ACKs when they detect the upstream is
>      >     filled with TCP ACK stream(s), which are unresponsive.
>     Otherwise the
>      >     ACKs constrain downstream throughput (and any upstream data
>     either
>      >     in the flow itself, or in others). We don't want these links to
>      >     attempt to guess which are the QUIC ACKs and try to thin
>     them. QUIC
>      >     can and should do this itself. QUIC has all the machinery for the
>      >     sender CC to detect ACK congestion, but not the protocol to
>     tell the
>      >     receiver to do the thinning. This protocol addition would
>     provide a
>      >     sufficient hook for hosts to unilaterally add this to their CC
>      >     behaviour.
>      >
>      >
>      >       Some scenarios where the sender's preferred ack_ratio changes
>      >       through the connection:
>      >
>      >  1.
>      >
>      >     A sender CC that wants the receiver to turn off DelAcks during
>      >     flow-start (e.g. it's using hybrid slow-start as in Cubic and
>     wants
>      >     to get delay measurements more frequently) sets ack_exp=0 during
>      >     flow-start (ack_ratio=1), then increases ack_exp during
>     congestion
>      >     avoidance. If it goes idle, then re-starts, it would set
>     ack_exp=0
>      >     again.
>      >
>      >     Note on heuristics: A Linux TCP receiver currently uses a
>     heuristic
>      >     to determine when the sender has exited slow-start. However,
>      >     heuristics -> ossification. A Linux receiver's heuristic only
>     works
>      >     with the current pattern of slow-start. In TCP, when we tried to
>      >     improve the pattern on the sender (paced chirping), the
>     heuristic on
>      >     the receiver killed us.
>      >
>      >  2.
>      >
>      >     Imagine a paced sender has hardware generic receive offload
>     (GRO),
>      >     so for a long-running flow it doesn't want a high rate of
>     QUIC ACKs
>      >     that are opaque to GRO. Let's say it would prefer at least 8 ACKs
>      >     per RTT. Again, it starts with ack_exp=0, but in congestion
>      >     avoidance it would use:
>      >
>      >     ack_ratio <= cwnd_in_packets/8
>      >
>      > Upshot: By remote controlling the receiver, the server offloads
>     nearly
>      > all the ACKs from large downloads, but still focuses its
>     ACK-receiving
>      > resources on getting each client up to speed.
>      >
>      > Note that calculation of ack_ratio lends itself to fast integer
>      > arithmetic.
>      >
>      >
>      >
>      >
>      >
>      >
>      >
>      > Bob
>      >
>      >
>      > --
>      > ________________________________________________________________
>      > Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
>      >
>      >
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Sat, 7 Dec 2019 12:30:39 +0100

Hi Richard,

Thanks a lot for your message!

It is very interesting to witness how the patch you mention offers a way
to trigger immediate ACKs, and the related context and benefits.

Perhaps a combination or (re)use of existing TCP header fields (e.g.
CWR/PSH as per your suggestion below, or the Urgent pointer as suggested
earlier by Bob) could be a way to enable triggering immediate ACKs.

A concern with these techniques is that they are limited to some extent.
For example, some applications produce small segments with the PSH flag
set and benefit from Delayed ACKs (e.g. to piggyback ACKs and replies). In
those cases, even regardless of the CWR flag, triggering immediate ACKs
will avoid achieving performance benefits of Delayed ACKs.

In contrast, a dedicated resource (e.g. a TCP header reserved bit, as we
propose in the draft) will not suffer such limitations. However, that
resource needs to be allocated.

Summarizing, there may be two approaches to enable ACK Pull:

1.- Overloading semantics of existing TCP header fields.
2.- Dedicating a resource.

Then the question is, considering pros and cons of both approaches: which
one do we prefer?

It would be great to hear the WG feedback on the above question!

Thanks,

Carles

---

From: Jeremy Harris <[email-masked]>
Date: Sat, 7 Dec 2019 13:01:39 +0000

> 2.- Dedicating a resource.

Dedicating a resource, specifically a new TCP option type,
if enabling this feature produces performance benefits,
would have the useful effect of inducing the removal of
middleboxes that filter unknown options.
-- 
Cheers,
  Jeremy

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 07 Dec 2019 13:08:42 +0000

> middleboxes that filter unknown options.
Personally, I suspect, that's not going to be a likely outcome in any 
reasonable time frame - and we should focus on understanding whether 
paths that do permit this will gain useful benefit.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 7 Dec 2019 16:03:53 +0200

> On 7 Dec, 2019, at 1:30 pm, Carles Gomez Montenegro <[email-masked]> wrote:
> 

I note that the semantics of the AKP flag differ subtly from that of PSH, in that the former says "I want an ack immediately", while the latter says "I have temporarily run out of data to send, so the receiving application's buffer should be flushed now".  It would be just as easy to modify existing receivers to treat PSH as an immediate ack request as to implement handling of AKP, so this simple use doesn't seem very compelling for allocating a scarce bit.

One potentially interesting application that was suggested to me, and which could legitimately motivate the use of a new reserved flag bit, was to provide a hint as to the correct Delayed Ack ratio, for use as a congestion control mechanism for acks.  Presently an ack stream can be anti-responsive, in that dropping acks from a heavily congested reverse path can actually increase the ack arrival rate, by improving the throughput of an uncongested forward path that was limited by ack-clocking.  AckCC exists as an RFC to address this problem - and would tend to benefit from Generalised ECN - but its reliance on a TCP option is less than optimal, given the traversal problems often cited.

A way that such a hint could work is by spacing AKP flags at the desired ack ratio.  A receiver observing AKP flags could then infer that the usual Delayed Ack rule requiring an ack for every two MSS received is suspended, and thus only send acks when AKP is set, the Delayed Ack timer expires, or some other forcing condition occurs.

It might be wise to restore normal Delayed Ack rules if the AKP flags stop indicating a reasonable ratio, for example if the Delayed Ack timer expires twice without an AKP flag arriving.  A maximum ratio could be specified as a sanity check, eg. 1:64, at which the receiver generates an ack even if an AKP hasn't arrived yet.

The above hinting semantics would remain compatible with other use cases mentioned, as AKP still forces an ack.  However this is also a theoretical disadvantage because the segments marked with AKP generally won't coincide with the trailing end of bursts caused, for example, by link-layer aggregation or IRQ "optimisation" in NICs.  Those trailing segments are where you would optimally want to trigger an ack, but as currently specified, AKP removes the receiver's discretion to synchronise a sparse Delayed Ack ratio with those segments.

This could be addressed by pairing AKP with PSH.  Both flags set together would require an immediate ack of that particular segment, with the implication both that an ack is expected promptly and that no more data is immediately forthcoming.  PSH by itself would continue to permit a degree of latitude with respect to ack timing, and merely indicates that prompt delivery to the application is required.  AKP by itself then requests a prompt ack, but one that may reflect following segments that are already in receive buffers and will be processed immediately after this one, as well as the hinting described above.

I think that's a reasonable amount of new functionality to introduce for the cost of one bit.  The above suggestion addresses both the original problem and another long-standing one, with minimal newly allocated resources and relatively little risk of traversal problems.  And (importantly) TCP doesn't break if receivers ignore or fail to understand it.

 - Jonathan Morton

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 10 Dec 2019 14:37:04 +0100

I believe your list here misses the AckCC option of RFC5690 (which is
only informational, not even experimental). Am Ack Ratio option of 1
followed by a an Ack Ratio of n could serve the same purpose...

Note that there have been multiple investigations showing good
traversability of new options across middleboxes nowadays.

Best regards,
    Richard

Am 07.12.2019 um 12:30 schrieb Carles Gomez Montenegro:
> ess how the patch you mention offers a way

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Dec 2019 18:34:59 +0000

Carles,

It strikes me that there is a lot of interest in doing something like 
ACK-pull, but also a lot of disagreement in where to draw the line 
between what to do and what not to do.

Can I suggest that a way to resolve this would be to start with an 
informational requirements draft, that I am sure would be adopted as WG 
business immediately given the interest. Whereas any particular choice 
of mechanism at this stage would probably not get the necessary 
consensus to be adopted.

The requirements draft could discuss solutions, not as a "on this we all 
agree" protocol, but in terms of their pros and cons, e.g. which 
mechanisms could satisfy which requirements, so they can be judged 
relative to each mechanism's complexity / traversibility / usage of bits 
/ etc.

I will warn you that a requirements draft could itself add years to the 
process (the accurate ECN requirements did [RFC7560]). But not if we 
agree to allow a solution draft to start in parallel once the 
requirements are starting to converge, rather than waiting for the 
requirements to go thru the full RFC process.

You might prefer to continue evangelising ack-pull as a sufficient 
solution, so I don't want to say a requirements draft is the only way 
forward. But you saw my comment earlier - that once you've been through 
all the years of effort of getting anything changed in TCP, you 
sometimes feel that you wish you'd been more ambitious than just one bit 
at the start.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Dec 2019 17:28:58 +0000

Carles, Jon,

Another motivating scenario for ack-pull (or similar) brought to my 
attention by Richard Scheffenegger and Neal Cardwell:

Re: [PATCH net-next] tcp: ack immediately when a cwr packet arrives
https://lists.openwall.net/netdev/2018/07/25/10

Bob