Subject: Re: [tcpm] TLP questions

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 9 May 2018 18:29:49 +0000

Including tcpm as this can be a review of the TLP portions of draft-ietf-tcpm-rack-03.

From: Praveen Balasubramanian
Sent: Tuesday, May 8, 2018 6:13 PM
To: Yuchung Cheng <[email-masked]>; Neal Cardwell <[email-masked]>
Subject: TLP questions

Hey folks, I have some questions on draft-ietf-tcpm-rack-03 mainly around TLP.

  1.  "Open state: the sender's loss recovery state machine is in its  normal, default state: there are no SACKed sequence ranges in the  SACK scoreboard, and neither fast recovery, timeout-based recovery, nor ECN-based cwnd reduction are underway. "

Open state is defined and then never used. I assume you require that TLP be scheduled only for connections in open state? I wonder why there is the requirement on "no SACKed sequence ranges in the  SACK scoreboard".

  1.  PTO += 2ms

If we already are being conservative by waiting 2*SRTT, then how is adding 2 msec going to help? Was this added due to a real world issue?

  1.  "If a previously unsent segment exists AND
         the receive window allows new data to be sent:
           Transmit that new segment
           FlightSize += SMSS
       Else:
           Retransmit the last segment"
This needs to be crisper about what is meant by "previously unsent" and "last segment". For example the sender could have sent a large amount of data and then taken a full RTO. In this case if PTO fires, do "previously unsent" and "last segment" refer to MSS size segments straddling just before and  after SND.NXT? OR do they straddle around the largest sent sequence number ever in the connection lifetime?

  1.  "On each incoming ACK, the sender should cancel any existing loss  probe timer."

Even on duplicate ACKs?

Thanks

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 09 May 2018 20:27:06 +0000

On Wed, May 9, 2018 at 2:29 PM Praveen Balasubramanian <[email-masked]>
wrote:

> *From:* Praveen Balasubramanian
> *Sent:* Tuesday, May 8, 2018 6:13 PM
> *To:* Yuchung Cheng <[email-masked]>; Neal Cardwell <
> [email-masked]>
> *Subject:* TLP questions
>
>
>

Praveen, thank you for all of these excellent questions, and thanks for
bringing this to the tcpm list for discussion!

>
>
>    1. “Open state: the sender's loss recovery state machine is in its
>     normal, default state: there are no SACKed sequence ranges in the  SACK
>    scoreboard, and neither fast recovery, timeout-based recovery, nor
>    ECN-based cwnd reduction are underway. “
>
> Open state is defined and then never used.
>
Yes, thanks for noticing. The phrase "Open state" is used in earlier
revisions of the TLP and RACK drafts as a precondition for sending a Tail
Loss Probe. We removed its use in draft-ietf-tcpm-rack-03 but forgot to
remove the definition in draft-ietf-tcpm-rack-03. We noticed shortly after
shipping the -03 text and already removed that definition in our internal
draft for draft-ietf-tcpm-rack-04.

> I assume you require that TLP be scheduled only for connections in open
> state?
>
That was the original (and long-time) implementation. The "Open" state is
the Linux TCP stack term for the default congestion control state where
there's nothing in the SACK scoreboard, no ECN cwnd reduction is underway,
and we're not doing Fast Recovery or RTO recovery. Originally the Linux TCP
code would only schedule a TLP in the "Open" state. But recently the TCP
team at Google has analyzed scenarios where this limitation seemed too
strict. In particular, we were analyzing a case in the public Linux netdev
mailling list ("Re: Linux ECN Handling") where the connection was in the
middle of a DCTCP ECN-triggered cwnd reduction, and then suffered a ~300ms
timeout that would have been vastly shorter if there had been a TLP for
recovery. So we changed the Linux TCP code to allow TLP in ECN-triggered
cwnd reductions (b4f70c3d4ec32 "tcp: allow TLP in ECN CWR").

This means that the Linux TCP code is allowing TLP in the ECN cwnd
reduction state as well as the "Open" state.  We have tried to update the
draft in draft-ietf-tcpm-rack-03 and, based on your feedback, will try to
make this more crisp/clear in draft-ietf-tcpm-rack-04. Please see below for
a proposed update to this part of the text.

> I wonder why there is the requirement on “no SACKed sequence ranges in the
>  SACK scoreboard”.
>
This condition dated from the original TLP design, where originally the
motivation was to keep the behavior cautious and simple. Now that we have
RACK, as a practical matter, we still don't really want a TLP in such
cases. Rather, if there is some SACKed sequence range but we are not yet in
loss recovery, then typically RACK will install a timer based on the
reordering window, and when that timer fires it will mark some packets lost
and enter fast recovery.

For this reason, the Linux TCP implementation of TLP+RACK still has a "no
SACKed sequence ranges in the SACK scoreboard" as a precondition for
scheduling a TLP, and IMHO this still makes sense, so I would proposed that
in draft-ietf-tcpm-rack-04 we document that by adding a clause to the list
of preconditions for scheduling a TLP ("5.4.1. Phase 1: Scheduling a loss
probe... A sender should schedule a PTO only if all of the following
conditions are met"):

     The connection has no SACKed sequences in the SACK scoreboard

Please see below for where this proposed addition fits into the surrounding
text.

>
>    1. PTO += 2ms
>
> If we already are being conservative by waiting 2*SRTT, then how is adding
> 2 msec going to help? Was this added due to a real world issue?
>
Yes, this is due to real-world issues. In datacenters, with Linux TCP the
SRTT could be much lower than 100usec, so even 2*SRTT is a very tight
timer, and could result in lots of spurious TLPs, and is not even supported
by current retransmission timer mechanisms in common OSes. So the 2ms is to
allow for real-world jitter in the network and end hosts. We tried to
describe some of these issues in the -03 draft:

   Similarly, current end-system processing latencies and timer
   granularities can easily delay ACKs, so senders SHOULD add at least
   2ms to a computed PTO value (and MAY add more if the sending host OS
   timer granularity is more coarse than 1ms).

I guess we can add network jitter to this as well, so I'd propose something
like the following for -04:

* Similarly, network delay variations and end-system processing*
* latencies and timer granularities can easily delay ACKs beyond 2*SRTT,*
* so senders SHOULD add at least 2ms to a computed PTO value*
* (and MAY add more if the sending host OS timer granularity is more*

* coarse than 1ms).*
Instead of 2ms we could add something like RTTVAR from RFC 6298, but that
would add complexity and delay, especially if the receiver is manifesting
~200ms delayed ACKs. And so we have not yet seen evidence that that would
be a good trade-off.

>    1. “If a previously unsent segment exists AND
>
>            Retransmit the last segment”
>
> This needs to be crisper about what is meant by “previously unsent” and
> “last segment”. For example the sender could have sent a large amount of
> data and then taken a full RTO. In this case if PTO fires, do “previously
> unsent” and “last segment” refer to MSS size segments straddling just
> before and  after SND.NXT? OR do they straddle around the largest sent
> sequence number ever in the connection lifetime?
>

Very good questions. I like your suggestion to be crisper in this section.
The Linux TCP stack does not "rewind" SND.NXT upon RTO, so in the Linux TCP
stack the SND.NXT point and "largest sent sequence number ever in the
connection lifetime" are basically the same point. That is the framework in
which we were thinking for those lines.

In my mind...

By "a previously unsent segment" we mean basically "the next segment (of
MSS or fewer bytes) that the sender would normally send if it had available
cwnd at this time." That is something that presumably every
production-quality TCP stack has very quick access to.

By "the last segment" we mean "the highest-sequence segment (of MSS or
fewer bytes) that has already been transmitted and not ACKed or SACKed." I
imagine this should also be generally very quick to access (at least it can
be quickly accessed in two generations of the Linux TCP write queue). Let
us know if not, and we can discuss.

Does that help clarify those parts? If so, we can update the text to
incorporate something like that (suggestions?).

>
>    1. “On each incoming ACK, the sender should cancel any existing loss
>    probe timer.”
>
> Even on duplicate ACKs?
>
Excellent question. This line is out of date, and does not reflect recent
fixes our team made in the Linux TCP stack (df92c8394e6e "tcp: fix xmit
timer to only be reset if data ACKed/SACKed").  In fact the TLP/RTO timers
should only be rearmed (cancelled and reinstalled at a later time) if
certain kinds of real "forward progress" are made. On our internal draft
for -04 I have proposed we change this to:

  Phase 3: ACK processing

  On each incoming ACK, the sender should check the conditions in Step 1 of
Phase 1 to see if it should schedule (or reschedule) the loss probe timer.

And then, putting this all together, I would propose something like the
following text for Step 1 of Phase 1:

----

*Phase 1: Scheduling a loss probeStep 1: Check conditions for scheduling a
PTO.A sender should check to see if it should schedule a PTO in the
following situations: 1. After transmitting new data2. Upon receiving an
ACK that cumulatively acknowledges data3. Upon receiving a SACK that
selectively acknowledges data that was last sent before the segment with
SEG.SEQ=SND.UNA was last (re)transmittedA sender should schedule a PTO only
if all of the following conditions are met: 1. The connection supports SACK
[RFC2018]2. The connection has no SACKed sequences in the SACK scoreboard3.
The connection is not in loss recovery4. The most recently transmitted data
was not itself a TLP probe (i.e. a sender MUST NOT send consecutive or
back-to-back TLP probes).If a PTO can be scheduled according to these
conditions, the sender should schedule a PTO. If there was a previously
scheduled PTO or RTO pending, then that pending PTO or RTO should first be
cancelled, and then the new PTO should be scheduled.If a PTO cannot be
scheduled according to these conditions, then the sender MUST arm the RTO
timer if there is unacknowledged data in flight.*
----

How does that sound?

Praveen, thank you again for all these excellent questions and points!

cheers,
neal

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 10 May 2018 16:08:00 -0700

Hi Neal,

On Wed, May 9, 2018 at 1:27 PM, Neal Cardwell <[email-masked]> wrote:
>> “Open state: the sender's loss recovery state machine is in its  normal,
>> default state: there are no SACKed sequence ranges in the  SACK scoreboard,
>> and neither fast recovery, timeout-based recovery, nor ECN-based cwnd
>> reduction are underway. “
>>
> This means that the Linux TCP code is allowing TLP in the ECN cwnd reduction
> state as well as the "Open" state.  We have tried to update the draft in
> draft-ietf-tcpm-rack-03 and, based on your feedback, will try to make this
> more crisp/clear in draft-ietf-tcpm-rack-04. Please see below for a proposed
> update to this part of the text.
>>
> RACK, as a practical matter, we still don't really want a TLP in such cases.
> Rather, if there is some SACKed sequence range but we are not yet in loss
> recovery, then typically RACK will install a timer based on the reordering
> window, and when that timer fires it will mark some packets lost and enter
> fast recovery.
>
>> PTO += 2ms
> SRTT could be much lower than 100usec, so even 2*SRTT is a very tight timer,
> and could result in lots of spurious TLPs, and is not even supported by
> current retransmission timer mechanisms in common OSes. So the 2ms is to
> Similarly, network delay variations and end-system processing
> latencies and timer granularities can easily delay ACKs beyond 2*SRTT,
> so senders SHOULD add at least 2ms to a computed PTO value
> (and MAY add more if the sending host OS timer granularity is more
> coarse than 1ms).
>
> ~200ms delayed ACKs. And so we have not yet seen evidence that that would be
> a good trade-off.
>
>> “If a previously unsent segment exists AND
>>
>> unsent” and “last segment” refer to MSS size segments straddling just before
>> and  after SND.NXT? OR do they straddle around the largest sent sequence
>> number ever in the connection lifetime?
>
>
> By "a previously unsent segment" we mean basically "the next segment (of MSS
> or fewer bytes) that the sender would normally send if it had available cwnd
> at this time." That is something that presumably every production-quality
> TCP stack has very quick access to.
>
> By "the last segment" we mean "the highest-sequence segment (of MSS or fewer
> bytes) that has already been transmitted and not ACKed or SACKed." I imagine
> this should also be generally very quick to access (at least it can be
> quickly accessed in two generations of the Linux TCP write queue). Let us
> know if not, and we can discuss.
>

I am thinking that we can just say "send one (SMSS or smaller) segment
that contains up to highest sequence number it has sent."
I am a bit wondering if not ACKed or SACKed requirement should be
mandatory, although it will be a bit redundant.

>>
>> “On each incoming ACK, the sender should cancel any existing loss  probe
>> timer.”
>>
>> Even on duplicate ACKs?
>
> certain kinds of real "forward progress" are made. On our internal draft for
> -04 I have proposed we change this to:
>
>   Phase 3: ACK processing
>
> Phase 1: Scheduling a loss probe
>
>
> Step 1: Check conditions for scheduling a PTO.
>
> A sender should check to see if it should schedule a PTO in the following
> situations:
>
> After transmitting new data
>
> Upon receiving an ACK that cumulatively acknowledges data
>
> Upon receiving a SACK that selectively acknowledges data that was last sent
> before the segment with SEG.SEQ=SND.UNA was last (re)transmitted
>
>
> A sender should schedule a PTO only if all of the following conditions are
> met:
>
> The connection supports SACK [RFC2018]

Do we need to check this? I mean RACK already presumes SACK nodes. I
thought TLP does as well.
Or, do we want to use TLP with non-SACK nodes?

> The connection is not in loss recovery
>
> The most recently transmitted data was not itself a TLP probe (i.e. a sender
> MUST NOT send consecutive or back-to-back TLP probes).

I think the third condition might be a bit ambiguous and may require
another variable. If we check TLPRtxOut and TLPHighRxt, might it be
sufficient?
--
Yoshi

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 10 May 2018 23:29:01 +0000

Thanks Neal for the detailed response along with the historical context. Looking forward to draft 04 updates.

A few more questions and comments.

> then typically RACK will install a timer based on the reordering window, and when that timer fires it will mark some packets lost and enter fast recovery
The "reordering settling" timer is defined as optional in the draft. The Windows implementation currently does not use this timer. Until we add such a timer, we plan to not prevent TLP even if SACK scoreboard is not empty. However if recovery is triggered, we’ll cancel the PTO and arm an RTO. Do you see any issues with this approach? Since the draft makes the "reordering settling" timer optional, I think it should suggest this alternative approach.

> So the 2ms is to allow for real-world jitter in the network and end hosts
Currently the Windows implementation doesn’t use TLP and RACK for connections with < 10 msec RTT. So until we change this logic, we will skip adding the 2 ms jitter protection.

> By "a previously unsent segment" we mean basically "the next segment (of MSS or fewer bytes) that the sender would normally send if it had available cwnd at this time
This is better but still not crisp enough to determine what exactly the Linux implementation does. Since you said that Linux does not roll back SND.NXT upon RTO, does this imply that "a previously unsent segment", is the one starting at SND.NXT?

> By "the last segment" we mean "the highest-sequence segment (of MSS or fewer bytes) that has already been transmitted and not ACKed or SACKed
Again given that SND.NXT is not rolled back this would presumably the first UN(S)ACKed MSS or fewer bytes walking back from SND.NXT (I am not suggesting an actual walk back, just for illustration). I am a bit confused that you used the “or SACKed” since you previously required that PTO not be armed if any SACK blocks were present. If that is followed, then "the last segment" would pretty much always be MSS or fewer bytes just before (non-rolled back) SND.NXT.

Thanks

From: Neal Cardwell [mailto:[email-masked]]
Sent: Wednesday, May 9, 2018 1:27 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; [email-masked]; Nandita Dukkipati <[email-masked]>; Priyaranjan Jha <[email-masked]>
Subject: Re: TLP questions

On Wed, May 9, 2018 at 2:29 PM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Including tcpm as this can be a review of the TLP portions of draft-ietf-tcpm-rack-03.

From: Praveen Balasubramanian
Sent: Tuesday, May 8, 2018 6:13 PM
To: Yuchung Cheng <[email-masked]<mailto:[email-masked]>>; Neal Cardwell <[email-masked]<mailto:[email-masked]>>
Subject: TLP questions

Praveen, thank you for all of these excellent questions, and thanks for bringing this to the tcpm list for discussion!

  1.  “Open state: the sender's loss recovery state machine is in its  normal, default state: there are no SACKed sequence ranges in the  SACK scoreboard, and neither fast recovery, timeout-based recovery, nor ECN-based cwnd reduction are underway. “

Open state is defined and then never used.
Yes, thanks for noticing. The phrase "Open state" is used in earlier revisions of the TLP and RACK drafts as a precondition for sending a Tail Loss Probe. We removed its use in draft-ietf-tcpm-rack-03 but forgot to remove the definition in draft-ietf-tcpm-rack-03. We noticed shortly after shipping the -03 text and already removed that definition in our internal draft for draft-ietf-tcpm-rack-04.

I assume you require that TLP be scheduled only for connections in open state?
That was the original (and long-time) implementation. The "Open" state is the Linux TCP stack term for the default congestion control state where there's nothing in the SACK scoreboard, no ECN cwnd reduction is underway, and we're not doing Fast Recovery or RTO recovery. Originally the Linux TCP code would only schedule a TLP in the "Open" state. But recently the TCP team at Google has analyzed scenarios where this limitation seemed too strict. In particular, we were analyzing a case in the public Linux netdev mailling list ("Re: Linux ECN Handling") where the connection was in the middle of a DCTCP ECN-triggered cwnd reduction, and then suffered a ~300ms timeout that would have been vastly shorter if there had been a TLP for recovery. So we changed the Linux TCP code to allow TLP in ECN-triggered cwnd reductions (b4f70c3d4ec32 "tcp: allow TLP in ECN CWR").

This means that the Linux TCP code is allowing TLP in the ECN cwnd reduction state as well as the "Open" state.  We have tried to update the draft in draft-ietf-tcpm-rack-03 and, based on your feedback, will try to make this more crisp/clear in draft-ietf-tcpm-rack-04. Please see below for a proposed update to this part of the text.

I wonder why there is the requirement on “no SACKed sequence ranges in the  SACK scoreboard”.
This condition dated from the original TLP design, where originally the motivation was to keep the behavior cautious and simple. Now that we have RACK, as a practical matter, we still don't really want a TLP in such cases. Rather, if there is some SACKed sequence range but we are not yet in loss recovery, then typically RACK will install a timer based on the reordering window, and when that timer fires it will mark some packets lost and enter fast recovery.

For this reason, the Linux TCP implementation of TLP+RACK still has a "no SACKed sequence ranges in the SACK scoreboard" as a precondition for scheduling a TLP, and IMHO this still makes sense, so I would proposed that in draft-ietf-tcpm-rack-04 we document that by adding a clause to the list of preconditions for scheduling a TLP ("5.4.1. Phase 1: Scheduling a loss probe... A sender should schedule a PTO only if all of the following conditions are met"):

Please see below for where this proposed addition fits into the surrounding text.

  1.  PTO += 2ms

If we already are being conservative by waiting 2*SRTT, then how is adding 2 msec going to help? Was this added due to a real world issue?
Yes, this is due to real-world issues. In datacenters, with Linux TCP the SRTT could be much lower than 100usec, so even 2*SRTT is a very tight timer, and could result in lots of spurious TLPs, and is not even supported by current retransmission timer mechanisms in common OSes. So the 2ms is to allow for real-world jitter in the network and end hosts. We tried to describe some of these issues in the -03 draft:

I guess we can add network jitter to this as well, so I'd propose something like the following for -04:

Instead of 2ms we could add something like RTTVAR from RFC 6298, but that would add complexity and delay, especially if the receiver is manifesting ~200ms delayed ACKs. And so we have not yet seen evidence that that would be a good trade-off.

This needs to be crisper about what is meant by “previously unsent” and “last segment”. For example the sender could have sent a large amount of data and then taken a full RTO. In this case if PTO fires, do “previously unsent” and “last segment” refer to MSS size segments straddling just before and  after SND.NXT? OR do they straddle around the largest sent sequence number ever in the connection lifetime?

Very good questions. I like your suggestion to be crisper in this section. The Linux TCP stack does not "rewind" SND.NXT upon RTO, so in the Linux TCP stack the SND.NXT point and "largest sent sequence number ever in the connection lifetime" are basically the same point. That is the framework in which we were thinking for those lines.

In my mind...

By "a previously unsent segment" we mean basically "the next segment (of MSS or fewer bytes) that the sender would normally send if it had available cwnd at this time." That is something that presumably every production-quality TCP stack has very quick access to.

By "the last segment" we mean "the highest-sequence segment (of MSS or fewer bytes) that has already been transmitted and not ACKed or SACKed." I imagine this should also be generally very quick to access (at least it can be quickly accessed in two generations of the Linux TCP write queue). Let us know if not, and we can discuss.

Does that help clarify those parts? If so, we can update the text to incorporate something like that (suggestions?).

  1.  “On each incoming ACK, the sender should cancel any existing loss  probe timer.”

Even on duplicate ACKs?
Excellent question. This line is out of date, and does not reflect recent fixes our team made in the Linux TCP stack (df92c8394e6e "tcp: fix xmit timer to only be reset if data ACKed/SACKed").  In fact the TLP/RTO timers should only be rearmed (cancelled and reinstalled at a later time) if certain kinds of real "forward progress" are made. On our internal draft for -04 I have proposed we change this to:

  Phase 3: ACK processing

  On each incoming ACK, the sender should check the conditions in Step 1 of Phase 1 to see if it should schedule (or reschedule) the loss probe timer.

And then, putting this all together, I would propose something like the following text for Step 1 of Phase 1:

----

A sender should check to see if it should schedule a PTO in the following situations:

  1.  After transmitting new data
  2.  Upon receiving an ACK that cumulatively acknowledges data
  3.  Upon receiving a SACK that selectively acknowledges data that was last sent before the segment with SEG.SEQ=SND.UNA was last (re)transmitted

A sender should schedule a PTO only if all of the following conditions are met:

  1.  The connection supports SACK [RFC2018]
  2.  The connection has no SACKed sequences in the SACK scoreboard
  3.  The connection is not in loss recovery
  4.  The most recently transmitted data was not itself a TLP probe (i.e. a sender MUST NOT send consecutive or back-to-back TLP probes).

If a PTO can be scheduled according to these conditions, the sender should schedule a PTO. If there was a previously scheduled PTO or RTO pending, then that pending PTO or RTO should first be cancelled, and then the new PTO should be scheduled.

If a PTO cannot be scheduled according to these conditions, then the sender MUST arm the RTO timer if there is unacknowledged data in flight.

----

How does that sound?

cheers,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 14 May 2018 17:21:36 -0400

> > On Wed, May 9, 2018 at 2:29 PM Praveen Balasubramanian <
[email-masked]>
...
> >> unsent” and “last segment” refer to MSS size segments straddling just
before
> > By "the last segment" we mean "the highest-sequence segment (of MSS or
fewer

> mandatory, although it will be a bit redundant.

Yes, good point; the "not ACKed or SACKed" I added in my email message was
redundant.

So we could replace:
   "Retransmit the last segment"
with:
   "Retransmit the highest-sequence segment sent so far"

> Or, do we want to use TLP with non-SACK nodes?

Yes, both RACK and TLP require SACK. I agree that it could be considered
redundant to specify the SACK dependency in the TLP section, given that we
have already  mentioned in section "3. Requirements" that RACK depends on
SACK. But IMHO it is worthwhile to keep this dependency separately listed
here for TLP, to try to make it clear/explicit that the TLP algorithm has
its own dependency on SACK.

> another variable. If we check TLPRtxOut and TLPHighRxt, might it be

Yes, I agree this condition is not as clear as it should be. There are
really several considerations:

(a) we don't want an unterminating chain of TLPs, where after sending a TLP
probe we rearm the PTO timer and send another TLP

(b) we can't allow multiple outstanding TLP retransmissions

(Checking TLPRxtOut would cover (b) but not (a). And checking TLPRxtOut is
discussed elsewhere in the draft.)

The way the Linux TCP TLP code handles this as follows:

For (a), we ensure this structurally in the call sequence: after sending a
TLP probe, we rearm the RTO timer, not the TLP timer. This is already
discussed in the section "Phase 2: Sending a loss probe".

For (b), before sending a TLP retransmission we effectively check TLPRxtOut
and do not send a TLP at all if there is an outstanding TLP retransmission.
This is already discussed in "Recording loss probe states".

Given that those issues are already discussed elsewhere, I'd propose to
remove that line "The most recently transmitted data was not itself a TLP
probe (i.e. a sender MUST NOT send consecutive or back-to-back TLP
probes).". Instead in the -04 draft I'd propose we just add a brief note at
the end of the "Phase 2: Sending a loss probe" section:

   Note that after transmitting a TLP, the sender MUST arm an RTO timer,
   and not the PTO timer. This ensures that the sender does not send
repeated,
   back-to-back TLP probes.

Thank you, Yoshifumi!

neal

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 14 May 2018 22:59:51 -0400

On Thu, May 10, 2018 at 7:29 PM Praveen Balasubramanian
<[email-masked]>

> A few more questions and comments.

Thanks again for another round of excellent questions and comments. :-)

> > then typically RACK will install a timer based on the reordering
> > window, and when that timer fires it will mark some packets lost
> > and enter fast recovery

> The "reordering settling" timer is defined as optional in the draft.
> The Windows implementation currently does not use this timer.
> Until we add such a timer, we plan to not prevent TLP even if SACK
> scoreboard is not empty. However if recovery is triggered, we'll
> cancel the PTO and arm an RTO. Do you see any issues with this
> approach? Since the draft makes the "reordering settling"
> timer optional, I think it should suggest this alternative approach.

That sounds to me like it would work OK. But IMHO it sounds like
it's missing out on the opportunity to use a reordering timer to
speed up fast recovery quite a bit by initiating a RACK-based
recovery using the reordering timer.

That scenario sounds like:

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP
+ wait 2*srtt
+ TLP timer fires, send TLP
+ 1*srtt passes
+ receive SACK of TLP, initiate RACK-based fast recovery

If there's a reordering timer, this could be:

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP
+ wait 0.25*min_rtt
+ reordering timer fires,  initiate RACK-based fast recovery

With the reordering timer, AFAICT it takes about 2.75*srtt less time before
the recovery begins, if I correctly noted all the details.

> Currently the Windows implementation doesn't use TLP and RACK for
> connections with < 10 msec RTT. So until we change this logic,
> we will skip adding the 2 ms jitter protection.

Skipping the 2ms jitter protection sounds OK in that case.

Are TLP and RACK skipped in the <10ms case due to the granularity of
the send and ACK timestamps, or the granularity of the timers, or both?
I could imagine that there could be some nice latency improvements from
using TLP and RACK at those low RTTs, as long as any timers that
were scheduled were rounded up appropriately (e.g. by 10ms-20ms).
But I haven't fully thought through all the implications of this kind
of scenario.

> > By "a previously unsent segment" we mean basically
> > "the next segment (of MSS or fewer bytes) that the sender
> > would normally send if it had available cwnd at this time

> This is better but still not crisp enough to determine what
> exactly the Linux implementation does. Since you said that
> Linux does not roll back SND.NXT upon RTO, does this imply
> that "a previously unsent segment", is the one starting at SND.NXT?

Yes, for the Linux TLP sender, "a previously unsent segment" is the one
starting at SND.NXT.

BTW, F-RTO RFC - https://tools.ietf.org/html/rfc5682 - in
section 2.1 2) b), in describing its similar "forward" probe packets,
uses the phrase "transmit up to two new (previously unsent) segments".
So for consistency with the F-RTO text I have tentatively
changed that line in our internal draft of -04 from:
  "Transmit that new segment"
to:
  "Transmit one new (previously unsent) segment"

It is probably worth noting here that in the Linux TCP stack the sender
would not send a TLP when in the middle of an RTO-triggered recovery.
This corresponds to one of the conditions in the draft, in section
"5.4.1. Phase 1: Scheduling a loss probe": the condition that says:
    " 2.  The connection is not in loss recovery"

So perhaps because of this the different SND.NXT "rewind" behavior of
different TCP stacks upon RTO may not matter ultimately? Or perhaps
there are more implications to unpack. :-)

> > By "the last segment" we mean "the highest-sequence segment
> > (of MSS or fewer bytes) that has already been transmitted
> > and not ACKed or SACKed

> Again given that SND.NXT is not rolled back this would presumably
> the first UN(S)ACKed MSS or fewer bytes walking back from SND.NXT
> (I am not suggesting an actual walk back, just for illustration).

Yes, exactly.

> I am a bit confused that you used the "or SACKed" since you
> previously required that PTO not be armed if any SACK blocks
> were present. If that is followed, then "the last segment" would
> pretty much always be MSS or fewer bytes just before
> (non-rolled back) SND.NXT.

Yes, you are exactly right. I should have left the "or SACKed" part out of
that line in my e-mail. :-)

By the way, regarding retransmitting the "last segment", this discussion
made me realize that AFAICT the draft did not yet discuss why the TLP
retransmission is the last segment. I have added some proposed text to our
internal draft of the -04 rev to try to describe the rationale:

"When the loss probe is a retransmission, the sender uses the
highest-sequence segment sent so far. This is in order to deal with the
retransmission ambiguity problem in TCP. Suppose a sender sends N segments,
and then retransmits the last segment (segment N) as a loss probe, and then
the sender receives a SACK for segment N. As long as the sender waits for
any required RACK reordering settling timer to then expire, it doesn't
matter if that SACK was for the original transmission of segment N or the
TLP retransmission; in either case the arrival of the SACK for segment N
provides evidence that the segments preceding segment N were likely lost."

Of course welcome comments/suggestions about this proposed paragraph as
well.

Thanks!

neal

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 18 May 2018 23:32:23 +0000

> Are TLP and RACK skipped in the <10ms case due to the granularity of the send and ACK timestamps, or the granularity of the timers, or both?
> I could imagine that there could be some nice latency improvements from using TLP and RACK at those low RTTs, as long as any timers that were scheduled were rounded up appropriately (e.g. by 10ms-20ms).
> But I haven't fully thought through all the implications of this kind of scenario.
Both but primarily due to timer granularity. I'd be interested to know if RACK/TLP in Linux is working well on mobile devices where I assume the timer isn’t as fine grained.

> It is probably worth noting here that in the Linux TCP stack the sender would not send a TLP when in the middle of an RTO-triggered recovery.
Windows stack exits recovery upon an RTO but keep track of the highest sent sequence number (SND.NXT value before the rewind). So it can do what Linux is doing post RTO. However I am not convinced if there is a safety issue with allowing TLP post an RTO. I can think of four cases that triggered the RTO: forward path loss, reverse path (ACK) loss, and a large RTT increase beyond RTO. It seems like if there is further loss for new data in slow start post RTO, then TLP may help in all cases except the RTT increase case. I get the part about implementation convenience to disallow TLP during all recovery, but do you see a safety issue?

-----Original Message-----
From: Neal Cardwell [mailto:[email-masked]] 
Sent: Monday, May 14, 2018 8:00 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Yuchung Cheng <[email-masked]>; [email-masked]; Nandita Dukkipati <[email-masked]>; Priyaranjan Jha <[email-masked]>
Subject: Re: TLP questions

> > window, and when that timer fires it will mark some packets lost and 
> > enter fast recovery

> timer optional, I think it should suggest this alternative approach.

That sounds to me like it would work OK. But IMHO it sounds like it's missing out on the opportunity to use a reordering timer to speed up fast recovery quite a bit by initiating a RACK-based recovery using the reordering timer.

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP wait 
+ 2*srtt TLP timer fires, send TLP 1*srtt passes receive SACK of TLP, 
+ initiate RACK-based fast recovery

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP wait 
+ 0.25*min_rtt reordering timer fires,  initiate RACK-based fast 
+ recovery

With the reordering timer, AFAICT it takes about 2.75*srtt less time before the recovery begins, if I correctly noted all the details.

> connections with < 10 msec RTT. So until we change this logic, we will 
> skip adding the 2 ms jitter protection.

Are TLP and RACK skipped in the <10ms case due to the granularity of the send and ACK timestamps, or the granularity of the timers, or both?
I could imagine that there could be some nice latency improvements from using TLP and RACK at those low RTTs, as long as any timers that were scheduled were rounded up appropriately (e.g. by 10ms-20ms).
But I haven't fully thought through all the implications of this kind of scenario.

> > By "a previously unsent segment" we mean basically "the next segment 
> > (of MSS or fewer bytes) that the sender would normally send if it 
> > had available cwnd at this time

> This is better but still not crisp enough to determine what exactly 
> the Linux implementation does. Since you said that Linux does not roll 
> back SND.NXT upon RTO, does this imply that "a previously unsent 
> segment", is the one starting at SND.NXT?

Yes, for the Linux TLP sender, "a previously unsent segment" is the one starting at SND.NXT.

BTW, F-RTO RFC - https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5682&data=02%7C01%7Cpravb%40microsoft.com%7C88aec2962f4c4782e54208d5ba0ff9ca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636619500137896021&sdata=mfd2Dun3d2dcamWx5xmLCsuZzEWiDNlqTJgXrc7LQQ8%3D&reserved=0 - in section 2.1 2) b), in describing its similar "forward" probe packets, uses the phrase "transmit up to two new (previously unsent) segments".
So for consistency with the F-RTO text I have tentatively changed that line in our internal draft of -04 from:
  "Transmit that new segment"
to:
  "Transmit one new (previously unsent) segment"

It is probably worth noting here that in the Linux TCP stack the sender would not send a TLP when in the middle of an RTO-triggered recovery.
This corresponds to one of the conditions in the draft, in section "5.4.1. Phase 1: Scheduling a loss probe": the condition that says:
    " 2.  The connection is not in loss recovery"

So perhaps because of this the different SND.NXT "rewind" behavior of different TCP stacks upon RTO may not matter ultimately? Or perhaps there are more implications to unpack. :-)

> > By "the last segment" we mean "the highest-sequence segment (of MSS 
> > or fewer bytes) that has already been transmitted and not ACKed or 
> > SACKed

> Again given that SND.NXT is not rolled back this would presumably the 
> first UN(S)ACKed MSS or fewer bytes walking back from SND.NXT (I am 
> not suggesting an actual walk back, just for illustration).

Yes, exactly.

> I am a bit confused that you used the "or SACKed" since you previously 
> required that PTO not be armed if any SACK blocks were present. If 
> that is followed, then "the last segment" would pretty much always be 
> MSS or fewer bytes just before (non-rolled back) SND.NXT.

Yes, you are exactly right. I should have left the "or SACKed" part out of that line in my e-mail. :-)

By the way, regarding retransmitting the "last segment", this discussion made me realize that AFAICT the draft did not yet discuss why the TLP retransmission is the last segment. I have added some proposed text to our internal draft of the -04 rev to try to describe the rationale:

"When the loss probe is a retransmission, the sender uses the highest-sequence segment sent so far. This is in order to deal with the retransmission ambiguity problem in TCP. Suppose a sender sends N segments, and then retransmits the last segment (segment N) as a loss probe, and then the sender receives a SACK for segment N. As long as the sender waits for any required RACK reordering settling timer to then expire, it doesn't matter if that SACK was for the original transmission of segment N or the TLP retransmission; in either case the arrival of the SACK for segment N provides evidence that the segments preceding segment N were likely lost."

Of course welcome comments/suggestions about this proposed paragraph as well.

Thanks!

neal

---

From: Yuchung Cheng <[email-masked]>
Date: Sat, 19 May 2018 09:27:37 -0700

> Both but primarily due to timer granularity. I'd be interested to know if RACK/TLP in Linux is working well on mobile devices where I assume the timer isn’t as fine grained.
If the loss recovery is based on "time", and the time measurement is
bad, then it simply won't work well.

RACK is designed for modern stack that equipped w/ good clocks and
per-pkt timestamps. We can probably make that more clear in the draft.

> Windows stack exits recovery upon an RTO but keep track of the highest sent sequence number (SND.NXT value before the rewind). So it can do what Linux is doing post RTO. However I am not convinced…

I don't see a safety issue. It's certain worth considering to include
post-timeout phase, especially if TLP is new data to avoid packet
ambiguity issue. hmm maybe we can implement that in Linux...

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 20 May 2018 19:03:19 -0700

Hi Neal,

On Mon, May 14, 2018 at 2:21 PM, Neal Cardwell <[email-masked]> wrote:
> (of MSS

Thanks. The texts look good to me.

> its own dependency on SACK.

Yes, I agree with it because RACK and TLP are two different algorithms.
But, I think it would be better being stated at the early part of the
TLP description,  but not in Scheduling a loss probe.

>    and not the PTO timer. This ensures that the sender does not send

I might want to think about it more, but it looks fine.
Thanks for addressing.
--
Yoshi