Subject: [tcpm] RFC 6675 SACK tracking

From: Matt Olson <[email-masked]>
Date: Tue, 4 Dec 2018 23:26:14 +0000

Hi, I have a question about the RFC 6675 conditions for determining bytes lost. Below I've pasted two relevant parts of the RFC. The conditions for marking a byte lost are "2MSS or 3 discontiguous SACKed regions come after the byte." The paragraph below from Section 9 provides some rationale for the "2MSS" part, which changed from RFC3517 (which had it as 3MSS): we want to detect loss even when three sub-MSS packets after the byte made it to the peer. In particular it says "this retains the requirement that at least three segments following the sequence number in question have been SACKed." But, this doesn't seem to add up: a single received SACK block covering 2MSS might only represent two packets.

Besides this inconsistency in the RFC, I wonder: why do we care about "discontiguous SACKed regions" anyway? If we receive three ACKs, each with a SACK block that is a proper superset of the previous one, can't we conclude that three packets arrived at the peer and (following the spirit, if not the letter, of RFC 6675) conclude that any bytes previous to the union of those SACKs were lost? Even if together they SACK less than 2MSS bytes?

Thanks!

https://tools.ietf.org/html/rfc6675

>From section 4:
IsLost (SeqNum):
      This routine returns whether the given sequence number is
      considered to be lost.  The routine returns true when either
      DupThresh discontiguous SACKed sequences have arrived above
      'SeqNum' or more than (DupThresh - 1) * SMSS bytes with sequence
      numbers greater than 'SeqNum' have been SACKed.  Otherwise, the
      routine returns false.

>From section 9:
   The function IsLost () has been modified to require that more than
   (DupThresh - 1) * SMSS octets have been SACKed above a given sequence
   number as indication that it is lost, which is changed from the
   minimum requirement of (DupThresh * SMSS) described in [RFC3517].
   This retains the requirement that at least three segments following
   the sequence number in question have been SACKed, while improving
   detection in the event that the sender has outstanding segments which
   are smaller than SMSS.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 5 Dec 2018 08:31:20 -0800

> Besides this inconsistency in the RFC, I wonder: why do we care about "discontiguous SACKed regions" anyway? If we receive three ACKs, each with a SACK block that is a proper superset of the previo…
In my opinion we should not care about the continuity of SACK, or even
counting SACKs. I've proposed a new way to detect losses (implemented
by Linux & NetFlix.BSD)

https://tools.ietf.org/html/draft-ietf-tcpm-rack-04

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 5 Dec 2018 18:45:23 +0100

> On 5. Dec 2018, at 17:31, Yuchung Cheng <ycheng=[email-masked]> wrote:
> 
It is in the FreeBSD code base and will be included in the upcoming release 12.0.
The code was contributed by Netflix.

Best regards
Michael

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 5 Dec 2018 18:57:12 +0000

It is understood that RACK loss detection doesn't have any ambiguity and most of it is implemented in Windows as an addendum to 6675. But RFC 6675 is Standards Track and is implemented in several OS versions that are deployed. Can someone with historical context please chime in on what the intent was so we can resolve the inconsistency and also clarify the requirement around continuity? 

-----Original Message-----
From: tcpm <[email-masked]> On Behalf Of Michael Tuexen
Sent: Wednesday, December 5, 2018 9:45 AM
To: Yuchung Cheng <ycheng=[email-masked]>
Cc: maolson=[email-masked]; [email-masked] Extensions <[email-masked]>
Subject: Re: [tcpm] RFC 6675 SACK tracking

Best regards
Michael
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools
> .ietf.org%2Fhtml%2Fdraft-ietf-tcpm-rack-04&amp;data=02%7C01%7Cpravb%40
> microsoft.com%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C636796287446259136&amp;sdata=x14zcF4RSsfSPb3lw
> 9zkfpcLZPYpWJZXllTcWOJZwhA%3D&amp;reserved=0
> 
>> 
>> Thanks!
>> 
>> 
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftool
>> s.ietf.org%2Fhtml%2Frfc6675&amp;data=02%7C01%7Cpravb%40microsoft.com%
>> 7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd011db47
>> %7C1%7C0%7C636796287446259136&amp;sdata=lc8lSSoypNl7PjuqVu2KYxjq3o971
>> oCMy4idlXDRWnk%3D&amp;reserved=0
>> 
>>> From section 4:
>> IsLost (SeqNum):
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
>> ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7
>> cd011db47%7C1%7C0%7C636796287446259136&amp;sdata=IqnoUOScKKei3PwnYM%2
>> B0HmbrY1EScWdIZoYlGvhAiUU%3D&amp;reserved=0
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.i
> etf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microso
> ft.com%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636796287446259136&amp;sdata=IqnoUOScKKei3PwnYM%2B0Hm
> brY1EScWdIZoYlGvhAiUU%3D&amp;reserved=0

_______________________________________________
tcpm mailing list
[email-masked]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796287446259136&amp;sdata=IqnoUOScKKei3PwnYM%2B0HmbrY1EScWdIZoYlGvhAiUU%3D&amp;reserved=0

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 5 Dec 2018 14:11:39 -0500

If we take RFC 6675 literally, I don't think there is an inconsistency.
Importantly, I don't think it quite says: "2MSS or 3 discontiguous SACKed
regions come after the byte."

In section 4 (IsLost) and Section 9 (Changes Relative to RFC 3517), it says
"more than (DupThresh - 1) * SMSS" bytes have been SACKed.  DupThresh is 3,
so that means "more than 2*SMSS". To get more than 2*SMSS requires at least
3 segments. So that requirement seems to mean essentially 3 or more
segments, where one of those segments may not be full sized.

So I think that is consistent with the statement:

  This retains the requirement that at least three segments following
   the sequence number in question have been SACKed

Does that sound plausible?

neal

On Wed, Dec 5, 2018 at 1:57 PM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> It is understood that RACK loss detection doesn't have any ambiguity and
> most of it is implemented in Windows as an addendum to 6675. But RFC 6675
> is Standards Track and is implemented in several OS versions that are
> deployed. Can someone with historical context please chime in on what the
> intent was so we can resolve the inconsistency and also clarify the
> requirement around continuity?
>
> >> Hi, I have a question about the RFC 6675 conditions for determining
> bytes lost. Below I've pasted two relevant parts of the RFC. The conditions
> for marking a byte lost are "2MSS or 3 discontiguous SACKed regions come
> after the byte." The paragraph below from Section 9 provides some rationale
> for the "2MSS" part, which changed from RFC3517 (which had it as 3MSS): we
> want to detect loss even when three sub-MSS packets after the byte made it
> to the peer. In particular it says "this retains the requirement that at
> least three segments following the sequence number in question have been
> SACKed." But, this doesn't seem to add up: a single received SACK block
> covering 2MSS might only represent two packets.
> >>
> >> Besides this inconsistency in the RFC, I wonder: why do we care about
> "discontiguous SACKed regions" anyway? If we receive three ACKs, each with
> a SACK block that is a proper superset of the previous one, can't we
> conclude that three packets arrived at the peer and (following the spirit,
> if not the letter, of RFC 6675) conclude that any bytes previous to the
> union of those SACKs were lost? Even if together they SACK less than 2MSS
> bytes?

---

From: Markku Kojo <[email-masked]>
Date: Wed, 5 Dec 2018 21:28:54 +0200 (EET)

This is exactly the correct interpretation, so no inconsistency.

And the other rule ("DupThresh discontiguous SACKed sequences") 
should be obvious:  DupThresh discontiguous SACKed sequences is an 
alternative way to guarantee that at least DupThresh segments following
the sequence number in question have been SACKed, regardless of the size 
of the segments.

/Markku

On Wed, 5 Dec 2018, Neal Cardwell wrote:

> If we take RFC 6675 literally, I don't think there is an inconsistency. Importantly, I don't think it quite says: "2MSS or 3 discontiguous SACKed
> In section 4 (IsLost) and Section 9 (Changes Relative to RFC 3517), it says "more than (DupThresh - 1) * SMSS" bytes have been SACKed.  DupThresh is
> 3, so that means "more than 2*SMSS". To get more than 2*SMSS requires at least 3 segments. So that requirement seems to mean essentially 3 or more
> On Wed, Dec 5, 2018 at 1:57 PM Praveen Balasubramanian <pravb=[email-masked]> wrote:
>       It is understood that RACK loss detection doesn't have any ambiguity and most of it is implemented in Windows as an addendum to 6675. But
>       RFC 6675 is Standards Track and is implemented in several OS versions that are deployed. Can someone with historical context please chime
>       in on what the intent was so we can resolve the inconsistency and also clarify the requirement around continuity?
>
>       >> Hi, I have a question about the RFC 6675 conditions for determining bytes lost. Below I've pasted two relevant parts of the RFC. The
>       conditions for marking a byte lost are "2MSS or 3 discontiguous SACKed regions come after the byte." The paragraph below from Section 9
>       provides some rationale for the "2MSS" part, which changed from RFC3517 (which had it as 3MSS): we want to detect loss even when three
>       sub-MSS packets after the byte made it to the peer. In particular it says "this retains the requirement that at least three segments
>       following the sequence number in question have been SACKed." But, this doesn't seem to add up: a single received SACK block covering 2MSS
>       might only represent two packets.
>       >>
>       >> Besides this inconsistency in the RFC, I wonder: why do we care about "discontiguous SACKed regions" anyway? If we receive three ACKs,
>       each with a SACK block that is a proper superset of the previous one, can't we conclude that three packets arrived at the peer and
>       (following the spirit, if not the letter, of RFC 6675) conclude that any bytes previous to the union of those SACKs were lost? Even if
>       together they SACK less than 2MSS bytes?
>       > .ietf..org%2Fhtml%2Fdraft-ietf-tcpm-rack-04&amp;data=02%7C01%7Cpravb%40
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7
> C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796287446259136&amp;sdata=IqnoUOScKKei3PwnYM%2B0HmbrY1EScWdIZoYlGv
>       hAiUU%3D&amp;reserved=0
>

---

From: Ilpo Järvinen <[email-masked]>
Date: Wed, 5 Dec 2018 21:43:46 +0200 (EET)

On Tue, 4 Dec 2018, Matt Olson wrote:

> bytes lost. Below I've pasted two relevant parts of the RFC. The 
> conditions for marking a byte lost are "2MSS or 3 discontiguous SACKed 
> regions come after the byte." The paragraph below from Section 9 
> provides some rationale for the "2MSS" part, which changed from RFC3517 
> (which had it as 3MSS): we want to detect loss even when three sub-MSS 
> packets after the byte made it to the peer. In particular it says "this 
> retains the requirement that at least three segments following the 
> sequence number in question have been SACKed." But, this doesn't seem to 
> add up: a single received SACK block covering 2MSS might only represent 
> two packets.

It has > 2MSS not >= 2MSS ("more than 2MSS").

With Nagle, only one <MSS segment is allowed in flight per RTT so for 
flows with it, that rule alone should be enough.

> "discontiguous SACKed regions" anyway? If we receive three ACKs, each 
> with a SACK block that is a proper superset of the previous one, can't 
> we conclude that three packets arrived at the peer and (following the 
> spirit, if not the letter, of RFC 6675) conclude that any bytes previous 
> to the union of those SACKs were lost? Even if together they SACK less 
> than 2MSS bytes?

You'd need to add variables for tracking those developments of supersets 
(in fact, a variable per seqno given how the IsLost function is specified 
in RFC6675).

But yes, in general, new SACK information in each ACK is even better 
condition (and if possible, matching against segment boundaries that were 
used for sending them out to protect also against any kind of SACK 
splitting would also be nice to have) but tracking that has its own 
implementation efficiency challenges even if the implementation would not 
be locked in how the scoreboard details are arranged in RFC6675 algorithm 
(this comes from a guy who has implemented significant portion of the 
scoreboard handling in one OS).

-- 
 i.

---

From: Matt Olson <[email-masked]>
Date: Wed, 5 Dec 2018 19:20:00 +0000

Great catch. I misread it as “>= 2MSS” rather than “> 2MSS”. With “> 2MSS”, it makes sense. Thanks, Neal!

The question about why “discontiguity” fundamentally matters still stands, though.

From: Neal Cardwell <ncardwell=[email-masked]>
Sent: Wednesday, December 5, 2018 11:12 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: Michael Tuexen <[email-masked]>; [email-masked]; Matt Olson <[email-masked]>; [email-masked]
Subject: Re: [tcpm] RFC 6675 SACK tracking

If we take RFC 6675 literally, I don't think there is an inconsistency. Importantly, I don't think it quite says: "2MSS or 3 discontiguous SACKed regions come after the byte."

In section 4 (IsLost) and Section 9 (Changes Relative to RFC 3517), it says "more than (DupThresh - 1) * SMSS" bytes have been SACKed.  DupThresh is 3, so that means "more than 2*SMSS". To get more than 2*SMSS requires at least 3 segments. So that requirement seems to mean essentially 3 or more segments, where one of those segments may not be full sized.

neal

On Wed, Dec 5, 2018 at 1:57 PM Praveen Balasubramanian <pravb=[email-masked]<mailto:[email-masked]>> wrote:
It is understood that RACK loss detection doesn't have any ambiguity and most of it is implemented in Windows as an addendum to 6675. But RFC 6675 is Standards Track and is implemented in several OS versions that are deployed. Can someone with historical context please chime in on what the intent was so we can resolve the inconsistency and also clarify the requirement around continuity?

-----Original Message-----
From: tcpm <[email-masked]<mailto:[email-masked]>> On Behalf Of Michael Tuexen
Sent: Wednesday, December 5, 2018 9:45 AM
To: Yuchung Cheng <ycheng=[email-masked]<mailto:[email-masked]>>
Cc: maolson=[email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]> Extensions <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tcpm] RFC 6675 SACK tracking

> On 5. Dec 2018, at 17:31, Yuchung Cheng <ycheng=[email-masked]<mailto:[email-masked]>> wrote:
>
> <maolson=[email-masked]<mailto:[email-masked]>> wrote:
>>

Best regards
Michael
> .ietf..org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384731348&sdata=ehsP59JuaR7eVEd%2BuAb6w8BE9tU9ptffTDizlwpPGg4%3D&reserved=0>%2Fhtml%2Fdraft-ietf-tcpm-rack-04&amp;data=02%7C01%7Cpravb%40
> microsoft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmicrosoft.com&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384731348&sdata=Jm32Bw6lKP4VPEBAi%2FyuAN01XYAUCWmkkQT06nGvHHc%3D&reserved=0>%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91a
>> s.ietf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fs.ietf.org&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384741356&sdata=KNHf%2Fca5na7m%2BYcTD1fjZ1oxAXZGYg1DUt6T1H30%2Feg%3D&reserved=0>%2Fhtml%2Frfc6675&amp;data=02%7C01%7Cpravb%40microsoft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2F40microsoft.com&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384751365&sdata=hDsR34w8TxlAq%2BsOqXa%2FFtwwmoqr32Vpy%2FQljwXCqWo%3D&reserved=0>%
>> [email-masked]<mailto:[email-masked]>
>> ietf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fietf.org&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384751365&sdata=opXA4V2rG5m0EcjYX2Y7XNdQ52VL5RRlyf%2Bcf9ERd6A%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> soft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsoft.com&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384761373&sdata=mSTZkq4Po0gFWLkgDZtj9VDyuydpYU6cthSjav%2Fslc0%3D&reserved=0>%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7
> [email-masked]<mailto:[email-masked]>
> etf.org<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fetf.org&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384761373&sdata=iK92veURpYghjHzSD4QncTZ9W%2BFsdx7EVe5hjkjYotE%3D&reserved=0>%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microso
> ft.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fft.com&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384771381&sdata=SEdhKadDBJvBdj%2BYKphsvF4gNFRuzwcRKtEF9WFtXsI%3D&reserved=0>%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd0

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40microsoft.com%7C3e30aef4f4f74719c27e08d65ad97ab3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796287446259136&amp;sdata=IqnoUOScKKei3PwnYM%2B0HmbrY1EScWdIZoYlGvhAiUU%3D&amp;reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384781390&sdata=vWz23fvILbo2R6ms8fAKHHs2hoFmRkzxjzY7JbUa%2BSQ%3D&reserved=0>

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Ftcpm&data=02%7C01%7Cmaolson%40microsoft.com%7C2afc39ca65e7455e5da908d65ae5918d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636796339384781390&sdata=vWz23fvILbo2R6ms8fAKHHs2hoFmRkzxjzY7JbUa%2BSQ%3D&reserved=0>