Subject: [tcpm] A question about Delayed ACK and RTO

From: "zhangyali (D)" <[email-masked]>
Date: Tue, 13 Dec 2016 08:58:03 +0000

Hi all,

Recent days, I am doing some simulation about TCP performance in NS3. I found a phenomenon is a default setting of TCP's delayed ACK is two packets or 200ms. I am wondering if this setting is accord with  some RFCs in IETF. But After I referred to some RFCs, I just found some restrictions, such as, the delay must be less than 0.5ms (RFC1122).

I think the delayed ACK has a close relationship with RTO. Take an extreme scenario, if the delay is longer than RTO, many packets will be retransmitted, which will waste many network resources.

So I want to know do we have some RFCs have given the exact value of both? And if we permit any TCP stack to set them freely, what is the mechanism to balance the mismatch between RTO and delayed ACK?

Best Regards,
Yali

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 13 Dec 2016 10:17:51 -0500

On Tue, Dec 13, 2016 at 3:58 AM, zhangyali (D) <[email-masked]>
wrote:

> Hi all,
>
>
>
> Recent days, I am doing some simulation about TCP performance in NS3. I
> found a phenomenon is a default setting of TCP’s delayed ACK is two packets
> or 200ms. I am wondering if this setting is accord with  some RFCs in IETF.
> But After I referred to some RFCs, I just found some restrictions, such as,
> the delay must be less than 0.5ms (RFC1122).
>
>
I believe the 200ms figure is due to historical reasons. AFAIK it's due to
the BSD delayed ACK behavior (see Stevens "TCP/IP Illustrated Volume 2",
section 25.4 and figure 25.7, which describes the 200ms delayed ACK timer).
Then this value was inherited by other widely-deployed OSes.

> I think the delayed ACK has a close relationship with RTO. Take an extreme
> scenario, if the delay is longer than RTO, many packets will be
> retransmitted, which will waste many network resources.
>

Yes, exactly. In theory, the RTO tries to be adaptive enough to measure any
RTT variations caused by delayed ACKs, and increase the RTO in response to
this. However, it's quite easy for a bulk transfer to never have any
delayed ACKs for most of its lifetime, during which the RTO gradually
converges toward the raw RTT value. Then when there is suddenly a delayed
ACK, there can be a spurious RTO.

To avoid that, many TCP stacks (at least the major open source OSes) have a
hard-coded 200ms "slop factor" or "fudge factor", to try to never let their
estimate of RTT variation fall below that 200ms value, to avoid this
effect.

> So I want to know do we have some RFCs have given the exact value of both?
> And if we permit any TCP stack to set them freely, what is the mechanism to
> balance the mismatch between RTO and delayed ACK?
>

I'm not aware of RFC specifications for exact values of both (delayed ACK
and RTO). However, the historical precedent is very strong, and the 200ms
delayed ACK value was very pronounced in Internet traces at least as
recently as 2011, when I last looked at the effect. (Probably others have
more recent data points for the prevalence of 200ms delayed ACKs.)

At IETF 97 our team at Google presented some features we use for internal
TCP traffic at Google, where the endpoints can negotiate the specific
constant to use for the maximum delayed ACK from the receiver and the
corresponding minimum RTT delay variation for budgeting in the RTO at the
sender:

https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tcp-options-for-low-latency-00.pdf

As this slide deck notes, within Google we negotiate 5ms for delayed ACKs.

cheers,
neal

---

From: Mark Allman <[email-masked]>
Date: Tue, 13 Dec 2016 14:22:46 -0500

I don't disagree with anything Neal said, but want to emphasize
something. 

>     I think the delayed ACK has a close relationship with RTO. Take
>     an extreme scenario, if the delay is longer than RTO, many
>     packets will be retransmitted, which will waste many network
>     resources.
> 
> Yes, exactly. In theory, the RTO tries to be adaptive enough to
> measure any RTT variations caused by delayed ACKs, and increase the
> RTO in response to this.

In other words, the delayed ACK timer and the RTO are not
competing.  The delayed ACK timer should be a subset of the RTO.

That said, ...

> However, it's quite easy for a bulk transfer to never have any
> delayed ACKs for most of its lifetime, during which the RTO
> gradually converges toward the raw RTT value.  Then when there is
> suddenly a delayed ACK, there can be a spurious RTO.

... this is certainly possible.  And, especially so as folks have
moved towards trying to make the RTO converge to the RTT (e.g., by
not reseting the RTO on ACK reception or not using a minRTO or the
like).  The less variance the RTO accounts for the more a spike in
delay (from various places) will fool the RTO.  Many moons ago we
showed there is a fundamental tradeoff between RTO timeliness and
RTO accuracy.  You can optimize for one of those, but it will come
at the expense of the other.

allman

---

From: "zhangyali (D)" <[email-masked]>
Date: Wed, 14 Dec 2016 09:48:22 +0000

Hi Neal,

Thanks for your providing request information.

About the delay of delayed ACK, I found another clue in a SIGCOMM paper in 1988. In the page 14, a sentence is “The 4.5KBps senders were talking to 4.3BSD receivers which would delay an ack until 35% of the window was filled or 200 ms had passed (i.e., an ack was delayed for 5-7 packets on average).” There is no reference about the 200 ms, so I guess this is the particular delay appears for the first time.
I try to calculate the value based on delayed packet numbers, packet size and sending rate. In this case, the delayed packet number is 7, the packet size is 576Byte (refer to RFC879 in 1983), and sundering rate is 4.5Bps. The value is 896 ms! Longer than 200 ms.

>. However, it's quite easy for a bulk transfer to never have any delayed ACKs for most of its lifetime, during which the RTO gradually converges toward the raw RTT value. Then when there is suddenly a delayed ACK, there can be a spurious RTO.
I am not very sure if I see your point. I think the key point is the RTT variation will be weakened when packets are sent in a burst (the consequence of delayed ACK).   And the spurious  could only happen for the last few packets whose number is smaller than delayed packets, right?

About your proposal about negotiation of delayed ACK, a potential problem is that RTO will be stretched than before because you add extra delay. As you have said, most of the packets will not exceed RTO even host enable delayed ACK for most large flows, but the retransmission will be delayed (i.e., 5ms)also once one packet is lost.

Best,
Yali
发件人: Neal Cardwell [mailto:[email-masked]]
发送时间: 2016年12月13日 23:18
收件人: zhangyali (D) <[email-masked]>
抄送: [email-masked]
主题: Re: [tcpm] A question about Delayed ACK and RTO

On Tue, Dec 13, 2016 at 3:58 AM, zhangyali (D) <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

Recent days, I am doing some simulation about TCP performance in NS3. I found a phenomenon is a default setting of TCP’s delayed ACK is two packets or 200ms. I am wondering if this setting is accord with  some RFCs in IETF. But After I referred to some RFCs, I just found some restrictions, such as, the delay must be less than 0.5ms (RFC1122).

I believe the 200ms figure is due to historical reasons. AFAIK it's due to the BSD delayed ACK behavior (see Stevens "TCP/IP Illustrated Volume 2", section 25.4 and figure 25.7, which describes the 200ms delayed ACK timer). Then this value was inherited by other widely-deployed OSes.

Yes, exactly. In theory, the RTO tries to be adaptive enough to measure any RTT variations caused by delayed ACKs, and increase the RTO in response to this. However, it's quite easy for a bulk transfer to never have any delayed ACKs for most of its lifetime, during which the RTO gradually converges toward the raw RTT value. Then when there is suddenly a delayed ACK, there can be a spurious RTO.

To avoid that, many TCP stacks (at least the major open source OSes) have a hard-coded 200ms "slop factor" or "fudge factor", to try to never let their estimate of RTT variation fall below that 200ms value, to avoid this effect.

I'm not aware of RFC specifications for exact values of both (delayed ACK and RTO). However, the historical precedent is very strong, and the 200ms delayed ACK value was very pronounced in Internet traces at least as recently as 2011, when I last looked at the effect. (Probably others have more recent data points for the prevalence of 200ms delayed ACKs.)

At IETF 97 our team at Google presented some features we use for internal TCP traffic at Google, where the endpoints can negotiate the specific constant to use for the maximum delayed ACK from the receiver and the corresponding minimum RTT delay variation for budgeting in the RTO at the sender:

cheers,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 14 Dec 2016 08:23:41 -0500

On Wed, Dec 14, 2016 at 4:48 AM, zhangyali (D) <[email-masked]>
wrote:

> About the delay of delayed ACK, I found another clue in a SIGCOMM paper in
> 1988. In the page 14, a sentence is “The 4.5KBps senders were talking to
> 4.3BSD receivers which would delay an ack until 35% of the window was
> filled or 200 ms had passed (i.e., an ack was delayed for 5-7 packets on
> average).” There is no reference about the 200 ms, so I guess this is the
> particular delay appears for the first time.
>
> I try to calculate the value based on delayed packet numbers, packet size
> and sending rate. In this case, the delayed packet number is 7, the packet
> size is 576Byte (refer to RFC879 in 1983), and sundering rate is 4.5Bps.
> The value is 896 ms! Longer than 200 ms.
>
>
>
> >. However, it's quite easy for a bulk transfer to never have any delayed
> ACKs for most of its lifetime, during which the RTO gradually converges
> toward the raw RTT value. Then when there is suddenly a delayed ACK, there
> can be a spurious RTO.
>
> I am not very sure if I see your point. I think the key point is the RTT
> variation will be weakened when packets are sent in a burst (the
> consequence of delayed ACK).   And the spurious  could only happen for the
> last few packets whose number is smaller than delayed packets, right?
>

Yes, there should only be a delayed ACK at the end of an application chunk
if there is an odd number of packets. But we would expect roughly half of
application chunks to have an odd number of packets. Though the proportion
is probably higher than that, since many application chunks are just one
packet (e.g. an HTTP or RPC request or response).

> About your proposal about negotiation of delayed ACK, a potential problem
> is that RTO will be stretched than before because you add extra delay. As
> you have said, most of the packets will not exceed RTO even host enable
> delayed ACK for most large flows, but the retransmission will be delayed
> (i.e., 5ms)also once one packet is lost.
>

The basic idea of the proposal is to tweak the RTO calculation and turn a
previously existing, historically motivated 200ms fixed "slop factor" into
a dynamically negotiated 5ms "slop factor". In our experience that is
almost always a win.

neal

>
>
> Best,
>
> Yali
>
> *发件人**:* Neal Cardwell [mailto:[email-masked]]
> *发送时间:* 2016年12月13日 23:18
> *收件人:* zhangyali (D) <[email-masked]>
> *抄送:* [email-masked]
> *主题:* Re: [tcpm] A question about Delayed ACK and RTO
>
>
>
> Yes, exactly. In theory, the RTO tries to be adaptive enough to measure
> any RTT variations caused by delayed ACKs, and increase the RTO in response
> to this. However, it's quite easy for a bulk transfer to never have any
> To avoid that, many TCP stacks (at least the major open source OSes) have
> a hard-coded 200ms "slop factor" or "fudge factor", to try to never let
> their estimate of RTT variation fall below that 200ms value, to avoid this
> effect.
>
>
>
>   https://www.ietf.org/proceedings/97/slides/slides-
> 97-tcpm-tcp-options-for-low-latency-00.pdf
>
>
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 14 Dec 2016 16:56:32 +0000

Historically, the minimum RTO is 1 second and actual RTT is very rarely more than 1 second, so all this RTT calculation hardly ever matters anyway.

Thanks,
Jakob.

On Dec 14, 2016, at 5:24 AM, Neal Cardwell <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Dec 14, 2016 at 4:48 AM, zhangyali (D) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Neal,

Yes, there should only be a delayed ACK at the end of an application chunk if there is an odd number of packets. But we would expect roughly half of application chunks to have an odd number of packets. Though the proportion is probably higher than that, since many application chunks are just one packet (e.g. an HTTP or RPC request or response).

The basic idea of the proposal is to tweak the RTO calculation and turn a previously existing, historically motivated 200ms fixed "slop factor" into a dynamically negotiated 5ms "slop factor". In our experience that is almost always a win.

neal

Best,
Yali
?件人: Neal Cardwell [mailto:[email-masked]<mailto:[email-masked]>]
?送??: 2016年12月13日 23:18
收件人: zhangyali (D) <[email-masked]<mailto:[email-masked]>>
抄送: [email-masked]<mailto:[email-masked]>
主?: Re: [tcpm] A question about Delayed ACK and RTO

cheers,
neal

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 14 Dec 2016 12:04:06 -0500

On Wed, Dec 14, 2016 at 11:56 AM, Jakob Heitz (jheitz) <[email-masked]>
wrote:

> Historically, the minimum RTO is 1 second and actual RTT is very rarely
> more than 1 second, so all this RTT calculation hardly ever matters anyway.
>

That may be true historically, but for many years major TCP implementations
(including Linux and FreeBSD) have used a minimum RTO closer to 200ms. And
in datacenter environments even 200ms can be infeasibly high.

neal

>
> Thanks,
> Jakob.
>
>
> On Dec 14, 2016, at 5:24 AM, Neal Cardwell <[email-masked]> wrote:
>
>> About the delay of delayed ACK, I found another clue in a SIGCOMM paper
>> in 1988. In the page 14, a sentence is “The 4.5KBps senders were talking to
>> >. However, it's quite easy for a bulk transfer to never have any
>> I believe the 200ms figure is due to historical reasons. AFAIK it's due
>> to the BSD delayed ACK behavior (see Stevens "TCP/IP Illustrated Volume 2",
>> I think the delayed ACK has a close relationship with RTO. Take an
>> extreme scenario, if the delay is longer than RTO, many packets will be
>> So I want to know do we have some RFCs have given the exact value of
>> both? And if we permit any TCP stack to set them freely, what is the
>> mechanism to balance the mismatch between RTO and delayed ACK?
>>
>>
>>
> [email-masked]

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Wed, 14 Dec 2016 17:21:32 +0000

The problem in a virtualized environment is that RTT is highly variable. Mostly sub millisecond with occasional spikes at several 100 ms. And not just because of delayed ack, but due to effects of VM scheduling and load.

Thanks,
Jakob.

On Dec 14, 2016, at 9:06 AM, Neal Cardwell <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Dec 14, 2016 at 11:56 AM, Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>> wrote:
Historically, the minimum RTO is 1 second and actual RTT is very rarely more than 1 second, so all this RTT calculation hardly ever matters anyway.

That may be true historically, but for many years major TCP implementations (including Linux and FreeBSD) have used a minimum RTO closer to 200ms. And in datacenter environments even 200ms can be infeasibly high.

neal

Thanks,
Jakob.

neal

Best,
Yali
发件人: Neal Cardwell [mailto:[email-masked]<mailto:[email-masked]>]
发送时间: 2016年12月13日 23:18
收件人: zhangyali (D) <[email-masked]<mailto:[email-masked]>>
抄送: [email-masked]<mailto:[email-masked]>
主题: Re: [tcpm] A question about Delayed ACK and RTO

cheers,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 14 Dec 2016 14:13:15 -0500

On Wed, Dec 14, 2016 at 12:21 PM, Jakob Heitz (jheitz) <[email-masked]>
wrote:

> The problem in a virtualized environment is that RTT is highly variable.
> Mostly sub millisecond with occasional spikes at several 100 ms. And not
> just because of delayed ack, but due to effects of VM scheduling and load.
>

True enough. But the RTO mechanism need not try to drive the spurious RTO
rate down to zero. There is some trade-off between faster loss recovery and
an increase in spurious retransmits, and some associated cost-benefit
analysis. These days IMHO the cost of a spurious timer-based loss repair
attempt is relatively low, now that we have TLP and a number of nice undo
mechanisms (FRTO, Eifel, DSACKs). By contrast, in datacenter apps w/
sub-millisecond RTTs, there are prohibitive costs for delaying every
timer-driven loss repair for 200ms.

cheers,
neal

>
> Thanks,
> Jakob.
>
>
> On Dec 14, 2016, at 9:06 AM, Neal Cardwell <[email-masked]> wrote:
>
> That may be true historically, but for many years major TCP
> implementations (including Linux and FreeBSD) have used a minimum RTO
> closer to 200ms. And in datacenter environments even 200ms can be
> infeasibly high.
>
> neal
>
>
>>
>> Thanks,
>> Jakob.
>>
>>
>>> I try to calculate the value based on delayed packet numbers, packet
>>> size and sending rate. In this case, the delayed packet number is 7, the
>>> packet size is 576Byte (refer to RFC879 in 1983), and sundering rate is
>>> 4.5Bps. The value is 896 ms! Longer than 200 ms.
>>>
>>>
>>>
>> Yes, there should only be a delayed ACK at the end of an application
>> chunk if there is an odd number of packets. But we would expect roughly
>> half of application chunks to have an odd number of packets. Though the
>> proportion is probably higher than that, since many application chunks are
>> just one packet (e.g. an HTTP or RPC request or response).
>>
>>
>>
>>> About your proposal about negotiation of delayed ACK, a potential
>>> problem is that RTO will be stretched than before because you add extra
>>> delay. As you have said, most of the packets will not exceed RTO even host
>>> enable delayed ACK for most large flows, but the retransmission will be
>>> delayed (i.e., 5ms)also once one packet is lost.
>>>
>>
>>> To avoid that, many TCP stacks (at least the major open source OSes)
>>> have a hard-coded 200ms "slop factor" or "fudge factor", to try to never
>>> let their estimate of RTT variation fall below that 200ms value, to avoid
>>> this effect.
>>>
>>>
>>>
>>> I'm not aware of RFC specifications for exact values of both (delayed
>>> ACK and RTO). However, the historical precedent is very strong, and the
>>> 200ms delayed ACK value was very pronounced in Internet traces at least as
>>> At IETF 97 our team at Google presented some features we use for
>>> internal TCP traffic at Google, where the endpoints can negotiate the
>>> specific constant to use for the maximum delayed ACK from the receiver and
>>> the corresponding minimum RTT delay variation for budgeting in the RTO at
>>> the sender:
>>>
>>>
>>>

---

From: Joe Touch <[email-masked]>
Date: Wed, 14 Dec 2016 11:20:01 -0800

We should not be optimizing TCP for specific environments; that's for
operators to override.

Joe

On 12/14/2016 9:04 AM, Neal Cardwell wrote:
> On Wed, Dec 14, 2016 at 11:56 AM, Jakob Heitz (jheitz)
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Historically, the minimum RTO is 1 second and actual RTT is very
>     rarely more than 1 second, so all this RTT calculation hardly ever
>     matters anyway.
>
>

---

From: "zhangyali (D)" <[email-masked]>
Date: Thu, 15 Dec 2016 01:45:10 +0000

> Yes, there should only be a delayed ACK at the end of an application chunk if there is an odd number of packets. But we would expect roughly half of application chunks to have an odd number of pack…

I agree with you that odd number of packets may occur spurious RTO more easily, but I think for the flows owning just one packet should not be affected by delayed ACK. AFAIK, slow-start stage will begin with one packet, and sender will send two packets after it receives an ACK. If the first packet is obstructed by the delayed ACK, the ‘clock algorithm’ will be broken down. So receiver should judge if this packet is the first one in the slow-start stage, if yes, send the ack immediately once receiving the packet.

Yali

发件人: Neal Cardwell [mailto:[email-masked]]
发送时间: 2016年12月14日 21:24
收件人: zhangyali (D) <[email-masked]>
抄送: [email-masked]
主题: Re: 答复: [tcpm] A question about Delayed ACK and RTO

neal

Best,
Yali

cheers,
neal

---

From: "zhangyali (D)" <[email-masked]>
Date: Thu, 15 Dec 2016 02:02:13 +0000

Hi Joe,

I really agree with you about TCP mechanism should general for all areas, including wan and data center.

But every scenarios have their own characteristics. Data center has some distinctive features, such as, ultralow RTT, because of centralized location, less hops, and so on. These features determines different requirement for TCP default configuration, compared with wan scenario. It is a solution by having the aids of operator, but maybe we could have a more automatic and general mechanism to solve discrepancy. Just a simple idea. :)

Best Regards,
Yali

发件人: tcpm [mailto:[email-masked]] 代表 Joe Touch
发送时间: 2016年12月15日 3:20
收件人: Neal Cardwell <[email-masked]>; Jakob Heitz (jheitz) <[email-masked]>
抄送: [email-masked]
主题: Re: [tcpm] 答复: A question about Delayed ACK and RTO

We should not be optimizing TCP for specific environments; that's for operators to override.

Joe

neal

---

From: "zhangyali (D)" <[email-masked]>
Date: Thu, 15 Dec 2016 03:08:52 +0000

Please allow me to add one more point. If the probability of odd number of packets occurring  spurious RTO is so outstanding, why nobody try to solve this problem?

发件人: tcpm [mailto:[email-masked]] 代表 zhangyali (D)
发送时间: 2016年12月15日 9:45
收件人: Neal Cardwell <[email-masked]>
抄送: [email-masked]
主题: [tcpm] 答复: 答复: A question about Delayed ACK and RTO

Yali

neal

Best,
Yali

cheers,
neal

---

From: Sepherosa Ziehau <[email-masked]>
Date: Thu, 15 Dec 2016 11:21:13 +0800

I believe delack timeout has been 100ms on FreeBSD since 1999, while
RTO min is >= 200ms by default on FreeBSD.

On Thu, Dec 15, 2016 at 11:08 AM, zhangyali (D) <[email-masked]> wrote:
> Please allow me to add one more point. If the probability of odd number of
> packets occurring  spurious RTO is so outstanding, why nobody try to solve
> this problem?
>
>
>
> I agree with you that odd number of packets may occur spurious RTO more
> easily, but I think for the flows owning just one packet should not be
> affected by delayed ACK. AFAIK, slow-start stage will begin with one packet,
> and sender will send two packets after it receives an ACK. If the first
> packet is obstructed by the delayed ACK, the ‘clock algorithm’ will be
> broken down. So receiver should judge if this packet is the first one in the
> slow-start stage, if yes, send the ack immediately once receiving the
> packet.
>
>
>
> Yali
>
>
>
> 4.3BSD receivers which would delay an ack until 35% of the window was filled
> or 200 ms had passed (i.e., an ack was delayed for 5-7 packets on average).”
> There is no reference about the 200 ms, so I guess this is the particular
> delay appears for the first time.
>
> size is 576Byte (refer to RFC879 in 1983), and sundering rate is 4.5Bps. The
> value is 896 ms! Longer than 200 ms.
>
>
>
> variation will be weakened when packets are sent in a burst (the consequence
> of delayed ACK).   And the spurious  could only happen for the last few
> packets whose number is smaller than delayed packets, right?
>
>
>
> About your proposal about negotiation of delayed ACK, a potential problem is
> that RTO will be stretched than before because you add extra delay. As you
> have said, most of the packets will not exceed RTO even host enable delayed
> ACK for most large flows, but the retransmission will be delayed (i.e.,
> 5ms)also once one packet is lost.
>
>
>
> previously existing, historically motivated 200ms fixed "slop factor" into a
> dynamically negotiated 5ms "slop factor". In our experience that is almost
> always a win.
>
>
>
> neal
>
>
>
>
>
> Best,
>
> Yali
>
> this. However, it's quite easy for a bulk transfer to never have any delayed
> estimate of RTT variation fall below that 200ms value, to avoid this effect.
>
>
>

-- 
Tomorrow Will Never Die

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 15 Dec 2016 00:24:24 -0800

A question would be when TCP sent odd number of packets, how it can
know whether another packets will come from upper layer very soon or
not. Also, I'm not very sure yet if the probability is so obvious. I
can agree that RTO can get close to raw rtt value under bulk transfer.
But, I am guessing not so many apps change the behavior from bulk
transfer to spontaneous small burst transfer.
--
Yoshi

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 15 Dec 2016 10:20:56 +0000

Just adding a few more comments here, because there has been much talk 
about how desirable it is for the RTO to converge to the RTT, and I 
think we need to be clear.

The TCP RTO has two distinct functions:

(1)  the RTO is of course the method of last resort for the recovery of 
a lost segment (the final segment of a burst, following persistent loss, 
etc).

(2) Second the RTO is  method to detect a path failure - where something 
has changed significantly. As such, triggering a conservative (1 or 3 
seconds) RTO implies that state about the path needs to be reset (RTT, 
perhaps invoking PMTUD, congestion state variables, etc, possibly in the 
future falling back from using ECN, etc)

For the first function, there are many ways to detect lost segments 
(Dupacks, probe packets, recovery timers, etc). All of these act faster 
than a conservatively set timeout. They also do not need to erase 
collected path state, they can simply re-transmit the lost segments.

An RTO close to the RTT will be disastrous for some paths. This is one 
of the reasons why people have deployed PEPs to support radio technology 
- we should not be encouraging this. An RTO set close to the RTT can 
produce complex interactions when the path characteristics change (which 
does happen with propagation impairments and radio resource management, 
on wireless, microwave satellite and other paths, and also with 
middleboxes that attempt to control capacity/volume usage). This type of 
path is not going away, talk of wireless links with top speeds of Gbps 
for 5G will likely be accompanied by very large variation in path 
characteristics - at the other extreme people in large parts of the 
world are still with kbps technology. If the timer only recovers 
packets, it would simply result in retransmission, if it triggers other 
probing that would be unfortunate. I think the IETF needs to ensure our 
specs work for all people.

What I am saying is that the second function does not need a timeout 
period near the RTT. Robust protocols have been designed to have a 
conservative RTO timer that is set to much more than a RTT, designed to 
detect an unresponsive endpoint or path problem. The IETF has argued - 
and to my knowledge still argues that Min_RTO should be one second, 
which I think reflects this position. If the loss recovery is efficient, 
this will trigger only infrequently (e.g., persistent loss of one 
packet). I think this is the correct way to design TCP. It motivates 
that we should actually strive for a one second (or so) RTO.

Gorry

---

From: Alejandro Popovsky <[email-masked]>
Date: Thu, 15 Dec 2016 08:25:19 -0300

Tcp would benefit much from and 'end of app write call' signal or may be 
'end of sender buffer data' signal.

TLP, adaptive RTO at the end of bursts, end of burst reordering, and may 
be some other mechanisms
were introduced to help with this issue.

May be the addition of an option for this signal may help, or may be the 
reinterpretation of the push flag.
This would need the modification of both senders and receivers, so the 
effect of not having a modified
receiver getting the flag should be contemplated.

Alejandro.

On 15/12/2016 7:20 a. m., Gorry Fairhurst wrote:
> (1)  the RTO is of course the method of last resort for the recovery 
> of a lost segment (the final segment of a burst, following persistent 
> loss, etc).
>
> (2) Second the RTO is  method to detect a path failure - where 
> something has changed significantly. As such, triggering a 
> conservative (1 or 3 seconds) RTO implies that state about the path 
> needs to be reset (RTT, perhaps invoking PMTUD, congestion state 
> variables, etc, possibly in the future falling back from using ECN, etc)
>
> (Dupacks, probe packets, recovery timers, etc). All of these act 
> faster than a conservatively set timeout. They also do not need to 
> erase collected path state, they can simply re-transmit the lost 
> segments.
>
> of the reasons why people have deployed PEPs to support radio 
> technology - we should not be encouraging this. An RTO set close to 
> the RTT can produce complex interactions when the path characteristics 
> change (which does happen with propagation impairments and radio 
> resource management, on wireless, microwave satellite and other paths, 
> and also with middleboxes that attempt to control capacity/volume 
> usage). This type of path is not going away, talk of wireless links 
> with top speeds of Gbps for 5G will likely be accompanied by very 
> large variation in path characteristics - at the other extreme people 
> in large parts of the world are still with kbps technology. If the 
> timer only recovers packets, it would simply result in retransmission, 
> if it triggers other probing that would be unfortunate. I think the 
> IETF needs to ensure our specs work for all people.
>
> conservative RTO timer that is set to much more than a RTT, designed 
> to detect an unresponsive endpoint or path problem. The IETF has 
> argued - and to my knowledge still argues that Min_RTO should be one 
> second, which I think reflects this position. If the loss recovery is 
> efficient, this will trigger only infrequently (e.g., persistent loss 
> of one packet). I think this is the correct way to design TCP. It 
> motivates that we should actually strive for a one second (or so) RTO.
>
> Gorry
>
>
>>> size
>>>> delayed
>>> consequence
>>> chunk
>>> problem is
>>> extreme
>>> measure any
>>> delayed
>>> have a
>>> both?
>>> internal

---

From: Mark Allman <[email-masked]>
Date: Thu, 15 Dec 2016 09:52:23 -0500

I agree with Gorry's take on variable path characteristics and all. 

> of a lost segment (the final segment of a burst, following
> persistent loss, etc).
> 
> something has changed significantly.

I think this is an accident.  And, the way we should handle it is to
not conflate such disparate functions into one mechanism.

The RTO is a ***retransmission*** timeout.  It's original function
was to decide when to trigger resending data that has not been ACKed
because after this time period chances are the packet was lost
(i.e., case (1)).

Independently we do need to understand path failure.  And, part of
detecting path failure may well be detecting that packets are not
being delivered.  However, this is a different issue than figuring
out when to resend a segment.  The fact that we use the RTO for this
function boils down to laziness in re-using something that was
laying around instead of doing it right.

By trying to shoehorn both functions into one mechanism we are
***guaranteeing*** that the mechanism will be lousy (for at least
one of the tasks and probably both!).  This thread perfectly
illustrates this...  If we were to use the RTO for both functions
then it would be impossible to reconcile Gorry's notion that path
failure happens on long timescales and so RTOs on the order of a
second are needed with Neal's world where he wants to trigger
retransmissions three orders of magnitude sooner.  Path failure may
*leverage* loss detection, but it needn't be the exact same such
that it *hampers* loss detection.

IMHO. :-)

allman

--
http://www.icir.org/mallman/
@mallman_icsi

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Dec 2016 07:29:39 -0800

On 12/15/2016 3:25 AM, Alejandro Popovsky wrote:
> Tcp would benefit much from and 'end of app write call' signal or may
> be 'end of sender buffer data' signal.

 In the current TCP API, there is absolutely no correlation between app
SEND calls and any notion TCP might have of boundaries - TCP presents a
bytestream service only. There would be many consequences to changing
that API, but it might be useful to note that this was once considered.
Just two days ago, Bob Braden and I were discussing TCP v3 (before the
current one  - as documented in IEN  21), which did have this sort of
signal, but it was carried into the packet as begin/end (of "letter") flags.

Note that his behavior is a consequence of the interaction between
delayed ACKs and a lack of application data, and happens any time the
app layer "starves" TCP. It was well studied in the 90's in its
interaction with persistent web connections. It isn't really related to
RTO per se.

Joe

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 15 Dec 2016 18:54:08 +0000

This so called "app starving the TCP" does happen and it happens
often enough to be an issue. The only recovery from a loss at
that time is the retransmission timer.

The retransmission timer is a compromise. If it is set too low, then
spurious retransmissions occur, but if it is set too high, then it
takes too long to retransmit when it is really needed.

The right balance is achieved when an acceptably small number of
spurious retransmissions occur (not zero).

About 3 years ago, I wrote a TCP stack in user space and it faced
highly bursty conditions. I implemented a different RTO mechanism
which worked well. I presented it here:

https://tools.ietf.org/html/draft-heitzhe-tcpm-vm-rto-00
https://www.ietf.org/proceedings/91/slides/slides-91-tcpm-1.pdf

More experience with it revealed an issue.
It would hum along very nicely at 10uS and then hit a spike
of 1 second, causing the session to drop due to session timeout.
A minimum of 1mS prevents the problem.

Thanks,
Jakob.

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Joe Touch
> Sent: Thursday, December 15, 2016 7:30 AM
> To: Alejandro Popovsky <[email-masked]>; [email-masked]
> Subject: Re: [tcpm] 答复: 答复: 答复: A question about Delayed ACK and RTO
> 
> 
>

---

From: Alejandro Popovsky <[email-masked]>
Date: Thu, 15 Dec 2016 20:13:08 -0300

On the second option (no more data to send) there would be no need to
change the API. The tcp sender tcp would generate the signal automatically
with the last segment. And the receiver could just use the info to 
adjust the delay
ack timer for that segment.

And in order for the extra ack not to produce an extra cwnd growth
the sender should follow the recommendation of rfc5681, page 6.

Alejandro.

---

From: hiren panchasara <[email-masked]>
Date: Thu, 15 Dec 2016 16:24:33 -0800

> that it *hampers* loss detection.

Interesting.
How can these 2 be differentiated? Has that been discussed/researched in
IETF or elsewhere? OR can tcp borrow such a scheme from somewhere else?

Cheers,
Hiren

---

From: Mark Allman <[email-masked]>
Date: Thu, 15 Dec 2016 19:48:09 -0500

> How can these 2 be differentiated?

Easy.

Retransmissions are triggered by the scheme outlined in RFC 6298.

Path failure is defined as not getting any packets through in 10sec.

There.  Differentiated.

I am not suggesting that is smart.  Or, the best way.  But, making
these two things triggered by their own criteria that is tuned to
the each specific task is trivial.

allman

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 16 Dec 2016 01:13:36 +0000

FWIW the windows TCP implementation preserves the notion of application send() message boundary by setting the PSH bit only on the last segment generated from the posted buffer. Unfortunately other stacks seem to set PSH on all segments making any receiver optimization of immediate ACK on PSH impossible, because that would effectively turn off delayed ACKs. That said, with ACK stretching, LRO etc. being very common maybe it is time for TCP implementations to turn off delayed ACKs across the board except for request-response applications.

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Alejandro Popovsky
Sent: Thursday, December 15, 2016 3:13 PM
To: Joe Touch <[email-masked]>; [email-masked]
Subject: Re: [tcpm] 答复: 答复: 答复: A question about Delayed ACK and RTO

On the second option (no more data to send) there would be no need to change the API. The tcp sender tcp would generate the signal automatically with the last segment. And the receiver could just use the info to adjust the delay ack timer for that segment.

And in order for the extra ack not to produce an extra cwnd growth the sender should follow the recommendation of rfc5681, page 6.

Alejandro.

On 15/12/2016 12:29 p. m., Joe Touch wrote:
>   In the current TCP API, there is absolutely no correlation between 
> app SEND calls and any notion TCP might have of boundaries - TCP 
> presents a bytestream service only. There would be many consequences 
> to changing that API, but it might be useful to note that this was once considered.
> interaction with persistent web connections. It isn't really related 
> to RTO per se.
>
> Joe
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 16 Dec 2016 01:21:06 +0000

While operators could tweak the settings, having TCP perform well out of box in low latency environments is very useful. For low latency connections Windows Server has shipped with a 20 msec min RTO and 10 msec delayed ACK timeout for several years now. But this method relies on a heuristic to use the SYN handshake RTT to detect low latency connections and pick the more aggressive timeouts which is less than ideal. An explicit negotiation of the timeout would be very useful because it removes any chances of mismatch and helps different OS TCP stacks interop.

From: tcpm [mailto:[email-masked]] On Behalf Of zhangyali (D)
Sent: Wednesday, December 14, 2016 6:02 PM
To: Joe Touch <[email-masked]>; Neal Cardwell <[email-masked]>; Jakob Heitz (jheitz) <[email-masked]>
Cc: [email-masked]
Subject: [tcpm] 答复: 答复: A question about Delayed ACK and RTO

Hi Joe,

But every scenarios have their own characteristics. Data center has some distinctive features, such as, ultralow RTT, because of centralized location, less hops, and so on. These features determines different requirement for TCP default configuration, compared with wan scenario. It is a solution by having the aids of operator, but maybe we could have a more automatic and general mechanism to solve discrepancy. Just a simple idea. ☺

Best Regards,
Yali

发件人: tcpm [mailto:[email-masked]] 代表 Joe Touch
发送时间: 2016年12月15日 3:20
收件人: Neal Cardwell <[email-masked]<mailto:[email-masked]>>; Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>
抄送: [email-masked]<mailto:[email-masked]>
主题: Re: [tcpm] 答复: A question about Delayed ACK and RTO

Joe

neal

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Dec 2016 17:50:59 -0800

> ack timer for that segment.

The two problems:
    - the sending app can't signal "last segment" of a data chunk to
TCP. All it can say is "close", after which it can't send data anymore.
    - the sending-side TCP can't signal "last segment" to the receiver
either. The only flag there is FIN, which similarly inhibits later data
on the connection in that direction.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Dec 2016 17:59:36 -0800

On 12/15/2016 5:13 PM, Praveen Balasubramanian wrote:
> FWIW the windows TCP implementation preserves the notion of application send() message boundary by setting the PSH bit only on the last segment generated from the posted buffer.
TCP isn't required to allow the app to decide when to set the PSH bit
(it's a MAY in the SEND call). It can coalesce multiple PSH bits to send
a larger segment.

However, it's also explicitly not a record boundary. PUSH is really just
to avoid deadlock between the app and TCP on the send side.

>  Unfortunately other stacks seem to set PSH on all segments making any receiver optimization of immediate ACK on PSH impossible, because that would effectively turn off delayed ACKs. That said, with ACK stretching, LRO etc. being very common maybe it is time for TCP implementations to turn off delayed ACKs across the board except for request-response applications.

There are many ways this could be fixed but they all require going
outside the current spec.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 15 Dec 2016 18:01:42 -0800

On 12/15/2016 5:21 PM, Praveen Balasubramanian wrote:
>
> While operators could tweak the settings, having TCP perform well out
> of box in low latency environments is very useful.
>

IMO, the only rule "out of the box" is that TCP ought to *work*.

Every other rule ends up compromising "works" for "works well", which is
a mistake IMO. It just makes the whole Internet brittle in the future,
WHEN (not if) your assumptions change.

Joe

---

From: hiren panchasara <[email-masked]>
Date: Thu, 15 Dec 2016 18:07:54 -0800

> the each specific task is trivial.

Thank you. I think I am getting the point you are trying to make and I
agree that we need to separate them but I thought coming up wiht that
'10sec' was the trickiest part. May be I am missing something or
over-thinking this. :-)

RTO, at least, gets derived from something realistic. I guess 'path
failure' needs to come up with something like that?

Cheers,
Hiren

PS: PMTUD implementation (at least in FreeBSD) comes to mind when I
think about detecting anomaly beyond just 'packets are getting lost'.
But thats again based on RTO in FreeBSD. Though I think we are talking
about more fatal failures here.

---

From: "zhangyali (D)" <[email-masked]>
Date: Fri, 16 Dec 2016 07:02:05 +0000

So, the probability of delaying segment caused by delayed ACK is obvious, right? I did found the phenomenon that some small flows’ FCT is very large, that I think sender’s timeout is triggered by receiver’s delayed ACK.

I think it really make sense to set longer RTO value than delayed ACK, which will reduce spurious timeout caused by delayed ACK. But another question needs to be considered is that how sender know how long will be delayed in the receiver if receiver could set it freely. For example, the default delayed ACK and RTO are both 200ms in some Linux OS, so the spurious timeout in the sender happen more frequently when one segment is delayed in the receiver and sender have no segment to be send.

So I think we should have one method to give a standard value or to coordinate these two parameters automatically.

Yali

发件人: Praveen Balasubramanian [mailto:[email-masked]]
发送时间: 2016年12月16日 9:21
收件人: zhangyali (D) <[email-masked]>; Joe Touch <[email-masked]>; Neal Cardwell <[email-masked]>; Jakob Heitz (jheitz) <[email-masked]>
抄送: [email-masked]
主题: RE: [tcpm] 答复: 答复: A question about Delayed ACK and RTO

From: tcpm [mailto:[email-masked]] On Behalf Of zhangyali (D)
Sent: Wednesday, December 14, 2016 6:02 PM
To: Joe Touch <[email-masked]<mailto:[email-masked]>>; Neal Cardwell <[email-masked]<mailto:[email-masked]>>; Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [tcpm] 答复: 答复: A question about Delayed ACK and RTO

Hi Joe,

Best Regards,
Yali

Joe

neal

---

From: "zhangyali (D)" <[email-masked]>
Date: Fri, 16 Dec 2016 07:28:57 +0000

Yes, I think it's a good choice to differentiate the last segment. But maybe it's not easy to confirm if it is the last segment, even there is just one segment in buffer, since it is determined by the upper application. 

Another question is that for the flow sending one segment per a long time, will it suffer from delayed ACK?

Yali

-----邮件原件-----
发件人: tcpm [mailto:[email-masked]] 代表 Alejandro Popovsky
发送时间: 2016年12月16日 7:13
收件人: Joe Touch <[email-masked]>; [email-masked]
主题: Re: [tcpm] 答复: 答复: 答复: A question about Delayed ACK and RTO

Alejandro.

---

From: "zhangyali (D)" <[email-masked]>
Date: Fri, 16 Dec 2016 07:47:00 +0000

I identify with your distinction function of RTO. I really agree that we should differentiate the effect caused by network congestion or path failure, and take different actions, such as, more aggressive action to the second cases. But now, we just have RTO as the last method to retransmission the segment that is lost or just be delayed if we do not have extra segments or window to trigger dupack. So another retansmission detection method faster than timeout and more effective should be thinked about.

Yali

-----邮件原件-----
发件人: Gorry Fairhurst [mailto:[email-masked]] 
发送时间: 2016年12月15日 18:21
收件人: Yoshifumi Nishida <[email-masked]>; zhangyali (D) <[email-masked]>
抄送: [email-masked]
主题: Re: [tcpm] 答复: 答复: 答复: A question about Delayed ACK and RTO

Just adding a few more comments here, because there has been much talk about how desirable it is for the RTO to converge to the RTT, and I think we need to be clear.

(1)  the RTO is of course the method of last resort for the recovery of a lost segment (the final segment of a burst, following persistent loss, etc).

(2) Second the RTO is  method to detect a path failure - where something has changed significantly. As such, triggering a conservative (1 or 3
seconds) RTO implies that state about the path needs to be reset (RTT, perhaps invoking PMTUD, congestion state variables, etc, possibly in the future falling back from using ECN, etc)

For the first function, there are many ways to detect lost segments (Dupacks, probe packets, recovery timers, etc). All of these act faster than a conservatively set timeout. They also do not need to erase collected path state, they can simply re-transmit the lost segments.

An RTO close to the RTT will be disastrous for some paths. This is one of the reasons why people have deployed PEPs to support radio technology
- we should not be encouraging this. An RTO set close to the RTT can produce complex interactions when the path characteristics change (which does happen with propagation impairments and radio resource management, on wireless, microwave satellite and other paths, and also with middleboxes that attempt to control capacity/volume usage). This type of path is not going away, talk of wireless links with top speeds of Gbps for 5G will likely be accompanied by very large variation in path characteristics - at the other extreme people in large parts of the world are still with kbps technology. If the timer only recovers packets, it would simply result in retransmission, if it triggers other probing that would be unfortunate. I think the IETF needs to ensure our specs work for all people.

What I am saying is that the second function does not need a timeout period near the RTT. Robust protocols have been designed to have a conservative RTO timer that is set to much more than a RTT, designed to detect an unresponsive endpoint or path problem. The IETF has argued - and to my knowledge still argues that Min_RTO should be one second, which I think reflects this position. If the loss recovery is efficient, this will trigger only infrequently (e.g., persistent loss of one packet). I think this is the correct way to design TCP. It motivates that we should actually strive for a one second (or so) RTO.

Gorry

On 15/12/2016 08:24, Yoshifumi Nishida wrote:
>> Please allow me to add one more point. If the probability of odd 
>> number of packets occurring  spurious RTO is so outstanding, why 
>> nobody try to solve this problem?
>>
>>
>>
>>> chunk if there is an odd number of packets. But we would expect 
>>> roughly half of application chunks to have an odd number of packets. 
>>> Though the proportion is probably higher than that, since many 
>>> application chunks are just one packet (e.g. an HTTP or RPC request or response).
>>
>>
>>
>> I agree with you that odd number of packets may occur spurious RTO 
>> more easily, but I think for the flows owning just one packet should 
>> not be affected by delayed ACK. AFAIK, slow-start stage will begin 
>> with one packet, and sender will send two packets after it receives 
>> an ACK. If the first packet is obstructed by the delayed ACK, the 
>> ‘clock algorithm’ will be broken down. So receiver should judge if 
>> this packet is the first one in the slow-start stage, if yes, send 
>> the ack immediately once receiving the packet.
>>
>>
>>
>> Yali
>>
>>
>>
>> About the delay of delayed ACK, I found another clue in a SIGCOMM 
>> paper in 1988. In the page 14, a sentence is “The 4.5KBps senders 
>> were talking to 4.3BSD receivers which would delay an ack until 35% 
>> of the window was filled or 200 ms had passed (i.e., an ack was delayed for 5-7 packets on average).”
>> There is no reference about the 200 ms, so I guess this is the 
>> size and sending rate. In this case, the delayed packet number is 7, 
>> the packet size is 576Byte (refer to RFC879 in 1983), and sundering 
>> rate is 4.5Bps. The value is 896 ms! Longer than 200 ms.
>>
>>
>>
>> I am not very sure if I see your point. I think the key point is the 
>> RTT variation will be weakened when packets are sent in a burst (the consequence
>> chunk if there is an odd number of packets. But we would expect 
>> roughly half of application chunks to have an odd number of packets. 
>> Though the proportion is probably higher than that, since many 
>> application chunks are just one packet (e.g. an HTTP or RPC request or response).
>>
>>
>>
>>
>>
>> problem is that RTO will be stretched than before because you add 
>> extra delay. As you have said, most of the packets will not exceed 
>> RTO even host enable delayed ACK for most large flows, but the 
>> retransmission will be delayed (i.e., 5ms)also once one packet is lost.
>>
>>
>>
>> The basic idea of the proposal is to tweak the RTO calculation and 
>> turn a previously existing, historically motivated 200ms fixed "slop 
>> factor" into a dynamically negotiated 5ms "slop factor". In our 
>> experience that is almost always a win.
>>
>>
>>
>> neal
>>
>>
>>
>>
>>
>> Best,
>>
>> Yali
>>
>> Recent days, I am doing some simulation about TCP performance in NS3. 
>> I found a phenomenon is a default setting of TCP’s delayed ACK is two 
>> packets or 200ms. I am wondering if this setting is accord with  some RFCs in IETF.
>> But After I referred to some RFCs, I just found some restrictions, 
>> such as, the delay must be less than 0.5ms (RFC1122).
>>
>>
>>
>> I believe the 200ms figure is due to historical reasons. AFAIK it's 
>> due to the BSD delayed ACK behavior (see Stevens "TCP/IP Illustrated 
>> Volume 2", section 25.4 and figure 25.7, which describes the 200ms delayed ACK timer).
>> extreme scenario, if the delay is longer than RTO, many packets will 
>> be retransmitted, which will waste many network resources.
>>
>>
>>
>> RTO in response to this. However, it's quite easy for a bulk transfer 
>> to never have any delayed ACKs for most of its lifetime, during which 
>> the RTO gradually converges toward the raw RTT value. Then when there 
>> is suddenly a delayed ACK, there can be a spurious RTO.
>>
>>
>>
>> have a hard-coded 200ms "slop factor" or "fudge factor", to try to 
>> never let their estimate of RTT variation fall below that 200ms value, to avoid this effect.
>>
>>
>>
>> And if we permit any TCP stack to set them freely, what is the 
>> ACK and RTO). However, the historical precedent is very strong, and 
>> the 200ms delayed ACK value was very pronounced in Internet traces at 
>> least as recently as 2011, when I last looked at the effect. 
>> (Probably others have more recent data points for the prevalence of 
>> 200ms delayed ACKs.)
>>
>>
>>
>> specific constant to use for the maximum delayed ACK from the 
>> receiver and the corresponding minimum RTT delay variation for 
>> budgeting in the RTO at the
>> sender:
>>
>>
>>
>>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 16 Dec 2016 08:16:09 +0000

> I identify with your distinction function of RTO. I really agree that we should differentiate the effect caused by network congestion or path failure, and take different actions, such as, more aggr…
On this I also agree -  Tail Loss has become a problem for 
latency-sensitive shorter traffic bursts, amd we need widely deployed 
mechanisms to deal with this. There have been proposals in TCPM to add 
mechanisms that would help and this seems worth looking at.

Gorry
> 收件人: Yoshifumi Nishida<[email-masked]>; zhangyali (D)<[email-masked]>
> 抄送: [email-masked]
>> On Wed, Dec 14, 2016 at 7:08 PM, zhangyali (D)<[email-masked]>  wrote:
>>> 收件人: Neal Cardwell<[email-masked]>
>>> 抄送: [email-masked]
>>> 收件人: zhangyali (D)<[email-masked]>
>>> 抄送: [email-masked]
>>> 收件人: zhangyali (D)<[email-masked]>
>>> 抄送: [email-masked]

---

From: David Borman <[email-masked]>
Date: Fri, 16 Dec 2016 08:06:41 -0600

> On Dec 15, 2016, at 7:59 PM, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Other TCP implementations have done the same thing for decades.

> to avoid deadlock between the app and TCP on the send side.

The PUSH bit is a way for the sending TCP to signal the receiving TCP that no more data is coming, and so if the receiving TCP is buffering data and has not yet made it available to the receiving application, it should now make all that data available to be read.  These days this doesn’t matter as much, because TCP stacks make all received data immediately available to the receiving application.  But senders should continue to set the PUSH bit.

			-David Borman

---

From: Joe Touch <[email-masked]>
Date: Fri, 16 Dec 2016 08:47:44 -0800

> Other TCP implementations have done the same thing for decades.
RFC793 states that the PSH bit is independent of segment boundaries.
Both it and RFC1122 indicate that the send() call MAY support setting
that flag and give provisions for what to do when that flag is not
supported. However, we can't simply assume that this means "last segment
generated".

It means only "last segment in this group" as part of an interactive
transfer. There could be another group following it immediately.

> The PUSH bit is a way for the sending TCP to signal the receiving TCP that no more data is coming, 

as part of this "group" only. There's no rule that a PSH means that the
sending app has paused at all.

> and so if the receiving TCP is buffering data and has not yet made it available to the receiving application, it should now make all that data available to be read. 

RIght - push this through as a group. This is about when to avoid delays
to the receiving app. It's not useful as a signal to override delayed
ACKs because it doesn't require the sender to pause.

>  These days this doesn’t matter as much, because TCP stacks make all received data immediately available to the receiving application.  But senders should continue to set the PUSH bit.

The conditions under which they need to do this are spec'd in RFC1122.
Applications can't assume access via the send() call, so they shouldn't
design their apps with this assumption.

Joe