Subject: [tcpm] Fast retransmit of Zero Window Probes with 1 byte

From: Dan Ciliske <[email-masked]>
Date: Thu, 15 Dec 2016 13:52:54 -0800

I appear to have run into either an edge case or ambiguity between RFC 793 and 5681 SHOULDs and MUSTs where the sending TCP stack chooses to implement Zero Window Probing by sending 1 byte packets. Error scenario is as follows:

No. Time        Source      Destination   Info
  1 0.000000    10.1.1.62   10.1.1.56     [PSH, ACK] Seq=1 Ack=1 Win=4644 Len=1460
  2 0.000853    10.1.1.62   10.1.1.56     [PSH, ACK] Seq=1461 Ack=1 Win=4644 Len=1460
  3 0.000855    10.1.1.62   10.1.1.56     [PSH, ACK] Seq=2921 Ack=1 Win=4644 Len=1460
  4 0.000855    10.1.1.62   10.1.1.56     [PSH, ACK] Seq=4381 Ack=1 Win=4644 Len=264
  5 0.000856    10.1.1.56   10.1.1.62     [ACK] Seq=1 Ack=1461 Win=3184 Len=0
  6 0.000856    10.1.1.56   10.1.1.62     [ACK] Seq=1 Ack=2921 Win=1724 Len=0
  7 0.002107    10.1.1.56   10.1.1.62     [ACK] Seq=1 Ack=4381 Win=264 Len=0
  8 0.002107    10.1.1.56   10.1.1.62     [TCP ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0

  9 4.966264    10.1.1.62   10.1.1.56     [TCP ZeroWindowProbe] [PSH, ACK] Seq=4645 Ack=1 Win=4644 Len=1
 10 4.966265    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0

 11 5.465671    10.1.1.62   10.1.1.56     [TCP ZeroWindowProbe] [PSH, ACK] Seq=4645 Ack=1 Win=4644 Len=1
 12 5.466429    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0

 13 6.465684    10.1.1.62   10.1.1.56     [TCP ZeroWindowProbe] [PSH, ACK] Seq=4645 Ack=1 Win=4644 Len=1
 14 6.466442    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
 15 6.466443    10.1.1.62   10.1.1.56     [TCP ZeroWindowProbe] [PSH, ACK] Seq=4645 Ack=1 Win=4644 Len=1
 16 6.466443    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
     ...

1-4. Sender Sends final data that fills RWND.
5-8. Receiver Sends ACK for SND.UNA. Receiver fails to respond in a timely manner with an increased window size.
9. Sender Sends Zero Window Probe with 1 byte, increasing SND.UNA by one.
10. Receiver responds with ACK for SND.UNA-1 and RWND of 0. This is the first Duplicate ACK.
11. After delaying, Sender Resends the ZWP.
12. Receiver Responds with ACK for SND.UNA-1 and RWND of 0. This is the second Duplicate ACK.
13. After delaying, Sender Resends the ZWD.
14. Receiver Responds with ACK for SND.UNA-1 and a RWND of 0. This is the third Duplicate ACK.
15. In response to three duplicate ACKs, the Sender now enters Fast Retransmission for the dropped packet, aka the ZWP.

15+. The Sender will respond to all ACKs from the Receiver with a retransmission of the Zero Window Probe due to Fast Retransmission.
16+. The Receiver will respond to all Probes with ACK for SND.UNA-1 and a RWND of 0, until it’s user application reads the data.

—————

On a fast network (i.e. LAN), and a slow receiver (i.e. an embedded device), this becomes a Denial of Service as the receiver ends up spending all its resources ACKing the Probes instead of reading from the socket.

Suggested change, in RFC 5681:
Section 3.2, paragraph 2 says:
    The TCP sender SHOULD use the "fast retransmit" algorithm to detect
    and repair loss, based on incoming duplicate ACKs.  The fast
    retransmit algorithm uses the arrival of 3 duplicate ACKs (as defined
    in section 2, without any intervening ACKs which move SND.UNA) as an
    indication that a segment has been lost.  After receiving 3 duplicate
    ACKs, TCP performs a retransmission of what appears to be the missing
    segment, without waiting for the retransmission timer to expire.

It should say:
    The TCP sender SHOULD use the "fast retransmit" algorithm to detect
    and repair loss, based on incoming duplicate ACKs.  The fast
    retransmit algorithm uses the arrival of 3 duplicate ACKs (as defined
    in section 2, without any intervening ACKs which move SND.UNA) as an
    indication that a segment has been lost.  After receiving 3 duplicate
    ACKs, TCP performs a retransmission of what appears to be the missing
    segment, without waiting for the retransmission timer to expire. 
    However, if all transmitted data from the sender has been acknowledged,
    the duplicate ACKs are in response to a Zero Window Probe, AND the
    advertised window is still zero, then the TCP sender MUST NOT consider
    these to be duplicate ACKs in relation to the fast retransmit algorithm.

-Dan

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 16 Dec 2016 12:47:46 -0500

Good point. The RFC does seem unclear on that.

The BSD code (tcp_input.c or Figure 29.3 in Stevens volume 2) is pretty
clear about simply not treating a window probe as "outstanding data":

  * If we have outstanding data (other than
  * a window probe), ...

So one option would be to go with that approach, and clarify in the
definition of a dupack that a window probe does not count as outstanding
data:

So the existing RFC 5681 text:

   DUPLICATE ACKNOWLEDGMENT: An acknowledgment is considered a
      "duplicate" in the following algorithms when (a) the receiver of
      the ACK has outstanding data,

could be:

   DUPLICATE ACKNOWLEDGMENT: An acknowledgment is considered a
      "duplicate" in the following algorithms when (a) the receiver of
      the ACK has outstanding data (other than a window probe),

Do you mind sharing which TCP implementation was running on the sender in
the scenario you describe? I haven't been able to reproduce that behavior
on Linux, and BSD doesn't seem like it would be susceptible. Just curious.

neal

On Thu, Dec 15, 2016 at 4:52 PM, Dan Ciliske <[email-masked]> wrote:

> I appear to have run into either an edge case or ambiguity between RFC 793
> and 5681 SHOULDs and MUSTs where the sending TCP stack chooses to implement
> Zero Window Probing by sending 1 byte packets. Error scenario is as follows:
>
>  10 4.966265    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP
> ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
>
>  12 5.466429    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP
> ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
>
>  14 6.466442    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP
> ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
>  16 6.466443    10.1.1.56   10.1.1.62     [TCP ZeroWindowProbeAck] [TCP
> ZeroWindow] [ACK] Seq=1 Ack=4645 Win=0 Len=0
>      ...
>
> 15. In response to three duplicate ACKs, the Sender now enters Fast
> Retransmission for the dropped packet, aka the ZWP.
>
> 15+. The Sender will respond to all ACKs from the Receiver with a
> retransmission of the Zero Window Probe due to Fast Retransmission.
> 16+. The Receiver will respond to all Probes with ACK for SND.UNA-1 and a
> RWND of 0, until it’s user application reads the data.
>
> —————
>
> On a fast network (i.e. LAN), and a slow receiver (i.e. an embedded
> device), this becomes a Denial of Service as the receiver ends up spending
> all its resources ACKing the Probes instead of reading from the socket.
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Dan Ciliske <[email-masked]>
Date: Fri, 16 Dec 2016 10:44:17 -0800

Hmm… looks like I may need to dredge up my copy. As to the stack: it’s our own custom embedded TCP/IP stack running on a derivative of uC/OS II. The code triggering this was written nearly 18 years ago… I have no idea why it’s just now showed up.

Also, your wording change seems to be more concise and clear.

-Dan

> On Dec 16, 2016, at 9:47 AM, Neal Cardwell <[email-masked]> wrote:
> 
> The BSD code (tcp_input.c or Figure 29.3 in Stevens volume 2) is pretty clear about simply not treating a window probe as "outstanding data":
> 
> So one option would be to go with that approach, and clarify in the definition of a dupack that a window probe does not count as outstanding data:
> 
> Do you mind sharing which TCP implementation was running on the sender in the scenario you describe? I haven't been able to reproduce that behavior on Linux, and BSD doesn't seem like it would be susceptible. Just curious.
> 
> neal
> 
> 
> On Thu, Dec 15, 2016 at 4:52 PM, Dan Ciliske <[email-masked] <mailto:[email-masked]>> wrote:
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>
>