Subject: [TLS] Sending fatal alerts over TCP

From: Florian Weimer <[email-masked]>
Date: Tue, 20 Dec 2011 13:15:31 +0000

TCP has the property that connection shutdowns can overtake application
data if connections are used in full-duplex mode.

Here's an example:

  X sends a fatal TLS alert to Y
  X shuts down the connection
  Y sends data to X
  X receives from Y, sends RST to signal data loss to Y
  Y receives RST

At this point, when Y tries to send further data, it will receive EPIPE
from the sockets API.  When it tries to read data, it will receive
ECONNRESET.  According to the socket API sepcification, there is no way
to access the initial TLS alert.  Note that this behavior is not related
to the SO_LINGER socket option.

(I see this with an OpenJDK server and a GNUTLS client, when the server
sends a certificate_unknown alert immediately after receiving the
Certificate message.  The client never receives it.)

What is the best way to deal with this?  I plan to avoid closing the
connection immediately after sending the alert, essentially waiting for
the client to close the connection.  (There will be a short timeout, but
denial-of-service issues have to be dealt with out-of-band anyway.)

Are there any better options?

--
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

---

From: Marsh Ray <[email-masked]>
Date: Tue, 20 Dec 2011 12:12:36 -0600

>    X receives from Y, sends RST to signal data loss to Y

This would seem to be against the spirit of
http://tools.ietf.org/html/rfc793 (section 3.5. Closing a Connection) 
which says "Users must keep reading connections they close for sending 
until the TCP says no more data."

> to access the initial TLS alert.

My interpretation is that RSTs are generally to be avoided by 
properly-coded applications and a fatal TLS alert should not prevent the 
graceful close of the TCP socket.

I have heard of client apps that crash badly upon receiving an RST. 
(Alternatively this could be used to argue that they should be sent much 
more frequently, but that's another discussion :-)

- Marsh

---

From: Nico Williams <[email-masked]>
Date: Tue, 20 Dec 2011 12:22:08 -0600

> denial-of-service issues have to be dealt with out-of-band anyway.)

Your approach seems to be a variable-sized queue with a minimum time
on the queue for each socket on that queue.

A variant on this: use a fixed-sized queue.  Create a fixed-sized
queue, shutdown() SHUT_RDWR (but don't close()) after sending the
fatal alert, add the socket to the queue; if the queue is full take
out the oldest socket and close() it.  No need for a reaper thread:
properly behaved clients will cause those sockets to close, leaving no
state beyond a file descriptor around on the server side; bad clients
will not be able to do anything more serious than a) consume a bound
number of server-side resources, b) cause legitimate clients to get
RST before being able to see fatal alerts.

The right size for the queue will depend on the latency for sending,
receiving and processing the fatal alert message relative to typical
connection lifetime.  But it will probably be easiest to size the
shutdown socket queue as a fraction of the max open connections you
can/want to handle.

A fixed-sized queue seems to have better DoS protection: you know what
is the max number of sockets in this state is that the attacker can
force you to have, and you convert a resource consumption DoS into a
case where bad clients can cause some legitimate clients will get an
RST without first seeing a fatal alert.

> Are there any better options?

I think a fixed-sized queue w/o a reaper thread is likely easier to
code and is simpler to analyze regarding DoS attacks than a variable
size queue with a reaper thread and a "reasonable" timeout per-socket
on that queue.

Nico
--

---

From: Bodo Moeller <[email-masked]>
Date: Tue, 20 Dec 2011 19:32:04 +0100

> until the TCP says no more data."

Note that RFC 1122 section 4.2.2.13 specifies that

            A host MAY implement a "half-duplex" TCP close sequence, so
            that an application that has called CLOSE cannot continue to
            read data from the connection.  If such a host issues a
            CLOSE call while received data is still pending in TCP, or
            if new data is received after CLOSE is called, its TCP
            SHOULD send a RST to show that data was lost.

I'm guessing that on typical systems, this behavior for close() *does*
depend on the SO_LINGER parameters, though.

Bodo

---

From: Nico Williams <[email-masked]>
Date: Tue, 20 Dec 2011 12:41:08 -0600

Ah, yes, just set SO_LINGER and a reasonable linger time (and have the
socket in non-blocking mode, but that probably goes without saying).

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Tue, 20 Dec 2011 22:25:43 +0100 (MET)

> to the SO_LINGER socket option.

Could you point me to the particular section of the socket API spec
that defines such behaviour (for the TCP stack of Y) to be correct?

The way that I read rfc1122 section 4.2.2.13, the TCP stack of Y is
only allowed to kill data upon the Y app performing a close operation
on the socket.  If the app on Y does not call close when receiving
the EPIPE from send(), then the data must IMHO remain readable,
and the ECONNRESET will follow the TLS alert.

-Martin

---

From: Martin Rex <[email-masked]>
Date: Tue, 20 Dec 2011 22:53:18 +0100 (MET)

> and the ECONNRESET will follow the TLS alert.

Application calling close() upon receiving EPIPE from send()
refers to "shutdown(socket,SHUT_WR)" in terms of BSD sockets.

With respect to this paragraph of rfc1122 4.2.2.13:

This means that a TCP implementation could decide to treat
shutdown(socket,SHUT_WR) and shutdown(socket,SHUT_RD) always and
exactly the same as shutdown(socket,SHUT_RDWR), i.e. not support
a unidirectional/partial closure.

Who is inappropriately calling shutdown(socket,SHUT_WR) after a
failed send() in the scenario that you're looking at?  If there is
none such code, then the TCP stack is not conforming to rfc1122.

-Martin

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 21 Dec 2011 15:29:23 +1300

Florian Weimer <[email-masked]> writes:

>What is the best way to deal with this? 

If it's the other side that does it then there's not much you can do, if
you're doing it then you need to wander into a twisty little maze of
shutdown()/close() passages, all different, buggy, and incompatible:

/* Close a connection.  Safely handling closes is extremely difficult due to 
   a combination of the way TCP/IP (and TCP stacks) work and various bugs 
   and quirks in implementations.  After a close (and particularly if short-
   timeout non-blocking writes are used) there can still be data left in 
   TCP send buffers, and also as unacknowledged segments on the network.  At 
   this point there's no easy way for the TCP stack to know how long it 
   should hang around trying to get the data out and waiting for acks to 
   come back.  If it doesn't wait long enough, it'll end up discarding 
   unsent data.  If it waits too long, it could potentially wait forever in 
   the presence of network outages or crashed peers.  What's worse, since 
   the socket is now closed, there's no way to report any problems that may 
   occur at this point back to the caller.

   We try and handle this with a combination of shutdown() and close(), but 
   due to implementation bugs/quirks and the TCP stack issues mentioned 
   above this doesn't work all of the time.  The details get very 
   implementation-specific, for example with glibc the manpage says that 
   setting SO_LINGER causes shutdown() not to return until queued messages 
   are sent (which is wrong, and non-glibc implementations like PHUX and 
   Solaris specifically point out that only close() is affected), but that 
   shutdown() discards unsent data.  glibc in turn is dependent on the 
   kernel it's running on top of, under Linux shutdown() returns immediately 
   but data is still sent regardless of the SO_LINGER setting.

   BSD Net/2 and later (which many stacks are derived from, including non-
   Unix systems like OS/2) returned immediately from a close() but still 
   sent queued data on a best-effort basis.  With SO_LINGER set and a zero 
   timeout the close was abortive (which Linux also implemented starting 
   with the 2.4 kernel), and with a non-zero timeout it would wait until all 
   the data was sent, which meant that it could block almost indefinitely 
   (minutes or even hours, this is the worst-case behaviour mentioned 
   above).  This was finally fixed in 4.4BSD (although a lot of 4.3BSD-
   derived stacks ended up with the indefinite-wait behaviour), but even 
   then there was some confusion as to whether the wait time was in machine-
   specific ticks or seconds (Posix finally declared it to be seconds).  
   Under Winsock, close() simply discards queued data while shutdown() has 
   the same effect as under Linux, sending enqueued data asynchronously 
   regardless of the SO_LINGER setting.

   This is a real mess to sort out safely, the best that we can do is to 
   perform a shutdown() followed later by a close().  Messing with SO_LINGER 
   is too risky and something like performing an ioWait() doesn't work 
   either because it just results in whoever initiated the shutdown being 
   blocked for the I/O wait time, and waiting for a recv() of 0 bytes isn't 
   safe because the higher-level code may need to read back a shutdown ack 
   from the other side which a recv() performed at this point would 
   interfere with.  Under Windows we could handle it by waiting for an 
   FD_CLOSE to be posted but this requires the use of a window handle which 
   we don't have access to, and which may not even exist for some classes of
   applications */

Peter.

---

From: Martin Rex <[email-masked]>
Date: Tue, 20 Dec 2011 22:53:18 +0100 (MET)

(clarified & re-sent)

> > data if connections are used in full-duplex mode.

No, it can not (for implementations conforming to rfc1122).

>From rfc1122, Section 4.2.2.13 "Closing a connection":

     http://tools.ietf.org/html/rfc1122#page-88

"Application calling CLOSE" refers to calling shutdown()
(rather than close(), which would destroy the socket handle)
in terms of BSD sockets.

That paragraph from rfc1122 means that a TCP implementation could
decide to treat shutdown(socket,SHUT_WR) and shutdown(socket,SHUT_RD)
always and exactly the same as shutdown(socket,SHUT_RDWR),
i.e. not support a unidirectional/partial closure.

> 
> > Here's an example:
> > 
> > (1)  X sends a fatal TLS alert to Y
> > (2)  X shuts down the connection
> > (3)  Y sends data to X
> > (4)  X receives from Y, sends RST to signal data loss to Y
> > (5)  Y receives RST
> > 
> > (6) At this point, when Y tries to send further data,
> > it will receive EPIPE from the sockets API.

Correct.

>
> > When it tries to read data, it will receive ECONNRESET.

That can not happen _unless_ some buggy code in the application on Y
called shutdown(socket,SHUT_WR) when being faced with EPIPE on send().

>
> > Note that this behavior is not related

I believe this is incorrect.  If X is using SO_LINGER, then
(2) will cause X to send a TCP FIN to Y, which the TCP
implementation on X ought to translate into a regular
End-Of-File on recv(), rather than ECONNRESET.

>
> > According to the socket API sepcification, there is no way

As long as Y does not call shutdown() on the socket, it will
receive all data that arrived on the socket prior to seeing
End-of-file (X with SO_LINGER) or ECONNRESET (X without SO_LINGER)
on recv().

If you have some app on Y _not_ seeing the fatal TLS alert
(but you saw it coming in on the network with a tool like Wireshark,
then there is probably incorrect app code that erroneously called
shutdown(socket,SHUT_WR) while using a TCP implementation that
does not support a full-duplex close, or an incorrect app that
called shutdown(socket,SHUT_RDWR), maybe as an inappropriate
knee-jerk reaction of being faced with EPIPE on send(),
prior to trying to recv() the alert.

-Martin

---

From: Bodo Moeller <[email-masked]>
Date: Wed, 21 Dec 2011 11:07:09 +0100

> End-of-file (X with SO_LINGER) or ECONNRESET (X without SO_LINGER)
That's not right.  Upon receiving RST, Y's TCP will reset the state for
this connection and discard any data already buffered; no further data can
be received by the application (in this case, by the TLS implementation);
check RFC 793 for RST processing details.  If X doesn't want that, it must
not send an RST.

(With a fatal alert, that doesn't sound *that* problematic because there's
a problem anyway.  I've seen TLS servers send RSTs instead of cleanly
closing the connection when there was *no* error, which is very bad because
then you depend on client-side scheduling -- if the TLS client isn't able
to process the response before the TLS connection is killed, you have a
problem.)

Bodo

---

From: Florian Weimer <[email-masked]>
Date: Wed, 21 Dec 2011 12:32:18 +0000

> shutdown()/close() passages, all different, buggy, and incompatible:

Yeah, that's about as much fine print as I feared.  Thanks.

Fortunately, I'm only trying to resolve an interoperability problem
between GNUTLS and the OpenJDK implementation.  There seems to be some
confusion about the error reporting for connection failures, and I'm
trying to provide better messages to consumers.  It turns out that both
GNUTLS and OpenJDK properly report the certificate_unknown alert during
handshake failure when talking to themselves implementation.  The
difference between GNUTLS and OpenJDK when talking to an OpenJDK server
is this: the GNUTLS client sends the Certificate and Client Key Exchange
messages in separate TLS records and (in the way we use it) TCP
segments, while the OpenJDK client uses a single TLS record in a single
TCP segment.  As a result, the race leading to reception of an early RST
is rarely lost by the OpenJDK client (if ever).

I changed the GNUTLS client to create just one TCP segment per server
round-trip, and I can now read the TLS alert before the RST segment
arrives.  So the fix was quite trivial after all.

---

From: Martin Rex <[email-masked]>
Date: Wed, 21 Dec 2011 20:13:33 +0100 (MET)

> check RFC 793 for RST processing details.  If X doesn't want that, it must

That is not what my copies of rfc793 and rfc1122 specify.

Unless the local application performs a shutdown(), the TCP implementation
of Y MUST continue to deliver data that was received.

(4) from above indicates that X must have changed the SO_LINGER
socket option and called close() on the connection rather than
shutdown(SHUT_RD). (see Stevens, Unix Network Programming Vol.1,
on my second edition it is section 7.5, figure 7.10, page 191,
"close, l_onoff=1, l_linger=0") with the result that the TCP of X
has no more memories about the connection and considers the
connection "CLOSED".    In the alternative situation, X's TCP would
have sent a TCP FIN and be in the synchronized state FIN_WAIT_1.

rfc793 on Reset generation:
  
http://tools.ietf.org/html/rfc793#page-36

  Reset Generation

  As a general rule, reset (RST) must be sent whenever a segment arrives
  which apparently is not intended for the current connection.  A reset
  must not be sent if it is not clear that this is the case.

Specified processing for X in (CLOSED):

  There are three groups of states:

    1.  If the connection does not exist (CLOSED) then a reset is sent
    in response to any incoming segment except another reset.  In
    particular, SYNs addressed to a non-existent connection are rejected
    by this means.

    If the incoming segment has an ACK field, the reset takes its
    sequence number from the ACK field of the segment, otherwise the
    reset has sequence number zero and the ACK field is set to the sum
    of the sequence number and segment length of the incoming segment.
    The connection remains in the CLOSED state.

which means that X must send an RST with a SEQ number of 0!

Specified processing for X in (FIN_WAIT_1):

    3.  If the connection is in a synchronized state (ESTABLISHED,
    FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT),
    any unacceptable segment (out of window sequence number or
    unacceptible acknowledgment number) must elicit only an empty
    acknowledgment segment containing the current send-sequence number
    and an acknowledgment indicating the next sequence number expected
    to be received, and the connection remains in the same state.

    If an incoming segment has a security level, or compartment, or
    precedence which does not exactly match the level, and compartment,
    and precedence requested for the connection,a reset is sent and
    connection goes to the CLOSED state.  The reset takes its sequence
    number from the ACK field of the incoming segment.

which means that X must not send an RST at all.
On Shutdown() X dropped its receive buffer (so that the receive window
is now 0 (Zero)), but still knows the incoming Seqence number.  So
any further incoming data is "out of window" and must elicit only an
empty acknowledgement segment (rather than RST).

Now looking at the requirements for the RST processing:

http://tools.ietf.org/html/rfc793#page-37

  Reset Processing

  In all states except SYN-SENT, all reset (RST) segments are validated
  by checking their SEQ-fields.  A reset is valid if its sequence number
  is in the window.  In the SYN-SENT state (a RST received in response
  to an initial SYN), the RST is acceptable if the ACK field
  acknowledges the SYN.

  The receiver of a RST first validates it, then changes state.  If the
  receiver was in the LISTEN state, it ignores it.  If the receiver was
  in SYN-RECEIVED state and had previously been in the LISTEN state,
  then the receiver returns to the LISTEN state, otherwise the receiver
  aborts the connection and goes to the CLOSED state.  If the receiver
  was in any other state, it aborts the connection and advises the user
  and goes to the CLOSED state.

The RST generated by X at (CLOSED) has a SEQ 0, so it very probably
fail SEQ-validation and must be completely ignored by Y's TCP
Y is still in ESTABLISHED, because X didn't send a FIN.
If X had sent a FIN, then Y would be in CLOSE_WAIT, but X would 
have been prohibited to sent a FIN (and required to send an empty ACK
instead).

  3.5. Closing a Connection

  CLOSE is an operation meaning "I have no more data to send."  The
  notion of closing a full-duplex connection is subject to ambiguous
  interpretation, of course, since it may not be obvious how to treat
  the receiving side of the connection.  We have chosen to treat CLOSE
  in a simplex fashion.  The user who CLOSEs may continue to RECEIVE
  until he is told that the other side has CLOSED also.  Thus, a program
  could initiate several SENDs followed by a CLOSE, and then continue to
  RECEIVE until signaled that a RECEIVE failed because the other side
  has CLOSED.  We assume that the TCP will signal a user, even if no
  RECEIVEs are outstanding, that the other side has closed, so the user
  can terminate his side gracefully.  A TCP will reliably deliver all
  buffers SENT before the connection was CLOSED so a user who expects no
  data in return need only wait to hear the connection was CLOSED
  successfully to know that all his data was received at the destination
  TCP.  Users must keep reading connections they close for sending until
  the TCP says no more data.

-Martin

---

From: Marsh Ray <[email-masked]>
Date: Wed, 21 Dec 2011 13:24:58 -0600

> of Y MUST continue to deliver data that was received.

This seems relevant http://tools.ietf.org/html/rfc793

> [when in state:] ESTABLISHED  FIN-WAIT-1 FIN-WAIT-2 CLOSE-WAIT
>
>         If the RST bit is set then, any outstanding RECEIVEs and SEND
>         should receive "reset" responses.  All segment queues should be
>         flushed.  Users should also receive an unsolicited general
>         "connection reset" signal.  Enter the CLOSED state, delete the
>         TCB, and return.

This part about "all segment queues should be flushed" suggests to me 
that received-but-undelivered application data MAY or SHOULD be 
discarded. Deleting the TCB altogether certainly sounds final.

- Marsh

---

From: Martin Rex <[email-masked]>
Date: Wed, 21 Dec 2011 20:39:50 +0100 (MET)

> discarded. Deleting the TCB altogether certainly sounds final.

In case that you're referring to  page 70,
this text appears in a section

  "second check the RST bit,"

and can only be ever reached when it was
not fully processed by the previous "first check sequence number"
a page before, which says:

  first check sequence number

      SYN-RECEIVED STATE
      ESTABLISHED STATE
      FIN-WAIT-1 STATE
      FIN-WAIT-2 STATE
      CLOSE-WAIT STATE
      CLOSING STATE
      LAST-ACK STATE
      TIME-WAIT STATE

        Segments are processed in sequence.  Initial tests on arrival
        are used to discard old duplicates, but further processing is
        done in SEG.SEQ order.  If a segment's contents straddle the
        boundary between old and new, only the new parts should be
        processed.

        There are four cases for the acceptability test for an incoming
        segment:

        Segment Receive  Test
        Length  Window
        ------- -------  -------------------------------------------

           0       0     SEG.SEQ = RCV.NXT

           0      >0     RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND

          >0       0     not acceptable

          >0      >0     RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND
                      or RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND

        If the RCV.WND is zero, no segments will be acceptable, but
        special allowance should be made to accept valid ACKs, URGs and
        RSTs.

*>      If an incoming segment is not acceptable, an acknowledgment
*>      should be sent in reply (unless the RST bit is set, if so drop
*>      the segment and return):

          <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>

        After sending the acknowledgment, drop the unacceptable segment
        and return.

-Martin

---

From: Bodo Moeller <[email-masked]>
Date: Wed, 21 Dec 2011 20:50:40 +0100

Specified processing for X in (CLOSED):

> which means that X must send an RST with a SEQ number of 0!

The segment sent by Y in step (3) will have the ACK bit set, so the RST
sent by X in (4) will have the appropriate sequence number that causes Y to
accept the RST in step (5), and thus to discard all data.

(By the way, I think the Linux TCP implementation will send a RST in that
situation if close() has been used to shut down the connection regardless
of SO_LINGER settings.  Stevens doesn't describe that particular TCP
implementation.)

Bodo

---

From: Marsh Ray <[email-masked]>
Date: Wed, 21 Dec 2011 14:04:02 -0600

On 12/21/2011 01:39 PM, Martin Rex wrote:
> *>       should be sent in reply (unless the RST bit is set, if so [...]

But let's say the RST segment passes the sequence number check.

It seems that this language applies:
> Initial tests on arrival  are used to discard old duplicates,
>   but further processing is  done in SEG.SEQ order.
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

So maybe I am starting to agree with you (but for a different reason :-)

- Marsh

---

From: Bodo Moeller <[email-masked]>
Date: Wed, 21 Dec 2011 21:28:32 +0100

>>  but further processing is  done in SEG.SEQ order.
Note that this only implies that Y's TCP needs to process the segment
containing X's TLS alert before it can process the RST segment. Processing
the segment with the alert merely means queuing it; the TLS layer would
have to "RECEIVE" it from the TCP to do something with it.  If the RST
segment gets processed before the TLS layer has received the TLS alert, the
receiving queue will be flushed, and the alert will be gone.

---

From: Martin Rex <[email-masked]>
Date: Wed, 21 Dec 2011 22:40:37 +0100 (MET)

> accept the RST in step (5), and thus to discard all data.

Whooops.  It seems I got lost in the woods.  I'm sorry.

You were correct.

TCP seems to not provide a means to gently signal "I'm no longer listening",
but only for "I'm finished talking" (FIN).  And the only alternative to
signal "I'm no longer listening" is (RST), but that has quite rude
side-effects of potentially killing stuff that may still be in the
receivers incoming TCP segment buffers not yet known to the receiver.

And SO_LINGER seems to help only for gently conveying the
"I'm finished talking" situation, but does not help in the
"I'm no longer listening" situation, as Florian asserted,
due to the lack of such a feature in the TCP design.

While the non-lingering close is similar to the "A Crashed" (X=A, Y=B)
in rfc793 page 35, Figure 11:

  An interesting alternative case occurs when TCP A crashes and TCP B
  tries to send data on what it thinks is a synchronized connection.
  This is illustrated in figure 11.  In this case, the data arriving at
  TCP A from TCP B (line 2) is unacceptable because no such connection
  exists, so TCP A sends a RST.  The RST is acceptable so TCP B
  processes it and aborts the connection.

        TCP A                                              TCP B

  1.  (CRASH)                                   (send 300,receive 100)

  2.  (??)    <-- <SEQ=300><ACK=100><DATA=10><CTL=ACK> <-- ESTABLISHED

  3.          --> <SEQ=100><CTL=RST>                   --> (ABORT!!)

           Active Side Causes Half-Open Connection Discovery

                               Figure 11.

What I haven't found is a description of the message flow for
"I'm no longer listening" situation.  The TCP Connection State Diagram
(rfc793, page 23, figure 6.) contains a "CLOSE/snd FIN" state transition,
but that means "I'm finished talking", but lacks the state transition
from ESTAB for "I'm finished listening".

So I'm slightly confused what that means for the POSIX API function
shutdown() that does offer the semantics "I'm no longer listening"
(first entry of the table in Stevens, Network programming vol.1 2nd ed,
page 191, Figure 7.10):

  shutdown, SHUT_RD  No more receives can be issued on socket; process can
                     still send on socket; socket receive buffer is
                     discarded; any further data received is discarded
                     by TCP (Excercise 6.5); no effect on socket send buffer

-Martin

---

From: Nico Williams <[email-masked]>
Date: Wed, 21 Dec 2011 16:31:20 -0600

Stevens suggests[0] doing a shutdown(2) of SHUT_WR then read(2)ing or
select(2)ing (or more modern equivalent) for read events on the same
socket to wait for the FIN|ACK from the peer; read(2) is supposed to
return 0 at that point (though I suppose it could also actually read
data if there's data to be read, as only the write-side of the socket
will have been shutdown at that point).

That wouldn't be an indication that the application received the fatal
alert, of course.  Nothing short of waiting for the peer to close its
end of the socket will do, but we can't wait forever.

Setting SO_LINGER with a zero l_linger is out, and a non-zero value is
useless unless the thread is willing to block in close(2) -- a
terrible waste of resources.

It may well be that the best thing to do is to shutdown(socket,
SHUT_WR) then queue up the socket for a later close(2) with SO_LINGER
and zero l_linger.  This gives the peer application a pretty good
chance to receive the fatal alert and close the socket normally, while
also limiting resource usage on the side that sent the alert.  And if
you already have an event-based async I/O framework you can trivially
set a timer upon which to close() the socket, and even process any
further data sent by the peer.

[0] Unix Network Programming volume 1, 2nd ed., section 7.5, page 189
/ fig. 7.8.

Nico
--

---

From: Peter Gutmann <[email-masked]>
Date: Thu, 22 Dec 2011 17:14:49 +1300

Bodo Moeller <[email-masked]> writes:

>[quoting RFC]

For situations like this, in which the OP asked about dealing with real-world
behaviour, you can't quote the TCP RFCs, you can't even quote the Berkely
sockets specs or manpages, you have to look at what the actual
implementation(s) do, because that often bears little or no resemblance to the
specs.  That's not just because of bugs, it's because the implementations are
based on extensive real-world experience and tuning (and, sometimes, de-
tuning) which require slight... adaptations away from what the specs say.
Going from the TCP RFC is at best going to give you an abstract model of what
some implementations may do in certain circumstances, but it won't tell you
how to deal with real-world implementations and behaviour.

Peter.

---

From: Bodo Moeller <[email-masked]>
Date: Thu, 22 Dec 2011 14:34:48 +0100

> tuning) which require slight... adaptations away from what the specs say.

Indeed.  However, I haven't quoted the RFCs to prove what real-world
implementation are doing -- I have looked at the RFCs to confirm that what
real-world implementations seem to be doing is, as far as relevant to this
discussion, not in violation of the RFC.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 22 Dec 2011 06:48:44 -0800

If someone wants to summarize where they think that this thread ended and send it over to the TCPM WG ([email-masked]), we can get a sanity check from the people who actually know how implementations act and how they are supposed to.

--Paul Hoffman

---

From: Florian Weimer <[email-masked]>
Date: Thu, 22 Dec 2011 15:03:37 +0000

> receiving queue will be flushed, and the alert will be gone.

Indeed, this is the correct explanation.

This flushing behavior was introduced that when you hit ^C in your FTP
client during a transfer, the transfer actually stops quickly, and does
not continue until the server has transmitted all data.  As a result, it
is impossible to tear down full-duplex connections without potential
data loss, unless the connection teardown is negotiated above the TCP
layer.