Subject: Channel close logic, needs clarification?

From: [email-masked] (Niels M�ller)
Date: 08 Mar 2001 17:31:58 +0100

Do we need to make some more detailed recommendations about channel
close logic? Different behaviour can cause interoperability problems.

For instance, I have heard reports that when logging in with the
openssh client to a recent lshd, the client will hang after logout.
I'll describe the logic that lsh and lshd uses. The basic rules are as
follows:

1. When SSH_MSG_CHANNEL_CLOSE is both sent and received, the channel
   is killed. This is unconditionally required by the spec.

2. When SSH_MSG_CHANNEL_EOF is both sent and received,
   SSH_MSG_CHANNEL_CLOSE is sent (unless it wasn't sent earlier). This
   makes sense in almost all cases (and the only exceptions I'm aware
   of seem out-of-scope here, they are related to proxying or
   gatewaying channels).

So, what is left to decide is when to send SSH_MSG_CHANNEL_EOF. In
most cases, the decision is easy, but sometimes it is not.

Consider the scenario where a client creates a session channel, gets a
login shell, and later exits the shell by explicitly typing exit or
logout. The result, at the server side, is

 * The server handles SIGCLD, discovers that its child process died,
   and sends an SSH_MSG_CHANNEL_REQUEST "exit-status". This is of
   course not done directly by the signal handler, but some time later
   from the main loop.

 * The server gets an EOF condition on the process' stdout and stderr
   (unless their are any grand children that still has one or the other
   open). When EOF has occured on both stdout and stderr, the server
   sends an SSH_MSG_CHANNEL_EOF.

The two events above can occur in more or less random order. The
server has not received any SSH_MSG_CHANNEL_EOF, so rule 2 doesn't
apply. The channel is still open.

What happens at the client? When the lsh client receives
an SSH_MSG_CHANNEL_REQUEST "exit-status" message, it decides that it's
time to stop talking, and it sends an SSH_MSG_CHANNEL_EOF. Now rule 2
applies, and the channel gets closed properly.

It's not obvious that the channel should be closed at this time; there
may be some grand child process that is still alive. If the client has
received a SSH_MSG_CHANNEL_EOF that implies that no process has its
*output* connected to the channel, but even in this case (which lsh
doesn't try to look for), there may be some process with input still
connected to the channel. And this is why the server doesn't close the
channel automatically at process death; the user may still have some
processes alive to talk to, and the client is more likely than the
server to know what the user really wants.

But appearantly, this logic doesn't work right when talking to the
openssh client. It doesn't send any SSH_MSG_CHANNEL_EOF, and to the
user it appears that the system just hangs.

Does the lsh behaviour above make sense to you? What are other
implementations doing?

Regards,
/Niels

---

From: Markus Friedl <[email-masked]>
Date: Sun, 11 Mar 2001 13:54:09 +0100

> close logic? Different behaviour can cause interoperability problems.

Yes, I think so :)
I'm quoting all of your message since the context is important.

> openssh client to a recent lshd, the client will hang after logout.

Yes, I can reproduce this problem. Do other SSH client implementations
hang, too?

>    is killed. This is unconditionally required by the spec.

So does OpenSSH.

>    of seem out-of-scope here, they are related to proxying or

I'm not sure that you can rely on getting an EOF message.

>    sends an SSH_MSG_CHANNEL_EOF.

Same here (more or less).

> apply. The channel is still open.

Same here.

> applies, and the channel gets closed properly.

The OpenSSH client considers the "exit-status" message just informational
and does _not_ send an EOF message back.

> user it appears that the system just hangs.

The OpenSSH server does not wait for the EOF message from the client.
However, since the servers' child process is dead, the server decides
that it will not write to more data to the 'stdin' handle and does _not_
wait for the EOF message from the client.  This way the the channel is
correctly closed.

> Does the lsh behaviour above make sense to you? What are other

Yes, I'm really interested in what other SSH implementations are
doing in these cases.

-markus

---

From: Sami Lehtinen <[email-masked]>
Date: Sun, 11 Mar 2001 19:52:49 +0200

Markus Friedl, on March 11. 2001, wrote:
  : On Thu, Mar 08, 2001 at 05:31:58PM +0100, Niels Möller wrote:
  : > For instance, I have heard reports that when logging in with the
  : > openssh client to a recent lshd, the client will hang after logout.
  : 
  : Yes, I can reproduce this problem. Do other SSH client implementations
  : hang, too?

I can reproduce this with our client and lshd version 1.1.7.

  : > I'll describe the logic that lsh and lshd uses. The basic rules are as
  : > follows:
  : > 
  : > 1. When SSH_MSG_CHANNEL_CLOSE is both sent and received, the channel
  : >    is killed. This is unconditionally required by the spec.
  : 
  : So does OpenSSH.

So does ssh2.

  : > What happens at the client? When the lsh client receives
  : > an SSH_MSG_CHANNEL_REQUEST "exit-status" message, it decides that it's
  : > time to stop talking, and it sends an SSH_MSG_CHANNEL_EOF. Now rule 2
  : > applies, and the channel gets closed properly.
  : 
  : The OpenSSH client considers the "exit-status" message just informational
  : and does _not_ send an EOF message back.

ssh2 doesn't send an EOF message, either.

  : > But appearantly, this logic doesn't work right when talking to the
  : > openssh client. It doesn't send any SSH_MSG_CHANNEL_EOF, and to the
  : > user it appears that the system just hangs.
  : 
  : The OpenSSH server does not wait for the EOF message from the client.
  : However, since the servers' child process is dead, the server decides
  : that it will not write to more data to the 'stdin' handle and does _not_
  : wait for the EOF message from the client.  This way the the channel is
  : correctly closed.

sshd2 sends an SSH_MSG_CHANNEL_CLOSE when the shell exists for session
channels.

Cheers,
-- 
[[email-masked]          --  Sami J. Lehtinen  --           [email-masked]]
[work:+358 9 85657425][gsm:+358 50 5170 258][http://www.iki.fi/~sjl]
[SSH Communications Security Corp               http://www.ssh.com/]

---

From: Tatu Ylonen <[email-masked]>
Date: Sun, 11 Mar 2001 22:41:00 +0200 (EET)

> > most cases, the decision is easy, but sometimes it is not.

The intention here was that the EOF message simply means "I am not going
to send more".  It is similar to TCP half-close.  However,
SSH_MSG_CHANNEL_EOF alone is not intended to initiate closing down the
channel.  However, in some situations it is possible that both sides
determine simulatenously that they will not be sending more.  A side
should initiate closing the channel (by sending SSH_MSG_CHANNEL_CLOSE)
when if it has both sent and received EOF.

It is legal to send other messages, e.g. channel requests, after both
sending and receing EOF.  However, it is illegal to send ANYTHING after
sending SSH_MSG_CHANNEL_CLOSE.

> and does _not_ send an EOF message back.

This is correct behavior.  It is intended as simply a notification.

> wait for the EOF message from the client.  This way the the channel is

It is legal to send SSH_MSG_CHANNEL_CLOSE before receiving/sending EOF (it
is illegal to send anything, including EOF, after sending
SSH_MSG_CHANNEL_CLOSE).  The channel data structures can be freed when
CLOSE has been both sent and received (in either order).

    Tatu

SSH Communications Security           http://www.ssh.com/
SSH IPSEC Toolkit                     http://www.ipsec.com/
SSH Secure Shell                      http://www.ssh.com/products/ssh

---

From: [email-masked] (Niels Möller)
Date: 11 Mar 2001 23:02:12 +0100

> I'm not sure that you can rely on getting an EOF message.

I'm not sure what you're saying here. I'm saying that the "normal"
reason the channel close process is initiated is that CHANNEL_EOF is
both sent and received. There are other, less "normal", reasons. For
instance, if lsh or lshd receives a CHANNEL_CLOSE, that is not
preceded by a CHANNEL_EOF, it displays a warning message "unexpected
channel close", and then it replies with CHANNEL_CLOSE so that the
channel goes away.

> and does _not_ send an EOF message back.

That seems reasonable (as the client can't really know if the server
implements exit-status), but on the other hand that message is the
only reliable hint the client gets that it will not be able to send
any more data.

The alternative is for the server to send CHANNEL_CLOSE after the
exit-status message. If I understand you and Sami correctly, that is
what both the ssh2 and openssh servers do.

But the server can't do that *immediately* after process death,
because then it may lose some of the data on stdout/stderr. Rather,
the rule has to be that the server sends CHANNEL_CLOSE after *all*
these events have occured:

  * Process death
  * EOF on stderr
  * EOF on stdout

Is that what you are doing? I considered doing that in lshd, and the
reason I didn't was the following scenario:

 1. Client sends CHANNEL_EOF

 2. The process' stdin is closed by the server.

 3. The process dies, and its stderr and stdout is closed
    automatically by the system.

 4. The server happens to notice the EOF conditions before it notices
    the process death. So the server sends CHANNEL_EOF.

 5. Now, the "CHANNEL_EOF both sent and received" kicks in, and the
    server sends CHANNEL_CLOSE.

Problem: No exit-status is ever sent (and the client exits with the
"default exit code" of 17, so my lsh testsuite at least has a chance
to notice the problem).

So in addition to the conditions above, one needs a special exception
to the "CHANNEL_EOF both sent and received" rule, which makes things
even more complex. Is this still the Right Way to go?

Suggestions?

/Niels

---

From: Markus Friedl <[email-masked]>
Date: Mon, 12 Mar 2001 00:32:13 +0100

On Sun, Mar 11, 2001 at 10:41:00PM +0200, Tatu Ylonen wrote:
> sending and receing EOF.

Is it legal to send SSH_MSG_CHANNEL_EXTENDED_DATA after a EOF is
sent?  I asume that the EOF refers to both SSH_MSG_CHANNEL_DATA and
SSH_MSG_CHANNEL_EXTENDED_DATA.

> It is legal to send SSH_MSG_CHANNEL_CLOSE before receiving/sending EOF

I think this is a typo. Is it really legal to send
a SSH_MSG_CHANNEL_CLOSE before a SSH_MSG_CHANNEL_EOF is sent?

-markus

---

From: Markus Friedl <[email-masked]>
Date: Mon, 12 Mar 2001 00:41:06 +0100

On Sun, Mar 11, 2001 at 11:02:12PM +0100, Niels Möller wrote:
> channel close",

I don't think this is correct. A CLOSE without an EOF is completely
legal in the SSH 2 protocol.

> and then it replies with CHANNEL_CLOSE so that the
> channel goes away.
> 

When the login shell dies, the OpenSSH server sends
the CLOSE message after:
        1) all data is read from the pipe/socket (that connects the
           server to the login-shell) and
	2) this data is sent to the client.

-m

---

From: Sami Lehtinen <[email-masked]>
Date: Mon, 12 Mar 2001 03:20:12 +0200

Markus Friedl, on March 12. 2001, wrote:
  : > I'm not sure what you're saying here. I'm saying that the "normal"
  : > reason the channel close process is initiated is that CHANNEL_EOF is
  : > both sent and received. There are other, less "normal", reasons. For
  : > instance, if lsh or lshd receives a CHANNEL_CLOSE, that is not
  : > preceded by a CHANNEL_EOF, it displays a warning message "unexpected
  : > channel close",
  : 
  : I don't think this is correct. A CLOSE without an EOF is completely
  : legal in the SSH 2 protocol.

I agree with Markus.

  : When the login shell dies, the OpenSSH server sends
  : the CLOSE message after:
  :         1) all data is read from the pipe/socket (that connects the
  :            server to the login-shell) and
  : 	2) this data is sent to the client.

Ditto in ssh2.

Cheers,

---

From: [email-masked] (Niels M�ller)
Date: 12 Mar 2001 10:23:30 +0100

>   : legal in the SSH 2 protocol.

I understand that it is legal, I just think that it is unusual.

If I change the lsh and lshd behaviour to match ssh and openssh, it
will still not send CLOSE without first sending an EOF. But it will
sometimes send CLOSE without having _received_ any EOF.

/Niels

---

From: Simon Tatham <[email-masked]>
Date: Mon, 12 Mar 2001 09:32:44 +0000

> a SSH_MSG_CHANNEL_CLOSE before a SSH_MSG_CHANNEL_EOF is sent?

I think Tatu meant `without', rather than `before': it's legal to
send SSH_MSG_CHANNEL_CLOSE _without_ having received or sent EOF. It
is not legal to send EOF once you've done that.

Cheers,
Simon
-- 
Simon Tatham         What do we want?        ROT13!
<[email-masked]>   When do we want it?     ABJ!

---

From: Bill Sommerfeld <[email-masked]>
Date: Mon, 12 Mar 2001 12:01:40 -0500

> 	2) this data is sent to the client.

Just to clarify, is this "all immediately available data" (what you
can get from a non-blocking read) or "all data until an EOF indication
is received"?

					- Bill

---

From: Markus Friedl <[email-masked]>
Date: Mon, 12 Mar 2001 18:09:08 +0100

> can get from a non-blocking read) or "all data until an EOF indication

OpenSSH's SSH v2 implementation reads until it gets EOF or an error.

-m

---

From: Tatu Ylonen <[email-masked]>
Date: Wed, 14 Mar 2001 11:32:57 +0200 (EET)

> SSH_MSG_CHANNEL_EXTENDED_DATA.

I actually never thought about this.

I think it should be legal, since extended data is used for things like
stderr.

> a SSH_MSG_CHANNEL_CLOSE before a SSH_MSG_CHANNEL_EOF is sent?

To be more accurate, it is legal to send SSH_MSG_CHANNEL_CLOSE without
sending SSH_MSG_CHANNEL_EOF.  A party MUST NOT send
anything to the channel after it has sent SSH_MSG_CHANNEL_CLOSE.

    Tatu

---

From: Markus Friedl <[email-masked]>
Date: Wed, 14 Mar 2001 10:48:55 +0100

> I think it should be legal, since extended data is used for things like

Hm, so now I'm even more confused.  What does I say if I
send an EOF message on a channel?  Does it mean:

(1) I will not send SSH_MSG_CHANNEL_DATA in the future, or

(2) I will neither send SSH_MSG_CHANNEL_DATA nor
    SSH_MSG_CHANNEL_EXTENDED_DATA in the future.

-m

---

From: [email-masked] (Niels M�ller)
Date: 14 Mar 2001 13:39:56 +0100

>     SSH_MSG_CHANNEL_EXTENDED_DATA in the future.

I have always used the interpretation (2). lsh and lshd currently
ignores extended data received after CHANNEL_EOF, and log a message.
They could just as well treat it as a fatal protocol error.

/Niels