Subject: fds beyond 0/1/2

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 10:12:15 -0400 (EDT)

I had occasion to wish that ssh supported running remote commands with
fds other than 0, 1, and 2 connected to useful things.

Has anyone written up a spec for an extension to support this?  I don't
recall seeing any, but that doesn't prove anything.

If not, I've got some ideas I can write up and float for comment.  Or,
if nobody but me cares, I suppose I can just hack something together
and forget the issue. :)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		[email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 11:19:29 -0400 (EDT)

>> I had occasion to wish that ssh supported running remote commands
>> with fds other than 0, 1, and 2 connected to useful things.

>> Has anyone written up a spec for an extension to support this?  I
>> don't recall seeing any, but that doesn't prove anything.

> I haven't seen any.

Then I probably haven't just missed it.

> I guess one ought to use MSG_CHANNEL_EXTENDED_DATA for the actual
> data, but then you need some session handshaking to set up which fd:s
> should be opened and which ids they should use in
> MSG_CHANNEL_EXTENDED_DATA.

Right.  I spent most of my last train trip thinking about this and
blocking out roughly how I'd do it, and what I came up with totally
fits with what you sketch here.

> All fd:s of the channel would then use a single window (or rather,
> one in each direction) for flow control.

Right.

> I take it this could be useful for both directions, although with the
> client in chanrge of setting everything up.

Yes.  So far it seems your thoughts and mine are tracking pretty
closely, so I'm probably not totally off in the weeds.

Hm, this actually raises a potential issue: we really should have
separate back-pressure on each fd, so that failure to consume data from
a particular fd does not wedge the whole channel.

> Do you want to be able to setup bidirectional fd:s as well?

Yes, largely because I see no particular reason to forbid it, and it
can be useful for programmatic use if nothing else.  The protocol I
have blocked out even supports requesting forwarding with data transfer
in neither direction, though I would expect servers to reject such
attempts unless-and-until someone invents a way to make them useful.

>> If not, I've got some ideas I can write up and float for comment.
>> If nobody but me cares, I suppose I can just hack something together
>> and forget the issue. :)
> I'm somewhat interested, but I'm not going to put in any real work
> any time soon.

I'm not asking anyone to.  I'd be interested in any thoughts anyone has
on the subject, but I don't expect anyone but me to be doing any coding
on it.  Given that we've lasted this long without anyone caring enough
to say anything public about it, it's obviously not a high priority for
most people.

I'll send another note to the list in a few minutes with what I have so
far.  It actually includes two extensions which, while strictly
orthogonal to one another, synergize nicely: one for fd forwarding and
one for indicating EOF on a data stream without indicating EOF on the
whole channel.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 11:32:11 -0400 (EDT)

>> I've got some ideas I can write up and float for comment.

Here's what I have.  Comments are invited.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

This file describes the [email-masked] and
[email-masked] requests, and the facilities they are
designed to provide.

[email-masked] is both a global request and a
channel request; [email-masked] is a channel request.
While they are related, in that each is more useful in conjunction with
the other, they are, strictly, orthogonal, so they are described
separately.

For brevity, [email-masked] is abbreviated to
fd-forward below; similarly, [email-masked] is
abbreviated to data-eof.

There are various MOUSSH_* constants used in the descriptions below.
Their values are given in a table at the end of this document.

---------------- [email-masked]

Standard ssh has no way to handle running remote processes with more
than one input channel (stdin, implemented with SSH_MSG_CHANNEL_DATA
from client to server) or more than two output channels (stdout,
implemented with SSH_MSG_CHANNEL_DATA from server to client, and
stderr, implemented with SSH_MSG_CHANNEL_EXTENDED_DATA of type
SSH_EXTENDED_DATA_STDERR from server to client).  This request supports
forwarding other file descriptors in either or both directions.  It is
fundamentally based on a file descriptor model more or less like
POSIX's; non-POSIX OSes will find it difficult to implement to
approximately the degree that they do not match the POSIX file
descriptor model.

As a global request, fd-forward may be sent only by the client.  It is
used to enquire whether the server supports this protocol.  In this
use, it MUST have want-reply set true and MUST have zero bytes of
request-specific data; servers supporting the protocol described here
MUST respond to such a request with with SSH_MSG_REQUEST_SUCCESS and
zero bytes of response-specific data.  Global requests with want-reply
false and/or nonempty request-specific data are reserved for future
specification; servers MUST respond to any such with
SSH_MSG_REQUEST_FAILURE.  Clients MUST respond to any fd-forward global
requests with SSH_MSG_REQUEST_FAILURE, regardless of want-reply and
associated data.

As a channel request, fd-forward may be issued by the client on any
channel of type "session" which has not yet had an "exec" or "shell"
request (or any other requests of similar semantics - collectively,
exec-style requests) made on it.  (moussh implements no exec-style
requests other than standard exec and shell requests as of this
writing.)  It also MUST have want-reply set true.  Using it in
violation of any of these requirements (on a channel of any other type,
on a "session" channel which has had an exec-style request made on it,
or with want-reply false) is reserved for future specification and MUST
draw SSH_MSG_REQUEST_FAILURE.  The server may also issue fd-forward
channel requests, but only as outlined below, in response to
client-issued fd-forward channel requests; other use in the
server->client direction is similarly reserved and likewise MUST elicit
SSH_MSG_REQUEST_FAILURE.  The same is true of requests in either
direction which meet the above criteria but carry request-specific data
(or lack thereof) violating the syntax below.

Data for forwarded fds is carried in SSH_MSG_CHANNEL_EXTENDED_DATA
messages.  The data_type_code values for these messages are selected,
at forwarding setup time, by the sender of the data.  Current
expectation is that they will be selected from the private-use range
0xfe000000-0xffffffff, but this is not a requirement of this
specification; if suitable IETF standardization occurs, they may be
drawn from the 0-0xfdffffff range.  (A descriptor which carries data in
both directions thus has two associated data_type_codes, one for each
direction.)

Forwarding of multiple fds may be requested by sending multiple channel
requests or by sending multiple blobs in a single channel request, or
any combination of the two.  However, requesting forwarding more than
once for a given fd, whether in a single message or in different
messages, is an error.  In this case, the server is permitted to reject
all of the relevant requests; it is also permitted to accept one of
them and reject the rest.  Similarly, requesting multiple forwardings
using identical data_type_code values is an error; at most one of any
such set may be accepted.  In either case, if one is accepted, there is
no requirement which one it is; it will usually be the first, for some
sense of "first" convenient for the implementation, but this is not a
requirement - the only specification here is that at most one request
be accepted for a given fd on a given channel, and at most one for any
given data_type_code on a given channel.  (This applies even if the
requests in question specify identical forwardings.)

The request-specific data in the request MUST consist of

	byte	MOUSSH_FDFWD_MSG_REQUEST

followed by a stream of forwarding request blobs.  Each of these blobs
consists of

	uint32	fd
	byte	flags
	(additional data may follow; see below)

fd is the descriptor to be forwarded.  (This protocol has no support
for descriptor numbers that do not fit in a uint32.)  flags is a
bitmask of flags:

	0x01	Input
	0x02	Output
	0x04	Inessential
	0xf8	Reserved for future specification, must be 0

If the input flag is set, data may flow from client to server.  In this
case, there is one uint32 of additional data after the flags byte,
which is the data_type_code for data on this fd in this direction.  If
the input flag is clear, there is no way for data to flow in the
client-to-server direction on this fd.

If the output flag is set, data may flow from server to client.  The
data_type_code for this data is selected by the server and is returned
in the response (see below).  If the output flag is clear, there is no
way for data to flow in the server-to-client direction on this fd.

If the inessential flag is set, failure to establish this forwarding
may be benign; if clear, it is always fatal.  See below for details.

It is legal (though mostly useless) for a request to carry zero request
blobs, ie, for its request-specific data to contain just the
MOUSSH_FDFWD_MSG_REQUEST byte.  Such a request must be responded to as
usual, with the response carrying zero response blobs.

It is legal for a request blob to specify any of the four possible
combinations of the input and output flags, but, depending on the
facilities available on the server, some of them may cause the
forwarding request to be rejected.  (Given only the facilities
specified here, a forwarding with neither input nor output enabled is
fairly useless, but there's no reason to forbid it, and someone may
find a use for it, or future expansion of this protocol may make such
forwardings useful.)

Because these requests MUST have want-reply set true, they will always
draw an ssh-level response.  If the request conforms to the above
syntax, this MUST be SSH_MSG_CHANNEL_SUCCESS, and the server MUST
generate a channel request in the other direction as described below.
Otherwise, the ssh-level response MUST be SSH_MSG_CHANNEL_FAILURE, and
the server generates no further response and does not process any of
the forwarding request blobs contained in the erroneous message.

The channel request generated by the server in response to a
MOUSSH_FDFWD_MSG_REQUEST request which draws SSH_MSG_CHANNEL_SUCCESS
MUST be sent with want-reply false; its request-specific data contains

	byte	MOUSSH_FDFWD_MSG_RESPONSE

followed by a stream of forwarding response blobs, exactly one for each
request blob in the request; they are in corresponding order (that is,
if request blob X occurs before request blob Y in the
MOUSSH_FDFWD_MSG_REQUEST data, then the response blob for request blob
X must occur before the response blob for request blob Y in the
resulting MOUSSH_FDFWD_MSG_RESPONSE).  MOUSSH_FDFWD_MSG_RESPONSE
messages MUST always occur in the same order as the
MOUSSH_FDFWD_MSG_REQUEST messages they are responses to.  This
MOUSSH_FDFWD_MSG_RESPONSE fd-forward channel request MUST occur after
the SSH_MSG_CHANNEL_SUCCESS generated for the corresponding
MOUSSH_FDFWD_MSG_REQUEST fd-forward channel request.

Each response blob contains either a failure indication, with a reason
message, or a success indication, possibly with a data_type_code
(depending on whether the corresponding request blob had its output bit
set).

If the forwarding request is rejected, the response blob contains

	byte	MOUSSH_FDFWD_REQ_REJECTED
	string	reason

The reason string gives the reason for the rejection, if the server
chooses to provide one; if not, it is zero-length string.  (RFC4251
section 4.5 is relevant to the generation of these strings.)  If the
corresponding request blob's inessential bit was set, this failure may
(but not must) be benign, in which case the server should operate as if
the corresponding forwarding request had not been made.  If the
inessential bit was clear, or it was set but the server chooses to
consider the failure fatal nevertheless, then the server MUST reject
any following exec-style request on this channel.

If the forwarding request is accepted and the corresponding request
blob's output flag bit was clear, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED

If the forwarding request is accepted and the corresponding request
blob's output flag bit was set, the response blob contains

	byte	MOUSSH_FDFWD_REQ_ACCEPTED
	uint32	data_type_code

where the data_type_code will be used for server-to-client data for
this fd.

There are often some conditions which can cause a forwarding to fail
but which cannot be detected until an exec-style request is
attempted.  To deal with these cases, if any fd forwarding requests
were accepted on a channel, then an exec-style request on that channel
MUST cause the server to generate, before its normal response to the
request, one or more fd-forward channel requests, with want-reply set
false, whose request-specific data consists of

	byte	MOUSSH_FDFWD_MSG_STATUS

followed by a stream of zero or more status blobs.  If no fd
forwardings were accepted, the server MAY generate one or more
MOUSSH_FDFWD_MSG_STATUS requests bearing zero status blobs (though, if
the client has not sent any fd-forward requests, the server MUST be
prepared for any such requests to fail, because the client may not
implement this protocol at all).  Clients MUST NOT fail such channel
requests, even if they have not requested any forwardings.  (There's no
real use for such MOUSSH_FDFWD_MSG_STATUS requests from a protocol
standpoint, but implementations may find them convenient.)

Each status blob contains the status for one fd for which a forwarding
request has been accepted; each accepted forwarding must be represented
exactly once in the status blobs sent, though there are no restrictions
on the order in which the status blobs occur, and no restrictions
except message size on the way the collection of status blobs are
grouped into MOUSSH_FDFWD_MSG_STATUS messages.  However, all accepted
forwarding requests' status blobs MUST be sent before the usual
response to the exec-style request occurs, and MOUSSH_FDFWD_MSG_STATUS
requests MUST NOT be sent under any other circumstances (before
receiving an exec-style request or after sending the usual response to
it, or sent by the client under any circumstances).

The status blob for a forwarding which succeeds contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_WORKED

whereas that for a forwading which fails contains

	uint32	fd
	byte	MOUSSH_FDFWD_STAT_FAILED
	string	reason

with the reason string being as described for
MOUSSH_FDFWD_REQ_REJECTED.

Note that if a forawrding request whose inessential bit was clear fails
at this point, the exec-family request MUST fail, even if it otherwise
would have succeeded.

It is not a violation of this spec to use these facilities to request
forwarding for fd numbers 0, 1, or 2, but most servers will reject such
attempts (because they use those fds for the standard protocol's input,
output, and error streams).

This facility has one major known problem: there is only one window for
data per direction, shared by all data streams.  We need a way to
support back-pressure on each forwarded fd independently.  Question:
would it be better to do this within a single channel framework, or
would it be better to open new channels and somehow associate them with
this channel?  I'm leaning towards the former, because we have the same
problem already with stdout and stderr (for example, if you run
"ssh foo producer | consumer" and consumer blocks long enough for pipes
to fill up, an error message from producer on stderr will get blocked
behind the stdout data).

Question: is it reasonable to require generating an
SSH_MSG_REQUEST_FAILURE in response to a request with want-reply set
false?  Perhaps ths spec should specify less about want-reply?  The
standard's text appears somewhat ambiguous.

---------------- [email-masked]

Standard ssh has no way to indicate EOF on one data flow without
indicating EOF on the entire channel.  For input, this is not a
problem, since there is only one data flow, but it is an issue for
output, and, if any input fds are forwarded with fd-forward, it can be
a problem for input too.  We deal with this issue with data-eof channel
requests.

To indicate EOF on the main data stream, the one carried via
SSH_MSG_CHANNEL_DATA, a sender sends a data-eof request whose
request-specific body consists of

	byte	MOUSSH_EOF_MAIN

To indicate EOF on an extended data stream, one carried via
SSH_MSG_CHANNEL_EXTENDED_DATA, the sender sends a data-eof request
whose request-specific body consists of

	byte	MOUSSH_EOF_EXTENDED
	uint32	data_type_code

where data_type_code is, of course, the data_type_code whose extended
data stream is being terminated.  It is a protocol error to send an
SSH_MSG_CHANNEL_DATA message on a channel after sending MOUSSH_EOF_MAIN
on that channel, or SSH_MSG_CHANNEL_EXTENDED_DATA with a particular
data_type_code after sending MOUSSH_EOF_EXTENDED with that
data_type_code.

These are valid with either setting of want-reply, though of course
they will normally be sent with want-reply set true.  If the request is
rejected, it is up to the sender to decide what to do.  In some
circumstances it will be appropriate to ignore the failure, allowing
the data stream to remain in operation as far as the receiver is
concerned; in others, a more appropriate action would be to send
SSH_MSG_CHANNEL_EOF, closing down all data streams in that direction.

Like SSH_MSG_CHANNEL_EOF, MOUSSH_EOF_MAIN and MOUSSH_EOF_EXTENDED do
not require or consume channel window space.  Also like
SSH_MSG_CHANNEL_EOF, indicating EOF on a data stream has no effect on
any data flowing in the other direction.

---------------- MOUSSH_* values

Here are the various MOUSSH_* values.  Note that these sometimes are
distinct even though they don't need to be; for example,
MOUSSH_FDFWD_MSG_* values occur in a different context than
MOUSSH_FDFWD_REQ_* values and thus could reuse numeric values.  This is
done this way largely to improve ease of finding bugs.  But note that
the MOUSSH_FDFWD_* values used by fd-forward messages do overlap with
the MOUSSH_EOF_* values used by data-eof messages; this is because they
are conceptually orthogonal capabilities.

	MOUSSH_FDFWD_MSG_REQUEST	1
	MOUSSH_FDFWD_MSG_RESPONSE	2
	MOUSSH_FDFWD_MSG_STATUS		3
	MOUSSH_FDFWD_REQ_ACCEPTED	4
	MOUSSH_FDFWD_REQ_REJECTED	5
	MOUSSH_FDFWD_STAT_WORKED	6
	MOUSSH_FDFWD_STAT_FAILED	7

	MOUSSH_EOF_MAIN			1
	MOUSSH_EOF_EXTENDED		2

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 25 Aug 2010 10:41:16 -0500

> Here's what I have.  Comments are invited.

I like the idea, though I'd like to see how it'd be used from a typical
Unix command-line.  I'm guessing something like a CLI option to list
file descripto numbers to forward, something like "ssh -r 0,1,2,5,7 ..."

> requests with SSH_MSG_REQUEST_FAILURE, regardless of want-reply and

Why is a global request needed?

> (or lack thereof) violating the syntax below.

I agree with the design based on a channel request.

Nico
--

---

From: [email-masked] (Niels Möller)
Date: Wed, 25 Aug 2010 16:55:24 +0200

I haven't seen any.

I guess one ought to use MSG_CHANNEL_EXTENDED_DATA for the actual data,
but then you need some session handshaking to set up which fd:s should
be opened and which ids they should use in MSG_CHANNEL_EXTENDED_DATA.
All fd:s of the channel would then use a single window (or rather, one
in each direction) for flow control.

I take it this could be useful for both directions, although with the
client in chanrge of setting everything up. Do you want to be able to
setup bidirectional fd:s as well? That's more obscure, at least in the
context of shell scripting.

> if nobody but me cares, I suppose I can just hack something together

I'm somewhat interested, but I'm not going to put in any real work any
time soon.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

---

From: Simon Tatham <[email-masked]>
Date: Wed, 25 Aug 2010 17:27:22 +0100

> a particular fd does not wedge the whole channel.

I would say that this is the biggest issue of all, and should be
carefully considered before even _starting_ to write the detailed
spec and nail down things like the names and formats of new channel
requests.

What are the pros and cons of folding multiple (fd,direction) pairs
into a single window? Why did we ever invent CHANNEL_EXTENDED_DATA
for stderr at all, instead of setting up two channels with separate
windows and linking them together somehow? Only by understanding
that question can we get a clear idea of whether we should extend
the mechanism for further fds or instead use separate channels.

As far as I can see, the key advantage of CHANNEL_EXTENDED_DATA is
that it allows the recipient to preserve the same interleaving of
the stdout and stderr data, in the case where this is perceptible
(because the SSH client's stdout and stderr fds are duplicates of
each other, i.e. target the _same_ terminal device or pipe or file).

However, in practice, this is of limited use to a server on a
POSIXlike platform, because it in turn can't know what the right
interleaving of stdout and stderr was from the point of view of the
application doing the writing. It can only select() between the
output and error channels, and when data becomes available on both
at once it can't know precisely what sequence of writes was
intended. Of course if _it_ folds stdout and stderr into the same fd
(as it does when allocating a pty) then interleaving is perfect -
but then you don't get CHANNEL_EXTENDED_DATA at all.

So it seems to me that the real added value of CHANNEL_EXTENDED_DATA
is only realised in cases where the server diverges from POSIX in
some way. Therefore, a consciously POSIX-oriented extension to
handle multiple fds ought not to be perpetuating the use of the
mechanism.

(A year or two ago I wrote an application which tried to forward an
arbitrary set of file descriptors across a serialisation boundary in
a completely different context from SSH, and what I actually did was
to have the client _work out_ which of its fds were duplicates of
each other and then pass that information to the server. So the
server would allocate a single pipe on its side for both stdout and
stderr, if and only if the client had not redirected them to
different places. Thus, stdout and stderr output were interleaved
exactly as the application intended in cases where you could tell
the difference, and separated correctly otherwise.)

Cheers,
Simon
-- 
Simon Tatham         "I'm going to pull his head off. Ear by ear."
<[email-masked]>                          - a games teacher

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 13:54:17 -0400 (EDT)

> I like the idea, though I'd like to see how it'd be used from a
> typical Unix command-line.  I'm guessing something like a CLI option
> to list file descripto numbers to forward, something like
> "ssh -r 0,1,2,5,7 ..."

Yes, though (because forwarding can go either way) you also need to
specify whether each fd is to be input, output, or both.  (Or neither,
I suppose, if-and-when anyone comes up with a use for that.)

I haven't really thought much about the command-line interface.  That's
an implementation-specific aspect (some implementations don't really
have a command line in the traditional sense, for example), and at this
point I've been thinking more about the protocol aspects.

I'd probably do something like

ssh -fd-in 3,4 -fd-out 7 -fd-both 10,12,13 ...

but this is totally up to each implementation to decide (for those that
choose to implement it at all, of course).

>> As a global request, [...]

It's not really _needed_.  It is designed to allow the client to probe
for the presence of support earlier than the channel request does.  (I
should have stated this; I'll fix that.)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 14:54:43 -0400 (EDT)

>> separate back-pressure on each fd, so that failure to consume data
>> from a particular fd does not wedge the whole channel.

I am inclined to agree.  If I'd thought of the back-pressure issue
before I'd written up the rest of it, I would have - but I didn't think
of that until this conversation.

> windows and linking them together somehow?

Good question.  I don't recall hearing; I certainly wasn't the person
who made that choice. :-)

> that it allows the recipient to preserve the same interleaving of the
> stdout and stderr data, in the case where this is perceptible

After a fashion.  As you point out, this isn't really possible with the
currently-widespread interfaces between sshd and the process being run.

While, as I said, I wasn't around when the decision was made, I suspect
that what happened is something like "oh, we want something separate
for stderr; let's throw in an alternative data mechanism", with the
implications not really thought through - in particular, the effects on
the data window and channel EOF not being given much thought.  For that
matter, while I thought of the EOF issues, I didn't think of the window
issues until this conversation....

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 25 Aug 2010 16:36:15 -0500

> should have stated this; I'll fix that.)

I thought so.  But it can be very simple then.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 25 Aug 2010 16:48:31 -0500

On Wed, Aug 25, 2010 at 02:54:43PM -0400, der Mouse wrote:
> On Wed, 25 Aug 2010 17:27:22 +0100, Simon Tatham wrote:

I think each FD should have its own flow-control.  Just as it works in
any Unix/POSIX OS.

The current channel design has two flows in each direction (data and
"extended" data), with just one flow control window in each direction.

That's... weird.  It works, I guess, but it strikes me as a poor design.

> who made that choice. :-)

I've always wondered myself.  Heck, we could have had a channel for each
direction, such that we'd have needed _three_ linked channels for a
shell session.

My guess is that SSHv1 design and simplicity informed the v2 channel
design.  Or, as der Mouse thinks, maybe it was an [ill] afterthought.

The more "extended" data flows we add, the more likely it is that we'll
have a problem with the single window for all flows in one direction
issue, though I'm not sure how severe any problems will be.

> > each other, i.e. target the _same_ terminal device or pipe or file).

That's true in the pty case (because the stdout/stderr distinction is
lost before the writes even happen as the two fildes will both refer to
the same pty).

It's not necessarily true in the no-pty case.  If the server uses two
distinct pipes for stdout and stderr then the relative ordering of
stdout/stderr writes will be lost.

It's the no-pty case where the distinction between stdout and stderr
becomes useful!  (Because the client can do redirection of the two
separately on the client side.)

Right.

> issues until this conversation....

Yeah, it seems like an afterthought.

Do you really need the multiple fd thing?  Sounds like you might need to
do some violence to the connection protocol...  Well, maybe not so much
violence, just a way to link channels.

Nico
--

---

From: Simon Tatham <[email-masked]>
Date: Wed, 25 Aug 2010 23:12:20 +0100

> stdout/stderr writes will be lost.

Indeed, as I said in my next paragraph. But if the server is running
on some entirely different mechanism which does permit distinguished
stdout and stderr data to be transferred from the server-side
application keeping both interleaving and identification, then the
CHANNEL_EXTENDED_DATA system can preserve that as far as the client.

An obvious example of such a server-side application would be one
that's _part_ of the SSH server process and is explicitly
constructing channel data packets! Some dedicated kind of server -
perhaps one running on a router or other embedded-class device -
is easy to imagine doing such a thing, and if it did, it could
extract some actual value from the EXTENDED_DATA mechanism.

But, as I also said, this is irrelevant to the POSIX world and
should be considered an amusing sidetrack only for these purposes.

> violence, just a way to link channels.

Yes. I think a basically sensible approach would be to simply
introduce a new class of channel, whose CHANNEL_OPEN message cites
(alongside all the normal stuff) the remote id of a "session"
channel currently in its setup phase, and indicates a file
descriptor (with optional direction) to be opened in the session
channel's process when it's launched. So you'd set up a few of
those, then kick off your shell or subsystem or whatever, and you've
got all the separate windows and EOF statuses you need.

(Perhaps it should also be legal to cite fd 2 in such a message, in
which case the server would never send CHANNEL_EXTENDED_DATA on the
session channel, and stderr would come down the auxiliary channel
instead.)

Cheers,
Simon
-- 
Simon Tatham         "_shin_, n. An ingenious device for
<[email-masked]>    finding tables and chairs in the dark."

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 25 Aug 2010 17:30:09 -0500

On Wed, Aug 25, 2010 at 06:13:44PM -0400, Jeffrey Hutzelman wrote:
> --On Wednesday, August 25, 2010 04:48:31 PM -0500 Nicolas Williams
> <[email-masked]> wrote:
> 
> But it's _already_ been lost, because there is no guarantee that the
> server will see those pipes become readable in the order in which
> the app wrote to them; they may even become readable at the same
> time.  [...]

There was never any way to preserve the interleave order of
stdout/stderr writes because if they are the same pipe/whatever then the
order is lost immediately, and if they are not then the order is lost...
immediately.

But I agreed already (maybe the "That's true in the pty case / not
necessarily true in the no-pty case" bit was confusing).  I think we're
just in agreement :)

> The only way to preserve relative ordering of stdout/stderr for the
> server to send them in the right order is for the server and app to
> have a communication channel that is already interleaved, either
> through something more complex than a pair of pipes or because
> they've already been combined into a single stream, as in the pty
> case.

No such thing exists in Unix/POSIX systems.

Attempting to preserve stdout/stderr relative ordering was always a
mistake.  But I don't know that that was ever the goal.  My guess is the
goal was to allow I/O redirection for stdout/stderr separately on the
client side, and the "extended" data thing was just a hack.  Using a
separate "linked" channel would have prompted one to wonder why have
bidirectional channels at all, which might have opened a can of worms.

OTOH, thinking about this now, I suspect linked one-directional channels
could actually simplify the implementations.

> >Do you really need the multiple fd thing?
> 
> It seems like it would be useful.  I've certainly encountered
> situations where it was useful to connect multiple fds between
> processes running on the same machine, and what is ssh if not a way
> to start and communicate with processes running on some other
> machine?

Complex shell scripting scenarios come to mind, yes.

> >Sounds like you might need to
> I don't think it'd be too bad.  I can think of three ways to do this:

Sure.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 25 Aug 2010 18:13:44 -0400

> stdout/stderr writes will be lost.

But it's _already_ been lost, because there is no guarantee that the server 
will see those pipes become readable in the order in which the app wrote to 
them; they may even become readable at the same time.  The app may write to 
stdout, then stderr, then stdout again before the server has read the first 
bit of stdout, so it doesn't even know there was an interleave, let alone 
where it is.

The only way to preserve relative ordering of stdout/stderr for the server 
to send them in the right order is for the server and app to have a 
communication channel that is already interleaved, either through something 
more complex than a pair of pipes or because they've already been combined 
into a single stream, as in the pty case.

> Do you really need the multiple fd thing?

It seems like it would be useful.  I've certainly encountered situations 
where it was useful to connect multiple fds between processes running on 
the same machine, and what is ssh if not a way to start and communicate 
with processes running on some other machine?

1) A channel request on the session channel that results in creation of
   a new channel a la SSH_MSG_CHANNEL_OPEN.

   I don't recommend this; it's a horrible abstraction violation and
   basically requires repeating all of the previously common bits of
   channel setup in a new place.  That'll get particularly annoying
   if we ever add a new variation on SSH_MSG_CHANNEL_OPEN.

2) A new "fd" channel type, whose type-specific channel request data
   includes the channel number of the existing session channel, the
   remote fd number, and the direction information.

3) A new generic "pipe" channel type, which establishes a channel but
   doesn't bind it to anything, combined with a new "dup2" channel
   request on the session channel that specifies the remote fd number,
   direction information, and the existing channel to connect the fd
   to.  The existing channel would have to be one of
   - an unused pipe channel
   - a pipe channel already attached to this session
   - the session channel

   Note that this allows you to attach multiple fds to the same channel,
   including the main session channel.  One could even use this to set
   up a "normal" session with separate flow control for stdout and stderr.

-- Jeff

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 25 Aug 2010 17:43:39 -0400

> a particular fd does not wedge the whole channel.

The way to do that is to use a separate channel, so that you get separate 
flow control, which is what you need to insure this.

-- Jeff

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 21:20:18 -0400 (EDT)

>> It's not really _needed_.  It is designed to allow the client to
>> probe for the presence of support earlier than the channel request
>> does.

It can.  It is.  Describing it takes one whole paragraph.

I'm not sure it will survive the flow-control-driven changes, though.
The flow control issue means the scheme I concocted is probably pretty
much useless in the form I described; see the rest of this thread. :)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Wed, 25 Aug 2010 22:03:50 -0400 (EDT)

Replying to multiple messages, here....

[[email-masked]]
> "extended" data), with just one flow control window in each

No, it has 4294967297 potential flows in each direction (data, and an
int32 worth of different extended data flows).  It has three
standardized data flows (one from the client to server and two from the
server to the client) and up to 67108864 nonstandardized flows
(33554432 each way), plus possible future standardization (or even past
standardization for all I know, though I would have expected any such
to go by on this list).  All this is per channel, of course.

And all the flows in a given direction for a given channel share the
same flow-control window.

Fine for some purposes.  Not a good fit to the generic Unixish model.
Tolerable in the input direction because so far there's only the one
flow.  Tolerable in the output direction largely because there are
few-to-no programs that depend on stderr getting through when stdout is
blocked by back-pressure (and even fewer that do the same with stderr
and stdout switched).

But add arbitrary fd forwarding and it all starts to fall apart.

> The more "extended" data flows we add, the more likely it is that
> we'll have a problem with the single window for all flows in one
> direction issue, though I'm not sure how severe any problems will be.

Agreed.

> Do you really need the multiple fd thing?

Need?  Arguably not.  But then, I arguably don't really _need_ ssh at
all.  Or even computers - it all depends on where you draw the line
between "need" and "want".  Let me put it thus: arbitrary fd forwarding
is valuable enough to me for me to put nontrivial amounts of time into
designing a mechanism to support it and implementing it in moussh.

[[email-masked]]
> I think a basically sensible approach would be to simply introduce a
> new class of channel, whose CHANNEL_OPEN message cites (alongside all
> the normal stuff) the remote id of a "session" channel [...]

[[email-masked]]
> I can think of three ways to do this:

> 1) A channel request on the session channel that results in creation
>    of a new channel a la SSH_MSG_CHANNEL_OPEN.
> 2) A new "fd" channel type, [basically anakin@'s idea above]
> 3) A new generic "pipe" channel type, [generalization of (2)]

There's another way, which nobody seems to have mentioned yet.  I'm not
sure where it fits into my preference list with respect to the others.

It is simple to state: give each data flow within a channel its own
flow-control window.

I can think of some pros and cons, some of which depend on the details.
I'm sure most of you people can too.  I'd probably go with two
completely new window-adjust messages, one for CHANNEL_DATA data and
another, with a data_type_code, for CHANNEL_EXTENDED_DATA data.  I can
imagine various possible ways to handle using this mechanism instead of
the shared window; my inclination after only a few seconds of thinking
is to make it a global request, which, if accepted, causes all data on
the connection which carried the global request to use per-flow
windows (use of standard window adjustments after issuing or accepting
the global request would be a protocol error).

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 26 Aug 2010 01:16:33 -0500

On Wed, Aug 25, 2010 at 10:03:50PM -0400, der Mouse wrote:
> ...

Sure, sure.  A funny nit :)

> Fine for some purposes.  Not a good fit to the generic Unixish model.

Indeed.

> designing a mechanism to support it and implementing it in moussh.

If you implement, then that's good enough for me.  Write up an I-D.

> the global request would be a protocol error).

That will do too.  In retrospect I wish the connection protocol had
always just had linkable uni-directional channels (with a way to create
channel pairs in one go, for things like port forwarding).  I suspect
the channel state machines would have been much simpler that way!  But,
it's too late :(

Nico
--

---

From: Damien Miller <[email-masked]>
Date: Thu, 26 Aug 2010 17:37:59 +1000 (EST)

On Wed, 25 Aug 2010, der Mouse wrote:

(responding to this part until I digest the rest of the proposal)

> a problem for input too.  We deal with this issue with data-eof channel

OpenSSH has had a [email-masked] SSH_MSG_CHANNEL_REQUEST to indicate
the missing half-close state for a while.

Unfortunately, it turns out that quite a few SSH implementations barf
when they see extension channel requests that they don't recognise, so
we turned it off for implementations that don't identify themselves
as OpenSSH. So much for an extensible protocol...

The documentation for this (and other OpenSSH extensions) is at:

http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL?rev=HEAD

-d

---

From: der Mouse <[email-masked]>
Date: Thu, 26 Aug 2010 06:49:03 -0400 (EDT)

>> indicating EOF on the entire channel.  [...]

According to the PROTOCOL document you (later) pointed me at (and thank
you, I've been wanting such a thing for a while), [email-masked] does
not do what my [email-masked] request does.  Rather,
[email-masked] pushes back what I might loosely call SIGPIPE/EPIPE
status, ie, "don't write more to me", and does it for the whole
channel.

data-eof@ is for signaling "I am not going to write anything more", ie,
a traditional EOF.  The difference between it and SSH_MSG_CHANNEL_EOF
is that data-eof@ is for one data flow, not for the whole channel.  (If
there's only one flow in use in that direction on the channel, it is
functionally equivalent to SSH_MSG_CHANNEL_EOF.)

Of course, depending on what ends up happening with the flow-control
discussion, it's possible the use of more than one flow per channel
will pretty much die except for legacy stdout/stderr use, in which case
data-eof@ will become correspondingly pointless.

> http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL?rev=HEAD

That file does not, amusingly enough, seem to include the one
@openssh.com extension I saw in the wild and went looking for
documentation on, [email-masked].

> when they see extension channel requests that they don't recognise,

Heh.  While it currently won't actually offer any functionality (I
don't yet implement any of your extensions), it should be safe to send
such things to moussh; it shouldn't fall over because of a channel
request it doesn't understand - and if it does I consider that a
critical bug.

Not that it really matters, I suppose; you're not likely to see moussh
in the wild very much - I haven't even heard of, much less heard from,
anyone besides me who uses it routinely.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: Damien Miller <[email-masked]>
Date: Thu, 26 Aug 2010 21:54:07 +1000 (EST)

> functionally equivalent to SSH_MSG_CHANNEL_EOF.)

I'm not sure I follow; SSH_MSG_CHANNEL_EOF doesn't shut down data
bidirectionally.  From rfc4254:

> 5.3.  Closing a Channel
>
>   When a party will no longer send more data to a channel, it SHOULD
>   send SSH_MSG_CHANNEL_EOF.
>
>      byte      SSH_MSG_CHANNEL_EOF
>      uint32    recipient channel
>
>   No explicit response is sent to this message.  However, the
>   application may send EOF to whatever is at the other end of the
>   channel.  Note that the channel remains open after this message, and
>   more data may still be sent in the other direction.  This message
>   does not consume window space and can be sent even if no window space
>   is available.

though it does not allow signalling of closure of the extended-data
subchannel - is this what you are talking about?

> documentation on, [email-masked].

that's because youd don't have to actually implement it, just follow the
spec and return SSH_MSG_UNIMPLEMENTED/SSH_MSG_CHANNEL_FAILURE

-d

---

From: [email-masked] (Niels Möller)
Date: Thu, 26 Aug 2010 14:33:40 +0200

> lost before the writes even happen as the two fildes will both refer to

It's possible (and perfectly logical, at least to me) to implement the
pty case on the server side by allocating one pty to use for stdin and
stdout, and a separate pipe for stderr. But when I tried thet, some
shells, notably bash, seemed to get into problems.

Anyway, I don't see a big problem with having the same flow control
window for stdout and stderr. Not sure what to think about the multiple
fd case, since it's unclear to me what the applications are.

Regards,
/Niels

---

From: der Mouse <[email-masked]>
Date: Thu, 26 Aug 2010 08:54:27 -0400 (EDT)

>> [...] [email-masked] [...] [email-masked] [...]
> bidirectionally.

No; it shuts it down in the same direction my data-eof@ does.  But
CHANNEL_EOF shuts down the entire channel.  My data-eof@ shuts down
just one data flow.  If there is only the one data flow (as is the case
for client->server data in the standard protocol), there is no
functional difference, but if there's more than one (as is the case for
server->client, or with forwarded fds if they end up being flows within
the main channel), data-eof@ can shut down one without affecting
others, which CHANNEL_EOF can't.

Yes, there is a need for shutdown capability in the other direction,
the direction eow@ does - but it too arguably should be per-flow rather
than whole-channel.

> that's because youd don't have to actually implement it, just follow
> the spec and return SSH_MSG_UNIMPLEMENTED/SSH_MSG_CHANNEL_FAILURE

Well, it would have been reassuring to find a spec saying so. :)

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Thu, 26 Aug 2010 09:24:34 -0400 (EDT)

> window for stdout and stderr.

Neither do I, but that's because of how they're used, not because
there's no problem in principle.  There are few-to-no programs that
depend on stderr getting through even when stdout is blocked by
back-pressure, and even fewer that depend on stdout working when stderr
is blocked by back-pressure.

> Not sure what to think about the multiple fd case, since it's unclear
> to me what the applications are.

Well, I certainly want it.  Some of the applications I have for it
would break if one flow's blocking wedged other flows in the same
direction.  To pick a simple example (so simple that it probably would
not be done in practice, but it illustrates the risk):

cat file1 | (exec 3<&0; cat file2 | ssh -fd-in 3 remote cmp - /dev/fd/3)

Now, let us suppose that cmp works like

while (there's data available) {
	read from file A
	read from file B
	compare
}

Now suppose that "cat file1" runs first, and fills up pipes and
eventually blocks on back-pressure.  cmp is then sitting there trying
to read "cat file2"'s output, but it can't, because ssh can't send it
because there's no window space.

If the producer programs are more complex than cat, this sort of timing
is not at all implausible - all that's required is that one program
produce enough output to block while the other produces nothing.

In the case of cmp, I probably wouldn't bother running it remotely, but
it could be any consumer; all that's needed is that it be unwilling to
buffer arbitrarily large amounts of output from the fast producer while
waiting for output from the slow producer.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: der Mouse <[email-masked]>
Date: Thu, 26 Aug 2010 09:36:15 -0400 (EDT)

> Why do you need to probe for that early?

Don't really _need_ to.  I just would like to be able to error out of
the client early if the user specifies forwardings and the server
doesn't support them at all.  It certainly would be a fairly small loss
to remove the global-request aspect of it entirely.

> And *if* you decide to do the channel association thing (rather than
> multiple data types on a single channel), consider how that mechanism
> works together with association of x11 channels with the
> corresponding session (lacking in the RFCs, but IIRC you have some
> alternative x11 messages of your own that do set that up this more
> properly).

Right, I do.  (And, actually, the RFC spec does permit it, by
shoehorning the relevant info into the unnecessary X11 cookie data.)

That's an interesting point; X11 connections, and for that matter
forwarded TCP connections, are something approaching prior art here.  I
should try to take what hints I can from them.

> *** Security implications of unexpected fds ***

> [...]

Yes.  Like any facility, this is one of the things that may need to be
restricted or eliminated for limited-access users.  As long as the
protocol doesn't make it difficult or impossible somehow (which I don't
think any of the ideas so far do), this is just a
quality-of-implementation issue.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign

---

From: [email-masked] (Niels Möller)
Date: Thu, 26 Aug 2010 14:54:58 +0200

Some comments, without reading the proposal too carefully:

*** Global probing request ***

> for the presence of support earlier than the channel request does.

Why do you need to probe for that early? If you need that only in obscure
cases, why not just

1. open a session channel,

2. send the channel-request and observe the reply

3. close the session channel again

Although I have difficulty seeing why you'd ever need to do that in
advance of actually trying to set up the real session with the various
extra fd:s.

If you want to go forward, I'd suggest specifying the fd forwarding
features (preferably as minimalistic as possible), and then leave out
"early probing for random features" until some clear need for that is
demonstrated. I think that ought to be an orthogonal issue.

*** Channel association ***

And *if* you decide to do the channel association thing (rather than
multiple data types on a single channel), consider how that mechanism
works together with association of x11 channels with the corresponding
session (lacking in the RFCs, but IIRC you have some alternative x11
messages of your own that do set that up this more properly).

When the ssh server is intended to give only limited access to a user
(only certain subsystems, or only exec requests with certain commands),
I can imagine that the started process might be confused or even
compromized if it is started with random fd's being open. E.g., some
crucial libc functions may fail if no fd:s below 256 are available. So
in this case, the client's ability to request that arbitrary fds be
opened may need to be restricted.

Regards,
/Niels