Subject: Re: HTTP2 server-side stream creation

From: Cory Benfield <[email-masked]>
Date: Thu, 9 Jul 2015 18:32:58 +0100

> On 9 Jul 2015, at 17:55, Juan Benet <[email-masked]> wrote:
> 
> From the above, it seems to me that the spec already defines this
> (servers opening regular streams) as valid. This may be a holdover 
> from SPDY's clear separation between the Framing (Sec2), and the 
> "layering of HTTP over SPDY" (Sec3), which scopes all the HTTP-
> specific rules). It does indeed seem a bit inconsistent with our 
> Section 8 [2].

Your analysis here is absolutely right. The HTTP/2 protocol has no limitations in its wire format or state machines that force it to be a client-server model. The vast majority of the client-server requirements really come out of Section 8, excepting a few settings and other parameters that expect there to be a client or server.

> I think the separation of concerns was a useful one. Though maybe
> it does not belong in the HTTP/2 spec given the traditional client-
> server model -- it could very well be a negotiated extension as Cory
> suggests.
> 
> It may also be less work simply extract the HTTP/2 Framing (Section
> 5) and make a symmetric framing muxer based on it. Implementation 
> libraries could then optionally expose the Framing part as a protocol
> that layers, or not.

Right now I think we should pursue the relative simplicity of a negotiated extension. We may find that it’s a bit painful to think of this as HTTP/2 in P2P mode, in which case we can revisit whether this needs to be standardised in a different way.

Cory

---

From: Juan Benet <[email-masked]>
Date: Thu, 09 Jul 2015 16:55:53 +0000

Greetings,

I'm responding too, as I helped trigger the question :).

Cory, that sounds great! In particular, the peer-to-peer use case is an 
important one. There are many systems which do not have a clear 
distinction between a "server" and "client", all "peers" merely have to 
adopt one of those roles because protocols force it (QUIC has a similar
story). It's usually fine, but the roles and limitations can force additional
complexity.

What do you think of these sections of RFC 7540 ?

From Section 5 [0]:
>  o  Streams can be established and used unilaterally or shared by
>      either the client or server.
> ...
>  o  Streams are identified by an integer.  Stream identifiers are
>      assigned to streams by the endpoint initiating the stream.

From Section 5.1 [1]:
> Both endpoints have a subjective view of the state of a stream that
> could be different when frames are in transit.  Endpoints do not
> coordinate the creation of streams; they are created unilaterally by
> either endpoint. 
>
> ...
>
> idle:
> ...
>   * Sending or receiving a HEADERS frame causes the stream to
>      become "open".  The stream identifier is selected as described
>      in Section 5.1.1.  The same HEADERS frame can also cause a
>      stream to immediately become "half-closed".

Either way, whatever seems the best route, David and I are here to
help! :)

Thanks,
Juan

[0]: https://tools.ietf.org/html/rfc7540#section-5 <https://tools.ietf.org/html/rfc7540#section-5>
[1]: https://tools.ietf.org/html/rfc7540#section-5.1 <https://tools.ietf.org/html/rfc7540#section-5.1>
[2]: https://tools.ietf.org/html/rfc7540#section-8 <https://tools.ietf.org/html/rfc7540#section-8>

On Wed, 08 Jul 2015 19:58:31, David Dias <[email-masked] <mailto:[email-masked]>> wrote:

(Replying from my personal email)
Thank you for adding me to the thread. We’ve and seen and worked with several implementations of the SPDY framing layer recently, most notable are muxado[1], yamux[2] and the one we currently use on the IPFS project, spdystream[3]. One of the main reasons behind these efforts is due to the fact that SPDY offers a spec for a stream multiplexing with back pressure and flow control, including priority mechanisms, which makes it extremely interesting  for cross compatibility between several implementations in different languages.
Currently, with have cross compatibility with spdystream (Docker’s Go implementation) and spdy-transport (Fedor’s Node.js implementation) working, both respecting the SPDY framing layer specification, which suits the purpose and as long as the listener is also able to ‘request’ a new stream to open from the ‘dialer', of course, other than a push stream. SPDY, or the HTTP2 framing layer, have a great use case for applications that need a great amount and several types of data to flow.
Right now this solutions exist due to the understanding that the SPDY framing layer can be mounted as a stand alone stream muxer, but if there is a way to make it official that there is a great use case and benefit of having it as its own extension to the protocol, we, Protocol Labs (http://ipn.io/ <http://ipn.io/> <http://ipn.io/ <http://ipn.io/>>), would love to contribute.
Thank you, 
[1] - https://github.com/inconshreveable/muxado <https://github.com/inconshreveable/muxado> <https://github.com/inconshreveable/muxado <https://github.com/inconshreveable/muxado>>
[2] - https://github.com/hashicorp/yamux <https://github.com/hashicorp/yamux> <https://github.com/hashicorp/yamux <https://github.com/hashicorp/yamux>>
[3] - https://github.com/docker/spdystream <https://github.com/docker/spdystream> <https://github.com/docker/spdystream <https://github.com/docker/spdystream>>

David Dias
@daviddias <https://twitter.com/daviddias <https://twitter.com/daviddias>>
> ----------- Forwarded message -----------
> From: [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
> Date: Wed Jul 08 2015 01:18:32 GMT-0700 (PDT)
> Subject: Re: HTTP2 server-side stream creation
> To: [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>, [email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>
> 
> 
>> +David Dias
>> 
>> Thank you very much for a detailed response, Cory!
>> 
>> I think it might be very useful for communication with docker
>> containers, and as far as I know it was one of the applications of
>> SPDY protocol for David.
>> 
>> David,
>> 
>> Maybe you have something to add here?
>> 
>> Thank you!
>> 
>> On Wed, Jul 8, 2015 at 12:49 AM, Cory Benfield <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>> On 7 July 2015 at 23:43, Fedor Indutny <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>> > Obviously, the most straightforward way is to do a PUSH_PROMISE on
>> > existing client-initiated stream, but it appears to me that the
>> > server-initiated streams created using HEADERS frame are valid
>> > too.
>> 
>> From section 8.1 of RFC 7540[0]:
>> 
>> > A client sends an HTTP request on a new stream, using a previously
>> > unused stream identifier (Section 5.1.1).  A server sends an HTTP
>> > response on the same stream as the request.
>> 
>> My reading is that this forbids a 'server' from sending a HEADERS
>> frame first, because servers send responses on already-opened streams.
>> 
>> You could pretty easily construct a semantic for this that essentially
>> turns HTTP/2 into a peer-to-peer communication stream, with both sides
>> of the connection being free to issue requests. This could plausibly
>> be very valuable in systems that use HTTP/2 as an RPC transport. I
>> suspect most clients will currently not allow that behaviour, however,
>> so if you wanted it it might be best to propose it as a negotiated
>> HTTP/2 extension, per section 5.5 of RFC 7540[1]. If you (or anyone
>> else on the list) think this is interesting I'd be happy to co-author
>> a draft to propose it.
>> 
>> Cory
>> 
>> [0]: https://tools.ietf.org/html/rfc7540#section-8.1 <https://tools.ietf.org/html/rfc7540#section-8.1> <https://tools.ietf.org/html/rfc7540#section-8.1 <https://tools.ietf.org/html/rfc7540#section-8.1>>
>> [1]: https://tools.ietf.org/html/rfc7540#section-5.5 <https://tools.ietf.org/html/rfc7540#section-5.5> <https://tools.ietf.org/html/rfc7540#section-5.5 <https://tools.ietf.org/html/rfc7540#section-5.5>>

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 10 Jul 2015 10:13:28 +0200

> Am 09.07.2015 um 19:32 schrieb Cory Benfield <[email-masked]>:

I like extending h2 this way. Not sure about the scope yet.

What would be the minimal behavior a conforming endpoint needs to exhibit? Say a client talks h2 with just this extension. Is there anything the server can assume about the nature of the streams it can open against the client? Does the client need to answer incoming HTTP requests? Against which authority? If left unspecified, how is this extension then used?

Assuming a client would want to implement a "HTTP server" in its side, this would then be an additional extensions with its own settings? That setting would then imply the exact same spec clarifications that this generic one does? How is peer-to-peer then used?

Let's say there are two extensions "HTTP" and "NTP". "HTTP = 1" is implied in RFC 7540 for the server endpoint. Client announces "NTP = 1" in its SETTINGS. Both enable SETTINGS_PEER_TO_PEER. What is the impact?

If I read 2.3, the "server" now needs to expose some sort of behavior to the NTP extension. Basically both endpoints somehow need to mirror each other?

I *do* see the value of the spec in describing "if you do an extension that let's servers open streams to clients, this is what the changes in protocol behavior must conform to". 

Cheers,

  Stefan

<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782

---

From: Cory Benfield <[email-masked]>
Date: Fri, 10 Jul 2015 09:47:27 +0100

> What would be the minimal behavior a conforming endpoint needs to exhibit? Say a client talks h2 with just this extension. Is there anything the server can assume about the nature of the streams it…

I think this is the critical question to answer. I think the server
can assume that the client would answer incoming HTTP requests, but we
definitely have to decide how to signal the client authority.

This cannot be negotiated in the SETTINGS frame because with only 32
bits available for the settings value a client would be pretty limited
in what it can offer. That really leaves us with either using a HTTP
header field, piggybacking on a different frame type, or introducing a
new frame type. I think using a header field for this is a bit
unpleasant (it smacks of the weirdness of Reverse HTTP[0] which I
don't particularly love), though it could work. I don't think there's
a good frame type to piggyback on, so the other option would be to
have a new frame type that is used in this case. That feels like the
best option: the client essentially emits a frame that says "I am
happy to receive requests for authority X".

> If I read 2.3, the "server" now needs to expose some sort of behavior to the NTP extension. Basically both endpoints somehow need to mirror each other?

Yeah, this is a good question. I think it depends on what kind of
extension NTP is.

Extensions like HTTP/2 Encoded Data[1], which advertise a capability
possessed by one of the peers, are unaffected by HTTP/2 P2P. If the
client advertises SETTINGS_ACCEPT_GZIPPED_DATA and
SETTINGS_PEER_TO_PEER, the server may send a request to the client and
use a GZIPPED_DATA frame in the body of that request.

For types of extensions like Alt Svc[2], which talk about 'client' and
'server', the terms vary on a stream-by-stream basis. That means that
if the server peer sends a request to the client in a HTTP/2 P2P
dialog, the client may emit an ALTSVC frame if it so chooses (because
it's technically the server for that stream).

The other type of extension that might exist is one that applies to
the connection as a whole, rather than to individual streams.
Currently Section 2.3. says that both peers should be considered both
servers and clients, but I'm now thinking that that's a bad idea. How
about the following:

> When this extension is deployed with other extensions to HTTP/2, the
> behaviour of this extension does not change. All other extensions
> that refer to 'client' or 'server' MUST be treated as though those
> terms apply on a per-stream basis.
>
> For stream 0, the connecting peer is the 'client', and the accepting peer
> is the 'server'.
>
> If a setting is defined as valid only for servers or only for clients (e.g.
> SETTINGS_ENABLE_PUSH can only be sent by clients), the peer that
> could not normally emit that setting MAY emit it after HTTP/2 P2P has
> been negotiated. Before HTTP/2 P2P has been negotiated, peers MUST
> NOT emit a setting they could not normally emit.

How does that seem?

[0]: http://tools.ietf.org/html/draft-lentczner-rhttp-00
[1]: https://datatracker.ietf.org/doc/draft-kerwin-http2-encoded-data/
[2]: https://datatracker.ietf.org/doc/draft-ietf-httpbis-alt-svc

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 11 Jul 2015 00:06:56 +1200

> servers and clients, but I'm now thinking that that's a bad idea. How

I agree on that being a bad idea. But disagree on your proposed solution...

>> NOT emit a setting they could not normally emit.

I think this extension should mean only that the sender (normally
client) is agreeing to be a server in the context of this whole
connection. The recipient may then perform stream initiation as per the
specifiation for how clients initiate streams to a server.

Streams initiated by either end must still conform to the Stream-ID
allocation behaviour defined for clients and servers. ID selection is
already relative to the emitters placement as client or server (even or
odd numbering). All this extension need do is make the connection
endpoints agree that client/server placement may change per-stream
depending on stream initiator.

Implicitly the initiator of the whole connection is the client by
default and can assume that the other end is a server. Any such flag
coming back from a "real" server on the connection is either not useful,
or equivalent to an ACK for the extension.

A real server that does not support the extension will not initiate
streams to the client other than perhapse PUSH_PROMISE. Thus there is no
actual need to ack the extension. Just dropping it as unknown is a fine
way to disable.

> [2]: https://datatracker.ietf.org/doc/draft-ietf-httpbis-alt-svc

Amos

---

From: Cory Benfield <[email-masked]>
Date: Fri, 10 Jul 2015 13:43:57 +0100

> specifiation for how clients initiate streams to a server.

So it seems like this would be fundamentally different to what I'm
proposing, unless I've misunderstood you. Your proposal is basically
entirely isomorphic to the Reverse HTTP draft, allowing the client and
server to swap over. That still leaves us in the situation where a
server that wants to become a client (to deliver a message) is out of
luck. What it doesn't allow is for both endpoints to be initiating
streams by sending requests simultaneously. Is that understanding
correct?

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 10 Jul 2015 16:12:39 +0200

> Am 10.07.2015 um 10:47 schrieb Cory Benfield <[email-masked]>:
> 

A bit more thinking: this whole server-side creation can only work when
there are *no* concentrating intermediates involved, right? Where would a 
proxy forward the server stream to?

Otherwise clients would need to allocate some sort of name at proxies
and then proxies need to inspect new HEADERs for matching :authority and
it gets only uglier from there...

Or am I missing something?

<green/>bytes GmbH

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 11 Jul 2015 05:57:24 +1200

On 11/07/2015 12:43 a.m., Cory Benfield wrote:
> On 10 July 2015 at 13:06, Amos Jeffries wrote:

I think you have misunderstood me.

The one-way property is only at the outset before the initial TCP
connection client has sent its NTP=1 flag/setting to the server. Once
that arrives at the server both parties are aware that the client is
willing to also be a client+server.

One the server has that awareness, simply sending a request down the
connection to the client is enough to signal its capability of being a
client+server as well.

Then everything is bi-directional.

Lets put it this way:
 If a client has *not* indicated clearly that it is capable of becoming
a server on the connection, how would the original server be able to be
a client ?

The original server needs to assume that all clients are traditional
one-way clients unless they state willingness to be bi-directional /
reverse HTTP / whatever. Anything else leads to breakage.

The original client can safely assume bi-directional servers. It simply
wont get any requests from the ones that actually aren't. Which is fine.

NP: that model could work in both HTTP/2 and HTTP/1.1 if one does not
use long pipelines in the 1.1. Though getting 1.1 software to do it at
all this late stage of HTTP evolution could be difficult.

I am looking at this from a proxy perspective. Our likely use case is
for the "sibling" relationship between proxy caches. Serving users who
deliver a request to one for data which is in the other.

 Today we do this with 2 TCP connections, one for each direction of
flow. Sometimes so much crossover happens that we hit socket limits on
how many inter-proxy TCP connections can be used. It would be a bit more
efficient to be able to bi-directional those connections and reduce the
needed sockets.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 11 Jul 2015 06:05:58 +1200

> proxy forward the server stream to?

Any client who both advertised willingness to be a server and had the
right :authority.

> it gets only uglier from there...

Yep. Certainly gets nasty. Not impossible, just really nasty.

I can forsee some error status being used a lot to reject
server-initiated requests to clients who disappeared already.

Amos

---

From: Cory Benfield <[email-masked]>
Date: Fri, 10 Jul 2015 20:39:30 +0100

> On 10 Jul 2015, at 19:05, Amos Jeffries <[email-masked]> wrote:
> 

So this is a good idea. However, this raises an important question: how do we trust a client that asserts validity for a specific :authority? There’s a risk here of being able to ‘intercept’ messages by asserting that I’m a valid destination for an :authority that some other client is also using.

It might be that we think this is OK: HTTP/2 P2P is likely to be used only in a trusted environment. Alternatively, we could come up with some other requirement: for example, the client might need to provide a TLS client certificate that is valid for the :authority it is asserting.

Ideas would be valuable here.

> server-initiated requests to clients who disappeared already.

This feels like a good addition. Got a name you feel like using?

Cory

---

From: Cory Benfield <[email-masked]>
Date: Fri, 10 Jul 2015 20:41:49 +0100

> On 10 Jul 2015, at 18:57, Amos Jeffries <[email-masked]> wrote:

I suspected as much!

> Then everything is bi-directional.

Right, so the assertion is that this needn’t be a negotiated extension, just an offered one? That feels reasonable to me.

> efficient to be able to bi-directional those connections and reduce the

If you think that proxies can gain benefit from this proposal, I’d love some suggestions for additions to the draft. You know the intermediary case a lot better than I do!

---

From: Stefan Eissing <[email-masked]>
Date: Sat, 11 Jul 2015 10:04:40 +0200

When I first asked people here about server-side streams, the proposed mechanism was that the client opens a "send me push promises on this one and no answer" stream.

Such special streams that do not expect a reply would be suitable to announce service endpoints.

Assume the client wants to serve http. It opens a stream X with special flag and a header like ":service http". The server then opens a stream Y to this service by a push promise on X. 

If the client no longer wants to serve http, it RST_STREAMS X. 

This should make it easy for intermediaries to make informed decisions about routing such streams or even announcing services themselves.

//stefan

> Am 10.07.2015 um 21:41 schrieb Cory Benfield <[email-masked]>:
> 
>

---

From: Cory Benfield <[email-masked]>
Date: Sun, 12 Jul 2015 11:09:34 +0100

> On 11 Jul 2015, at 09:04, Stefan Eissing <[email-masked]> wrote:
> 

I don’t think this model works quite right. The semantic of PUSH_PROMISE is that the server provides both the request and its response. This does not allow the client to serve HTTP because it does not allow the client to generate its responses itself. This model allows for a non-HTTP bi-directional communication stream, but does not move to a fully P2P solution.

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 13 Jul 2015 10:07:58 +0200

> Am 12.07.2015 um 12:09 schrieb Cory Benfield <[email-masked]>:
> 
>> 

> I don’t think this model works quite right. The semantic of PUSH_PROMISE is that the server provides both the request and its response. This does not allow the client to serve HTTP because it does…

Yeah, you probably rigtht. It might bend the normal http/2 model more than necessary.

So, the issue remains with server-initiated streams to define what they exactly connect against. In the case of special data backend server connections, this might be clear by the configuration of it, so outside of protocol context. And for that it is useful, no doubt. For it to work in the wild net, something is missing, I think.

//Stefan

<green/>bytes GmbH

---

From: Cory Benfield <[email-masked]>
Date: Mon, 13 Jul 2015 09:21:50 +0100

> So, the issue remains with server-initiated streams to define what they exactly connect against. In the case of special data backend server connections, this might be clear by the configuration of…

Yeah, I think that's the question: if the client says "Hey, you can
send me requests too, I'm authority foo.bar", what reason does the
server have to trust that statement? It might be possible to do some
fun stuff with TLS client certificates here, but it'd be nice if we
had a plaintext solution too.

Maybe it's enough to say that the server MUST have some out-of-band
reason to believe the client is validly representing that authority,
and suggest some options. Those options could be TLS client certs,
service discovery mechanisms, reverse DNS, etc.