Subject: Sending responses with unknown body size in HTTP/2

From: Lucas Pardue <[email-masked]>
Date: Fri, 26 Jan 2018 18:08:20 +0000

Hello,

I've been thinking about how a server can approach sending a body of unknown total size.

In HTTP/1.1, we had the chunked transfer coding. From RFC 7230 section 4.1:

   The chunked transfer coding wraps the payload body in order to
   transfer it as a series of chunks, each with its own size indicator,
   followed by an OPTIONAL trailer containing header fields.  Chunked
   enables content streams of unknown size to be transferred as a
   sequence of length-delimited buffers, which enables the sender to
   retain connection persistence and the recipient to know when it has
   received the entire message.

In HTTP/2, RFC 7540 section 8.1 says

   HTTP/2 uses DATA frames to carry message payloads.  The "chunked"

   transfer encoding defined in Section 4.1 of [RFC7230]<https://tools.ietf.org/html/rfc7230#section-4.1> MUST NOT be

   used in HTTP/2.

RFC 7540 section 8.1.3 has an example where it shows how an HTTP/1.1 chunked response could look in HTTP/2 frame form.

HTTP/1.1 200 OK                  HEADERS

     Content-Type: image/jpeg   ==>     - END_STREAM

     Transfer-Encoding: chunked         + END_HEADERS

     Trailer: Foo                         :status = 200

                                          content-length = 123

     123                                  content-type = image/jpeg

     {binary data}                        trailer = Foo

     0

     Foo: bar                         DATA

                                        - END_STREAM

                                      {binary data}

                                      HEADERS

                                        + END_STREAM

                                        + END_HEADERS

                                          foo = bar

The HTTP/1.1 response does not contain a Content-Length header field (presumably because the length is unknown). The HTTP/2 format does include a content-length but I'd argue that the example now implies the length is now known before the payload is sent.

Would it be valid here, to omit the content-length in the HTTP/2 HEADERS?

When thinking through this problem a while back, I had thought that in the HTTP/2 case it would be possible to append the content-length as a trailer but I convinced myself that was also invalid due to a number of other HTTP semantic rules.

Kind regards
Lucas

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 27 Jan 2018 10:50:39 +1300

On 27/01/18 07:08, Lucas Pardue wrote:
> Hello,
> 
>  
> 
> I’ve been thinking about how a server can approach sending a body of
> unknown total size.
> 
>  
> 
>    transfer encoding defined in Section 4.1 of [RFC7230]
> <https://tools.ietf.org/html/rfc7230#section-4.1> MUST NOT be
> 
>    used in HTTP/2.
> 

Nod. chunked encoding == DATA frame. Double-encoding is only harmful, so
is prohibited.

> 
> RFC 7540 section 8.1.3 has an example where it shows how an HTTP/1.1
> chunked response could look in HTTP/2 frame form.
> 
>  
> 
> HTTP/1.1 200 OK                  HEADERS
> 
> The HTTP/1.1 response does not contain a Content-Length header field
> (presumably because the length is unknown). The HTTP/2 format does
> include a content-length but I’d argue that the example now implies the
> length is now known before the payload is sent.
> 

Note that Section 8.1.2.6 makes it mandatory for intermediaries to fully
receive these infinite-length streams in their entirety before
forwarding onwards. So that they can meet the "MUST NOT forward"
requirement when Content-Length is present.

> Would it be valid here, to omit the content-length in the HTTP/2 HEADERS?

Valid yes. Section 8.1.2.3 "MAY send a Content-Length header". So the
inverse is also logically true, SHOULD not send one - especially in the
case where length is unknown.

Senders are not obliged to send Content-Length for HTTP/2, they can
simply END_STREAM on a DATA frame.

> 
> When thinking through this problem a while back, I had thought that in
> the HTTP/2 case it would be possible to append the content-length as a
> trailer but I convinced myself that was also invalid due to a number of
> other HTTP semantic rules.

Which ones? IMO it is valid, though only useful for the message
integrity checking.

AYJ

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 27 Jan 2018 00:01:25 +0000

Hi Amos,

Thanks for the reply!

> simply END_STREAM on a DATA frame.

That's what I was thinking, good to hear others interpret the same way.

>Which ones? IMO it is valid, though only useful for the message

Well, RFC7230 section 4.1.2 says:

   A sender MUST NOT generate a trailer that contains a field necessary
   for message framing (e.g., Transfer-Encoding and Content-Length),
   routing (e.g., Host), request modifiers (e.g., controls and
   conditionals in Section 5 of [RFC7231]), authentication (e.g., see
   [RFC7235] and [RFC6265]), response control data (e.g., see Section
   7.1 of [RFC7231]), or determining how to process the payload (e.g.,
   Content-Encoding, Content-Type, Content-Range, and Trailer).

I originally had a design where I sent a 206 Partial Reponse. In the leading HEADERS I had a content-length and in the trailing headers I had a content-range and a signature (draft-cavage-http-signatures-09). However, after rereading the above statement and combining it with RFC section 4.1 I came to the conclusion that not only was that design invalid but that the selection of headers that *are* allowed in a trailer is actually quite small. Especially in HTTP/2 where chunked encoding is not permitted.

I'd love to know if my evaluation was incorrect, which would allow me to return to the original design.

Regards
Lucas

-----------------------------
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in
error, please delete it from your system.
Do not use, copy or disclose the
information in any way nor act in reliance on it and notify the sender
immediately.
Please note that the BBC monitors e-mails
sent or received.
Further communication will signify your consent to
this.
-----------------------------

---

From: Patrick McManus <[email-masked]>
Date: Sat, 27 Jan 2018 14:59:19 +0000 (UTC)

> I’ve been thinking about how a server can approach sending a body of
omit content-length. you're done. For most purposes CL is just a heads up
if you happen to know it (so the peer can do things like reserve disk space
or draw a progress meter, etc..)

(CL never delimits according to 7540, but if its present and it does not
match the size of the aggregated data chunks that is an error according to
8.1.2.6)