Subject: HTTP/2 and Constrained Devices

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Mon, 4 Aug 2014 19:46:57 +0000

I would very much like to see HTTP/2 be useful in constrained devices /
embedded applications.  I believe it to be unfortunate that there is
currently a position to bifurcate the "human web" and the "machine web"
(e.g., HTTP/1.1 vs. CoAP).  In my read of -14, I believe (without having
yet implemented) that HTTP/2 is very close to being quite useful to the
constrained devices / IoT world.

Here are a few concerns:

In 4.2, "All implementations MUST be capable of receiving and minimally
processing frames up to 2^14 octets in length, plus the 9 octet frame
header (Section 4.1)."  We could go back and forth on the term "minimally
processing," but a lot of constrained devices are going to have difficulty
buffering frames this large.  Is there a reason we can not lower or remove
this requirement?

In 5.2.1, "The initial value for the flow control window is 65,535 bytes
for both new streams and the overall connection."  Once again, I believe
constrained devices will have difficulty here.  I realize that for this
case, a new setting can be negotiated, but this could potentially occur
after data have been sent (resulting in a negative window size).  Ideally,
we could also have this be <= to the minimum maximum frame size (2^14).

It would also be helpful to try and maintain byte and/or word alignment in
headers.

Finally, I have not yet studied HPACK, but I do applaud the move away from
deflate.  deflate has terrible memory requirements for constrained devices.

For reference, RFC 7228 has some definitions on "constrained devices."  I
would think ~10 KB RAM and ~100 KB flash would be the most constrained
target.

In addition to discussion on the above, I welcome any thoughts on HTTP/2
and constrained devices.

- Robby

Robby Simpson, PhD
System Architect
GE
Digital Energy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Aug 2014 13:37:46 -0700

Hi Robby,

The limits that we have, minimum and default, might be difficult for
severely limited devices, but that is a problem that can be managed by
streaming.  There is nothing inherent in the protocol that requires
buffering of an entire block of anything.  For a constrained device
prior to being able to get settings out, it might be necessary to
discard frames (c.f. REFUSED_STREAM) to avoid having to commit extra
resources.

Note that we can't do things like change the header table size without
affecting other users, like the web, more than we'd like.

As for alignment, I realize that we're not perfect, but the only way
to get alignment is to pad, and no one has been willing to do that.

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Mon, 4 Aug 2014 21:01:19 +0000

Hi Martin,

>discard frames (c.f. REFUSED_STREAM) to avoid having to commit extra

This seems a bit wasteful and could also result in the peer not
understanding the issue.  Let's say I have a constrained device that can
only buffer 4K at a time.  If I receive frames > 4K, there is nothing that
can be done to inform the peer to stop sending frames that large.  At the
end of the day, it seems one must assume the ability to handle a 16K
frame, unless I am missing something.  Granted, I suppose one could use
the TCP receive window, but we have windowing here and I'm guessing that
there will be cases where an entire frame will need to be held before it
can be processed.

In the end, is there a problem with having a minimum < 16K?

>affecting other users, like the web, more than we'd like.

Understood.  I'm not suggesting changing the default HEADER_TABLE_SIZE.
And I certainly understand web is the predominant use case.

>to get alignment is to pad, and no one has been willing to do that.

My first suggestion is to not make the 'E' bit optional in the HEADERS
Frame Payload.

>> we could also have this be <= to the minimum maximum frame size (2^14).

Is it problematic to have the initial value for the flow control window <=
the minimum maximum frame size (whatever it may be)?

---

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Aug 2014 14:39:13 -0700

> there will be cases where an entire frame will need to be held before it

The simplest thing to do if you care about avoiding the waste is to
not buffer.  I'll note that the default TCP CWND is larger than 4k on
most clients, so you are more or less forced to deal with more than
you can hold.

> In the end, is there a problem with having a minimum < 16K?

There are, though I doubt that you'd find the reasons compelling :)
Currently, implementations are not obligated to look for the frame
size limit if they don't intend to use more than 16K.  Lower values
would mean that they need to support the setting.

> And I certainly understand web is the predominant use case.

Well header table is going to be a major source of pain for
constrained devices, hence my mention of it.

> My first suggestion is to not make the 'E' bit optional in the HEADERS

Whether E is present is coupled to whether the priority field is
present.  We do have byte alignment, just not word alignment (whatever
that happens to be on your platform [1]).

> the minimum maximum frame size (whatever it may be)?

This one is a real problem.  This would have a major impact on our
ability to send requests prior to getting settings.  Those first round
trips are crucial for performance.  The stream flow control window
might be reduced without serious impact, I guess, but I'm sure that
you are most concerned about the connection window.  Note also that
headers don't consume flow control credits.

--Martin

[1] https://en.wikipedia.org/wiki/Word_%28computer_architecture%29#Table_of_word_sizes

---

From: Jason Greene <[email-masked]>
Date: Mon, 4 Aug 2014 16:58:57 -0500

> would mean that they need to support the setting.

There are still some things you can do to deal with large frames:

- Advise low header space (using SETTINGS_MAX_HEADER_LIST_SIZE)
- Tune flow control down limiting the size of data frames
- Support > 4k data frames, but process them in smaller chunks
- REFUSED_STREAM and drain all frames > your internal limit, until you get a SETTINGS ACK
- In the case that clients ignore the header advisory after ACK, REFUSED_STREAM or GOAWAY sending a 431

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Mon, 4 Aug 2014 22:05:42 +0000

>most clients, so you are more or less forced to deal with more than

My key concern is not the default, but rather the minimum, when it comes
to MAX_FRAME_SIZE.  I can handle the compromise that the default may be
large and constrained devices may need to deal and then lower it. But with
a minimum of 16K, I fear that a lot of devices may simply be unable to
ever participate (at least reliably).

Also, as an aside, by "waste" I also worry about transmissions (battery
life).

>would mean that they need to support the setting.

Yea, at least not strongly compelling.   :)

Was the number based on the average response size or something?  Some of
those fun heavy tail measurements?

>constrained devices, hence my mention of it.

I agree, but at least it can be lowered, unless I am missing something.

>that happens to be on your platform [1]).

How does one have byte alignment without 'E'?

[Pad Length (8)] + Stream Dependency (31) + [Weight (8)] is my read.

>headers don't consume flow control credits.

OK

- Robby

>--Martin
>
>[1] 
>https://en.wikipedia.org/wiki/Word_%28computer_architecture%29#Table_of_wo
>rd_sizes

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Mon, 4 Aug 2014 22:09:39 +0000

>[Pad Length (8)] + Stream Dependency (31) + [Weight (8)] is my read.

Doh!  Please ignore - I see my mistake now.  My apologies.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 4 Aug 2014 15:31:15 -0700

> those fun heavy tail measurements?

The number was based on experience with SPDY.  I'm not sure that
anyone shared the exact process by which this number was selected.  I
recall it being the smallest size that people could tolerate when we
discussed it at some meeting (which all blur into one, I'm afraid).
There has been no suggestion that the default be reduced further.

A minimum of 256 was floated, but without a great deal of support.

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 5 Aug 2014 09:22:22 +0300

> ever participate (at least reliably).

If one sets compression state size to 0 and disables push (in case of
constrained client), then all frames that don't have a small fixed upper
bound for size can be internally split into smaller subframes.

One then gets pretty much all the problems with CONTINUATION, but one must
be prepared to get those anyway (unless that part of the spec turns out not
to work in practice).

Compression state size of 0 makes compression stateless, so one can just
reset any streams where header decompression fails.

The reason MAX_FRAME_SIZE exists is not that endpoints can't handle 16MB
frames with minimal resources, it is to limit the degree bad endpoints can
screw interactivity by sending overly large frames (not that it is even
useful if using TLS, excluding for HEADERS and PUSH_PROMISE).

> I agree, but at least it can be lowered, unless I am missing something.

Yes, it can be forced to 0 for both directions (receving side using
SETTING(SETTINGS_HEADER_TABLE_SIZE=0), sending side using 0x20 HPACK opcode
(header table size update, Max size = 0)).

> [Pad Length (8)] + Stream Dependency (31) + [Weight (8)] is my read.

Presumably if the E wasn't there, it would be reserved bit.

-Ilari

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Tue, 5 Aug 2014 14:20:17 +0000

>A minimum of 256 was floated, but without a great deal of support.

I'll not suggest to change the default, but I am suggesting that the
minimum be lowered.

Currently, the minimum allowed MAX_FRAME_SIZE is 2^14 (16K). For
constrained devices with ~10KB of RAM (RFC 7228), this effectively means a
maximum-sized frame can not be held in its entirety in RAM.

To my knowledge, the current IETF-imposed maximum on constrained devices
is the IPv6 minimum MTU of 1280 octets, and some constrained devices
struggle there.  But I would suggest somewhere around here for the minimum
allowed MAX_FRAME_SIZE.

AFAIK, HTTP/1.1 did not have any frame size limitations on the low end.

I realize that for the vast majority of web browsing and serving endpoints
out there, these numbers are quite strange.  However, I'd ask the WG to at
least consider this to help enable the use of HTTP/2 for IoT applications.
 Or, let me know if I'm missing something or if there is some tradeoff
that is unbearable.

---

From: David Krauss <[email-masked]>
Date: Tue, 5 Aug 2014 22:45:42 +0800

> maximum-sized frame can not be held in its entirety in RAM.

Note that HTTP/2 frames have no application-level meaning. The frames received are not the same frames that are sent; only the byte payloads are significant. There should be no more reason to attempt to buffer an entire frame than to attempt to buffer an entire message body in HTTP/1.1. (Message body lengths are unbounded.)

Getting a message body which you need to buffer, but cannot, is improved in HTTP/2 vs. HTTP/1.1 and HTTP/2. In HTTP/1.1 you would have to reset the connection, but in HTTP/2 you can reset only the offending stream.

> struggle there.  But I would suggest somewhere around here for the minimum

It’s relevant for IP only when it supports datagram protocols. HTTP/2 is not a datagram protocol.

> AFAIK, HTTP/1.1 did not have any frame size limitations on the low end.

HTTP/1.1 didn’t have any frames. HTTP/2 clients should not expose frames to the application layer, but only fully demultiplexed streams. HTTP/1.1 didn’t have multiplexing either.

A highly constrained machine (only a few kilobytes free) probably won’t support multiple streams at all, and it should set SETTINGS_MAX_CONCURRENT_STREAMS = 1 and then simply strip frame boundaries. Doing anything more with frames known to be on a single stream would be, in fact, an error.

A constrained embedded system would benefit from unlimited-size frames by the reduced frame header overhead. There is no possible downside for a conforming implementation.

> Or, let me know if I'm missing something or if there is some tradeoff

I think you’re not the first embedded engineer to see a framing layer and expect datagrams. The RFC perhaps should clarify this point.

---

From: "Simpson, Robby (GE Energy Management)" <[email-masked]>
Date: Tue, 5 Aug 2014 15:43:12 +0000

Hi David,

Thanks for your thoughtful reply and clarifications.  Some responses
inline below.

On 8/5/14, 10:45 AM, "David Krauss" <[email-masked]> wrote:

>
>On 20140805, at 10:20 PM, Simpson, Robby (GE Energy Management)
>Note that HTTP/2 frames have no application-level meaning. The frames
>received are not the same frames that are sent; only the byte payloads
>are significant. There should be no more reason to attempt to buffer an
>entire frame than to attempt to buffer an entire message body in
>HTTP/1.1. (Message body lengths are unbounded.)
>
>Getting a message body which you need to buffer, but cannot, is improved
>in HTTP/2 vs. HTTP/1.1 and HTTP/2. In HTTP/1.1 you would have to reset
>the connection, but in HTTP/2 you can reset only the offending stream.

I agree with the above.  However, in the instance where there is a message
body that one needs to buffer, but cannot, if the frame size could be
limited, then a more thoughtful response (FRAME_SIZE_ERROR) could be sent
rather than simply a reset.  Or a 413 (see below).

>It¹s relevant for IP only when it supports datagram protocols. HTTP/2 is
>not a datagram protocol.

Agreed. I was merely taking a shot at finding a number.

>HTTP/1.1 didn¹t have any frames. HTTP/2 clients should not expose frames
>to the application layer, but only fully demultiplexed streams. HTTP/1.1
>didn¹t have multiplexing either.
>
>A highly constrained machine (only a few kilobytes free) probably won¹t
>support multiple streams at all, and it should set
>SETTINGS_MAX_CONCURRENT_STREAMS = 1 and then simply strip frame
>boundaries. Doing anything more with frames known to be on a single
>stream would be, in fact, an error.
>
>A constrained embedded system would benefit from unlimited-size frames by
>the reduced frame header overhead. There is no possible downside for a
>conforming implementation.

OK, perhaps here is my misunderstanding. If an endpoint received a message
that was smaller than MAX_FRAME_SIZE, would it still be valid to send 413
(Request Entity Too Large)?

If so, then I think most of my concerns fall away and then the only
considerations are really around DoS attacks and stream blocking when it
comes to frame sizes.

>I think you¹re not the first embedded engineer to see a framing layer and
>expect datagrams. The RFC perhaps should clarify this point.

I was more seeing frames and thinking messages (and therefore thinking all
implementations MUST be able to handle messages of at least 16K).  Thanks
for helping me to better understand!

---

From: David Krauss <[email-masked]>
Date: Wed, 6 Aug 2014 00:01:07 +0800

> (Request Entity Too Large)?

Certainly. Any status code that was valid in HTTP/1.1 is also valid in HTTP/2, per the charter.