Subject: Re: Structured Headers: length limits

From: Martin Thomson <[email-masked]>
Date: Tue, 15 May 2018 08:19:49 +1000

This seems like an improvement.

I think that the underlying concern is that the amount of memory a
recipient has to allocate is essentially unbounded by the underlying
protocol.  In practice however, limits are much tighter.

Looking at the draft, you have changed in the following ways:

1. to note that there is (probably) a limit on the encoded size of header
fields, but not to say what it is.  Here it would be good to establish some
expectations about a size that is "guaranteed" to work.  That might be
unrealistic, but it would be consistent with your other
statements/recommendations regarding minimums.

2. to establish minimums for the size of structures.  This is a fine
addition, because the addition of structure to a header field can add to
the memory commitment a recipient has to make.  Establishing an expectation
that strings can be 1024 octets is good, for example.

You retain the hard upper limit on identifiers though.  Is that really
necessary?  Or is this to preserve interoperability somehow?
On Fri, Apr 20, 2018 at 4:09 PM Mark Nottingham <[email-masked]> wrote:

> Hi all,

> We discussing this issue in London:
>    https://github.com/httpwg/http-extensions/issues/433

> There was a fair amount of discussion on that issue, but it's gone quiet
recently, so I thought I'd bring it up here.

> One thing that Kazuho pointed out was that we might want to consider
limiting the overall size of generated headers, rather than the individual
items.

> That started me to think about what we're trying to do here. It seems to
me that limiting the overall size of generated headers is already done by
most implementations, so we don't need to repeat that (and it could cause
some weird interop interactions). Many Structured Headers libraries aren't
going to have a way to control that (e.g., if they're written in for
PHP/CGI/Node/etc.), since they don't have access to the code that imposes
that limit (which is sometimes imposed on another box, e.g., in a CDN).

> If that's the case, what seems important is assuring that implementations
don't impose their own arbitrary limits that hurt interop. It'd also be
nice to put some limitations in so that a future alternative serialisation
of Structured Headers doesn't have to be able to encode arbitrary-length
structures.

> I've put together a PR that takes the approach of not limiting structure
sizes, but instead specifying a minimum size / enumeration that
implementations are required to support. See:
>    https://github.com/httpwg/http-extensions/pull/578

> Note that numbers and identifiers are still limited; numbers for the
various reasons already discussed, and identifiers so that they can be
reasonably transported / represented in programming languages, etc. (since
they're used as keys for parameters and dictionaries).

> Does this make sense?

> --
> Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 15 May 2018 06:42:33 +0200

> protocol.  In practice however, limits are much tighter.

Definitely!

> statements/recommendations regarding minimums.

In fact to guarantee to work we need to specify more values. For example,
haproxy by default can receive 15 kB of headers (16 kB buffer size minus
1 kB reserved to add extra fields). In practice, I normally set it to
7+1 kB and never face any issue except in places where bogus cookies are
added in loops. So these 7 kB are both a limit on the total header size,
and for each header field, including the request/status line. I remember
Apache 1.3 (didn't check on newer) used to support 100 lines of headers
plus one request/status line, each line up to 8 kB. So here already we
see a very different model : by default Apache supports shorter header
fields but the sum can be larger. If we take the intersection between
the two, we end up at ~8 kB total and ~8 kB per header field (which is
why I set haproxy to 8kB since Apache is everywhere and more or less
defines what must be supported).

I think that in order to set realistic limits we need to take in
consideration the allocated resources on the recipient because ultimately
this is what drives implementations to set hard limits. I really like what
was done for HPACK, consisting in counting the size adding some overhead
for internal uses and counting the total size. The simple fact that it is
not easy to determine on the sender is a good thing, because it encourages
ignorant senders to stay away from the limit. For example if we say that
no single header line may be longer than 8 kB and the total header size
cannot be longer than 16 kB, including a few bytes per line and a few
extra bytes total, it will become clear that it's pointless to try to
serialize exactly 8kB on a single line.

Also, haproxy sets a limit on the header field name's length. I don't
remember what it is, probably something around 63 or 255 characters,
depending on the number of bits used for this in the index. In practice
this limit was never met, but it definitely is an implementation specific
limit. I remember seeing some CGIs fail a long time ago because header
fields were turned into environment variables whose name was too long
for example!

> that strings can be 1024 octets is good, for example.

In fact, one thing that made HTTP succeed is that everyone has its own
limits depending on the local requirements. Maybe we could define a few
"profiles" with a set of reasonable values. HTTP messages observed in
enterprise environments are much larger than the ones you see at home,
with many large cookies and authentication tokens for example. Those
you see from your browser when connecting to the net are much larger
than the ones used my scripts, bots or IoT devices, where storing 1kB
for a string might already appear huge. Maybe 3 profiles would be
enough : application, internet, iot.

Just my two cents,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 15 May 2018 15:16:44 +1000

> On 15 May 2018, at 8:19 am, Martin Thomson <[email-masked]> wrote:
> 

Yes. Overall, I think that's probably better done in draft-ietf-httpbis-semantics, since these limits are often imposed by components that aren't under the control of a SH implementation (e.g., a Web server, proxy server, or browser), and since it'd affect all headers, not just structured ones.

That said, SH could explain the situation better. I'll modify the PR.

> necessary?  Or is this to preserve interoperability somehow?

I struggled with this; since it's used for dict and parameter names, it seemed reasonable to be a bit more constraining here. Happy to open it up if that causes discomfort.

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Wed, 16 May 2018 10:40:44 +1000

> I struggled with this; since it's used for dict and parameter names, it
seemed reasonable to be a bit more constraining here. Happy to open it up
if that causes discomfort.

Yeah, not sure if it needs action, but it stands out a little.  There's two
competing concerns, in my view.  If these identifiers are just implemented
as pointers to the receive buffer, there is no real cost in allowing longer
values.  The concerns about string encoding don't really apply given the
restrictive grammar you define.  On the other hand, it is helpful to have
some constraints on identifiers.  But perhaps the most effective constraint
we have is that people don't like insanely long identifiers for many other
reasons.

I'm leaning slightly toward removing the cap here too, and establishing a
lower expectation for minimum supported length (say 64 octets).

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 16 May 2018 13:09:17 +1000

> On 16 May 2018, at 10:40 am, Martin Thomson <[email-masked]> wrote:
> 

SGTM; I've pushed a change to the PR:
  https://github.com/httpwg/http-extensions/commit/1745c45107e22

I think we're converging here, so I'm going to merge soon unless I hear otherwise.

Cheers,