Subject: Empty lists in Structured Headers (#781)

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 May 2019 15:15:18 +1000

(Editor hat on)

<https://github.com/httpwg/http-extensions/issues/781>

PHK and I have discussed this, and I think we agree that this issue should be closed without any change to the specification. 

Any further discussion? We'd like to get this spec shipped.

Thanks,

--
Mark Nottingham   https://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 2 May 2019 08:24:46 +0200

On 02.05.2019 07:15, Mark Nottingham wrote:
> ...

In
<https://greenbytes.de/tech/webdav/draft-ietf-httpbis-header-structure-10.html#specify>,
we currently say:

"Specify the header field’s allowed syntax for values, in terms of the
types described in Section 3, along with their associated semantics.
Syntax definitions are encouraged to use the ABNF rules beginning with
“sh-“ defined in this specification."

Does this mean, that a definition like

   MyField = [ sh-list ]

is an acceptable use of the syntax? (see
<https://github.com/httpwg/http-extensions/issues/781#issue-426418064>).

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 May 2019 16:28:11 +1000

> <https://github.com/httpwg/http-extensions/issues/781#issue-426418064>).

I think it's a *possible* use of the syntax; however, you'd need to accompany it with some prose that directed the parser what to do when SH parsing fails on an empty value. SH pretty strongly steers people away form doing that, so if by "acceptable" you mean "recommended", I think no.

Happy to clarify the text you quote above to make that more clear (we probably need to take another pass at the author recommendations anyway).

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 May 2019 06:32:46 +0000

--------
In message <[email-masked]>, Julian Reschke writes

><https://github.com/httpwg/http-extensions/issues/781#issue-426418064>).

Your entire header would not be SH-parsable/compliant, because it
fails the parsing under SH rules.

However, if for some reason or other you cannot use SH anyway, it is
certainly be a good and sensible way to minimize the amount of work
for people producing a custom header-parser.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 2 May 2019 08:49:13 +0200

> I think it's a *possible* use of the syntax; however, you'd need to accompany it with some prose that directed the parser what to do when SH parsing fails on an empty value. SH pretty strongly stee…

The simplest possible processing model likely would be to parse as
sh-list, and then, upon failure, to inspect the value (after stripping
whitespace it should be empty or only contain commas).

> Happy to clarify the text you quote above to make that more clear (we probably need to take another pass at the author recommendations anyway).

The outcome I'd like to see is: yes, it can be used that way (with care,
see above)

Best regards, Julian

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 2 May 2019 17:34:24 +1000

> I think it's a *possible* use of the syntax; however, you'd need to accompany it with some prose that directed the parser what to do when SH parsing fails on an empty value. SH pretty strongly stee…

By that reasoning, the only "recommended" use of the syntax is
all-or-nothing.  Or rather, you oughtn't use an optional sh-foo unless
its presence is clearly signalled by something else, which you can use
to avoid invoking the generic SH parser in the first place. (Lest you
have to deal with the fallout from whatever "fail parsing" actually
does)

> Happy to clarify the text you quote above to make that more clear (we probably need to take another pass at the author recommendations anyway).

Definitely warranted, I think.

Cheers
-- 
  Matthew Kerwin
  https://matthew.kerwin.net.au/

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 02 May 2019 05:55:30 -0400

On Thu, May 2, 2019, at 15:18, Mark Nottingham wrote:
> PHK and I have discussed this, and I think we agree that this issue 
> should be closed without any change to the specification. 

I agree.  An empty value is a footgun and that's a big part of why we're doing this spec.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 2 May 2019 09:25:39 -0700

> On May 2, 2019, at 2:55 AM, Martin Thomson <[email-masked]> wrote:
> 

In what way is an empty field value a footgun?

The reason why we started doing this spec was to allow field values to encapsulate other
field values in a regular way.  That is, given any arbitrary field definition foo, one could define

    More-Foo: #container{ foo }

and not trip over the syntax in horrible ways. That's why it started as JSON.

Then the spec morphed into a mechanism for self-descriptive binary content and more
efficient date formats that could be mapped back to HTTP/1 in less efficient ways. IOW,
something that might be forward-useful. Unfortunately, that seems to have been abandoned.

Now it is just a set of arbitrary decisions that have nothing to do with HTTP and cannot be
defended without falling back on a straw poll on how tired the editors are of receiving
further comments.

This is not how the IETF is supposed to work on standards track proposals.

....Roy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 May 2019 16:56:39 +0000

--------
In message <[email-masked]>, "Roy T. Fielding" w

>This is not how the IETF is supposed to work on standards track proposals.

I think that era died with Jon Postel and the rubberstamping of H2 ?

I will 100% agree with you that the SH draft is a lot less ambitious
than pretty much anything which have been discussed along the way.

JSON died on the the general distaste for UniCode in HTTP headers,
and the fact that once it had been shoe-horned into HTTP/1 headers
it would no longer be JSON anyway.

Then I tried to synthesize a hierachial format, almost as powerful
as JSON but without those two problems, and that died because the
result did not look like HTTP headers used to look, and some people
were uneasy about the deep/recursive abilities once they started
thinking more about it.

Then we retreated further by restricting the depth to one, hoping
to at least curb the enthusiasm for inventing new syntax in this
space, and through successive cuttings of heels and toes, the SH
draft we have now has resulted.

If you have any ideas how this could have gone better, I'm all ears ?

---

From: Julian Reschke <[email-masked]>
Date: Thu, 2 May 2019 19:09:06 +0200

> it would no longer be JSON anyway.

You keep saying it, but it continues to be incorrect.

I'm in favor of SH in general, but please stick to the facts instead of
making up reasons for how we got here.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 May 2019 17:31:08 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>You keep saying it, but it continues to be incorrect.

You are as entitled to your *opinion* on that matter as I am to mine.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 May 2019 19:41:29 +0200

On Thu, May 02, 2019 at 04:56:39PM +0000, Poul-Henning Kamp wrote:
(...)
> If you have any ideas how this could have gone better, I'm all ears ?

Guys, I think the problem can easily be addressed. I think that many
of us will agree that there are some reasonable limitations that add
simplicity and/or security at the expense of tradeoffs that should
almost affect nobody, such as the limited range of integers.

I think that the opposition against support of empty values only came
from an almost inexistent need at a moment where the spec looks quite
polished, resulting in limited motivation to do some further work on
this. I continue to think it's a bad idea to explicitly exclude them
as the time saved by *not* working on this point is wasted discussing
why it's not being worked on, and will be wasted 100 times more once
the document becomes an RFC, to deal with the ugly workarounds that
people will have invented and/or how to find a safer update to the
spec that remains backwards compatible with deployments.

I don't think we've discussed a *technical* reason for not supporting
these, most only taste based on (possibly inappropriate) proposals
coming out of a few of us' heads. Let's try again to see how that
could be included without breaking things apart nor creating traps
for implementers, and see if the resulting solution looks acceptable
or not. If it does not, at least we'll have a compelling reason
explaining why this is not supported and to encourage users to stop
emitting or dealing with these. Otherwise we'll have a better and
safer coverage, which is always welcome.

Just my two cents,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 May 2019 17:59:40 +0000

--------
In message <[email-masked]>, Willy Tarreau writes:

>from an almost inexistent need at a moment [...]

As far as I could tell, that was a completely nonexistent need, but
to me it matters little if we are trying to generalize from only
one example or from no example at all.

And please remember that SH is a meta-specification, a specification
to write other specifications with.

If a documented use case appears and a concensus for how to meet
its needs can be found, publishing a SHbis will not hurt anybody,
or create any compatibility issues since the specs that references
SHorig are not changed by SHbiss mere existence.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 2 May 2019 20:17:06 +0200

> You are as entitled to your *opinion* on that matter as I am to mine.

Yes, but this is not a question of *opinion*.

If you think that the JSON used in jfiv spec is not JSON, please explain
why that is the case.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 2 May 2019 20:19:34 +0200

> SHorig are not changed by SHbiss mere existence.

Hmm.

So do yo envision implementers to have multiple code paths for SH,
SHbis, SHbisbis???

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 02 May 2019 19:01:09 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>So do yo envision implementers to have multiple code paths for SH,

No.

But if implementers want to support the FooBar: header which requires
SHter, they will have to upgrade their SH parser to SHter.

Doing so will not change the parsing or validation of headers specified
against SH or SHbis.

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 3 May 2019 22:01:03 +1200

> safer coverage, which is always welcome.

Nod. This is the first I recall the topic coming up.

That said. I am aware of several technical reasons for having explicit
handling of empty headers mentioned - at least somewhere centralized.

* empty is not always considered by implementors. That leads to
interoperability differences which can provide privacy leaks for
fingerprinting the HTTP agents themselves.

* assumptions by implementations not expecting to see empty values can
result in security flaws. I have seen certain empty headers crashing
servers and sometimes dynamic sites.

* It is a waste of bandwidth, memory, CPU cycles to any agent receiving
empty headers.

RFC7230 omits field-value from the list of protocol elements that are
called out for special attention to value lengths - though that text
does say the list is incomplete. So this spec about headers or a revised
RFC7230 descendant would seem to be good places to set some formal
handling requirement (ie MUST accept, MUST NOT send [except when
required. eg, Host]).

Amos

---

From: Tommy Pauly <[email-masked]>
Date: Tue, 14 May 2019 10:46:11 -0700

Hello all,

Thanks for the lively discussion we had on this topic! However, it looks like the conversation has petered out. As a working group, we should come to some consensus on the best way forward for this issue.

I'd like to ask everyone to reply with which option they prefer of the of following, so we can get a sense of the group's opinion:

A. Leave the document as is, not defining empty header values for SH (as requested by the editors). As noted on the list, this can allow future revisions to add support.
B. Define empty header values for SH (as the issue requests).
C. Do not allow empty header values for SH, but add formal text to the document explaining how to handle empty values.

Please evaluate these based on what you think will help us converge and ship this document, and note that this is deciding how we define formal Structured Headers, not all or previous HTTP headers.

Best,
Tommy (chair hat on)

> On May 1, 2019, at 10:15 PM, Mark Nottingham <[email-masked]> wrote:
> 
> (Editor hat on)
>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 14 May 2019 20:01:58 +0200

> Please evaluate these based on what you think will help us converge and ship this document, and note that this is deciding how we define formal Structured Headers, not all or previous HTTP headers.

B is my preference. C would be helpful if we can't do B.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 14 May 2019 20:20:22 +0200

Hello Tommy,

On Tue, May 14, 2019 at 10:46:11AM -0700, Tommy Pauly wrote:
> I'd like to ask everyone to reply with which option they prefer of the of
> following, so we can get a sense of the group's opinion:
> 
> Please evaluate these based on what you think will help us converge and ship
> this document, and note that this is deciding how we define formal Structured
> Headers, not all or previous HTTP headers.

I'd prefer B first, then C as a fallback. I'm not strong on this, I just
find it too bad to purposely not support a single feature which already
works even if seldom used. But I won't make a fuss if we don't have it,
this will only leave me with a feeling of half-finished work.

Thanks,
Willy

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 15 May 2019 07:15:33 +1000

> Please evaluate these based on what you think will help us converge and ship this document, and note that this is deciding how we define formal Structured Headers, not all or previous HTTP headers.

Swinging between C and A.

I wouldn't want to attempt B without having a solid understanding of
how it affects other things, like automatic concatenation.  Plus not
including it now leaves a greener field in which to invent something
later (null?).

Cheers

---

From: Jeffrey Yasskin <[email-masked]>
Date: Tue, 14 May 2019 15:28:02 -0700

I have a conditional vote, and sorry for the complexity.

As a less-experienced person, a large part of the value of SH is that
it points me and my team toward best practices. We can say "X is a
structured header" and then only make the mistakes that are possible
within the guard rails.

If empty headers are something you recommend that new headers treat as
errors, I'd vote for A. If you instead recommend that new headers
allow empty values (and treat them either the same or different from
that header being absent), I'd vote for B. I don't think C is a good
idea, since it removes the advice SH could otherwise provide.

Jeffrey

From: Tommy Pauly <[email-masked]>
Date: Tue, May 14, 2019 at 10:49 AM
To: [email-masked] Group
Cc: Poul-Henning Kamp, Mark Nottingham

---

From: Kari Hurtta <[email-masked]>
Date: Wed, 22 May 2019 15:29:01 +0300 (EEST)

> A. Leave the document as is, not defining empty header values for SH
>    (as requested by the editors). As noted on the list, this can allow
>    future revisions to add support.
> C. Do not allow empty header values for SH, but add formal text to the
>   document explaining how to handle empty values.

I can't select any of these.

I think that empty
   sh-list

and similarly
   sh-listlist
   sh-dictionary
   sh-param-list

make sense.

After all also string and binary value can be empty.

   sh-string = DQUOTE *(chr) DQUOTE
   sh-binary = "*" *(base64) "*"

so why not other types also, where some number of elements.
So zero elements looks logical.

This results empty http header field, but there is also precedents
as noted on issue.

When header is empty, parser can not determine that
is that sh-list, sh-listlist, sh-dictionary or sh-param-list
with zero elements.

If one of these are expected for given header field, then
this is not problem.

Logically empty values for
   sh-integer
   sh-float
   sh-token

does not make lot of sense.  So I not have very fan
for null token (value ?N suggested) if it apply
for sh-integer, sh-float or sh-token.

I think there on some cases there is C what is closest.

For some B is closest (allow empty header field value
when expected syntax naturally leads zero elements on 
that case). If several syntax is allowed (ie there is
alternatives), then it using specification need specify
how to process ambiguity.

/ Kari Hurtta