Subject: Structured Fields: whitespace in binary content

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Oct 2020 19:23:09 +1100

Structured Fields is in AUTH48 and we've addressed everything that's come up except for one very late entrant. I know this is very last minute, but I'm becoming convinced that this is something we should consider changing before shipping.

Background: I've written a script that validates HTTP messages in RFC XML, including Structured Fields. See:
  https://pypi.org/project/rfc-http-validate/

Applying this to our current drafts, I encountered a problem; if a header field contains binary data, it's extremely likely that it will need to wrap across multiple lines to fit into the RFC. As a reminder, such folded lines are required by HTTP to be replaced by one or more spaces in <https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field-values>.

For example, here is the PR for the Signature draft:
  https://github.com/httpwg/http-extensions/pull/1319

At first I thought this could be addressed by an editorial note explaining that whitespace folding is different in examples. However, things like this make that unworkable:

~~~ http-message
Signature-Input: sig1=(*request-target *created host date
     cache-control x-empty-header x-example); keyid="test-key-a";
     alg=hs2019; created=1402170695; expires=1402170995
 Signature: sig1=:K2qGT5srn2OGbOIDzQ6kYT+ruaycnDAAUpKv+ePFfD0RAxn/1BUe
     Zx/Kdrq32DrfakQ6bPsvB9aqZqognNT6be4olHROIkeV879RrsrObury8L9SCEibe
    oHyqU/yCjphSmEdd7WD+zrchK57quskKwRefy2iEC5S2uAH0EPyOZKWlvbKmKu5q4
    CaB8X/I5/+HLZLGvDiezqi6/7p2Gngf5hwZ0lSdy39vyNMaaAT0tKo6nuVw0S1MVg
    1Q7MpWYZs0soHjttq0uLIA3DIbQfLiIvK6/l0BdWTU7+2uQj7lBkQAsFZHoA96ZZg
    FquQrXRlmYOh+Hx5D9fJkXcXe5tmAg==:
~~~

As you can see, whitespace in folding is semantically significant in Signature-Input (if it's lost, delimitation will be lost too), whereas it needs to be removed for Signature to contain valid binary content.

So, the obvious fix is to allow whitespace inside binary content. Delimitation won't be lost, because it's ":" on both ends. The base64 parsers I checked already swallow whitespace in input (not surprising since the motivating use case for base64 was line-wrapped MIME).

The question is whether it's too late to do this. Personally I think it's worth it; otherwise we're going to have some pretty confusing specs, and that's likely to lead to problems. Also, the delta to the spec and implementations is very small. Also, if there's some implementation lag I think that's workable, because this is less likely to be seen on the wire, and there aren't too many adopters of binary content yet.

What do folks think? I'll start a PR to show what it'd be like, but I wanted to get early impressions ASAP.

Thanks (and sorry for not seeing this earlier),

--
Mark Nottingham   https://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Oct 2020 09:38:40 +0100

Am 28.10.2020 um 09:23 schrieb Mark Nottingham:
> ...
> ...

Go for it.

(Optimally, have a PR *and* apply it to the WG Github repo).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 28 Oct 2020 08:37:33 +0000

--------
Mark Nottingham writes:

> So, the obvious fix is to allow whitespace inside binary content. 
> Delimitation won't be lost, because it's ":" on both ends. The base64 
> parsers I checked already swallow whitespace in input (not surprising 
> since the motivating use case for base64 was line-wrapped MIME).

I dont see any obvious harm, and being able to "lint" IDs and RFCs
seems a worthwhile argument.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 28 Oct 2020 18:42:59 +1000

On Wed, Oct 28, 2020 at 6:28 PM Mark Nottingham <[email-masked]> wrote:

> parsers I checked already swallow whitespace in input (not surprising since
> the motivating use case for base64 was line-wrapped MIME).
>
> The question is whether it's too late to do this. Personally I think it's
> worth it; otherwise we're going to have some pretty confusing specs, and
> that's likely to lead to problems. Also, the delta to the spec and
> implementations is very small. Also, if there's some implementation lag I
> think that's workable, because this is less likely to be seen on the wire,
> and there aren't too many adopters of binary content yet.
>
My instinct is that it's too substantial to skip both last-calls, but I'm
interested to see the diff anyway.

Cheers
-- 
  Matthew Kerwin
  https://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 28 Oct 2020 19:48:42 +1100

Here's the diff:
  https://github.com/httpwg/http-extensions/pull/1322/files

There are other ways to do this, of course, and I suspect that many implementations will just feed the SP into their base64 parser, if they're confident it will handle it well.

Cheers,

> On 28 Oct 2020, at 7:23 pm, Mark Nottingham <[email-masked]> wrote:
>

---

From: Cory Benfield <[email-masked]>
Date: Wed, 28 Oct 2020 10:12:37 +0000

I don't love this change. It's a frustrating additional complexity in
the parser, and it seems unfortunate that we're adding it for reasons
that have nothing to do with whether we want it and everything to do
with limitations of the format we publish the spec in. I think we
shouldn't do it because if we wrote down in the RFC _why_ SP is
allowed, we'd feel fairly silly.

---

From: Ian Clelland <[email-masked]>
Date: Wed, 28 Oct 2020 08:35:44 -0400

On Wed, Oct 28, 2020 at 6:17 AM Cory Benfield <[email-masked]> wrote:

> > There are other ways to do this, of course, and I suspect that many
> implementations will just feed the SP into their base64 parser, if they're
> confident it will handle it well.
>

I just checked, and Chromium's base64 implementation (
https://source.chromium.org/chromium/chromium/src/+/master:third_party/modp_b64/modp_b64.cc;
originally
http://web.archive.org/web/20060617172031/http://modp.com/release/base64/)
does not accept whitespace in base64-encoded data. Not that it's
necessarily a huge engineering task to produce or switch to a decoder which
does, but not trivial either. I'm not as familiar with Gecko source code,
but I did find at least one decoder in that source tree with the same
issues.

> > > Structured Fields is in AUTH48 and we've addressed everything that's
> come up except for one very late entrant. I know this is very last minute,
> but I'm becoming convinced that this is something we should consider
> changing before shipping.
> > >
> > > Applying this to our current drafts, I encountered a problem; if a
> header field contains binary data, it's extremely likely that it will need
> to wrap across multiple lines to fit into the RFC. As a reminder, such
> folded lines are required by HTTP to be replaced by one or more spaces in <
> https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field-values
> >.
> > >
> > > At first I thought this could be addressed by an editorial note
> explaining that whitespace folding is different in examples. However,
> things like this make that unworkable:
> > >
> > > ~~~ http-message
> > > As you can see, whitespace in folding is semantically significant in
> Signature-Input (if it's lost, delimitation will be lost too), whereas it
> needs to be removed for Signature to contain valid binary content.
> > >
> > > The question is whether it's too late to do this. Personally I think
> it's worth it; otherwise we're going to have some pretty confusing specs,
> and that's likely to lead to problems. Also, the delta to the spec and

---

From: Nick Harper <[email-masked]>
Date: Wed, 28 Oct 2020 10:59:36 -0700

On Wed, Oct 28, 2020 at 1:28 AM Mark Nottingham <[email-masked]> wrote:

> Structured Fields is in AUTH48 and we've addressed everything that's come
> up except for one very late entrant. I know this is very last minute, but
> I'm becoming convinced that this is something we should consider changing
> before shipping.
>
> Applying this to our current drafts, I encountered a problem; if a header
> field contains binary data, it's extremely likely that it will need to wrap
> across multiple lines to fit into the RFC. As a reminder, such folded lines
> are required by HTTP to be replaced by one or more spaces in <
> At first I thought this could be addressed by an editorial note explaining
> that whitespace folding is different in examples. However, things like this
> make that unworkable:
>
> ~~~ http-message

Another option is to use a line-folding strategy from RFC 8792 instead of
using the obsolete line folding from section 3.2.4 of RFC 7230.

> wanted to get early impressions ASAP.

I don't think it makes sense to modify the format of structured headers to
be compatible with an obsolete line folding format (which we already have
language that states a sender MUST NOT use that format). Allowing
whitespace only seems to add complexity and ambiguity. The only benefit I
see is for writing documentation where lines need to be wrapped, and there
are better solutions for that, such as using RFC 8792 line folding, an
editorial comment that whitespace folding is different for specific header
fields (e.g. apply different rules to Signature vs Signature-Input), or
tooling detecting that certain header fields use structured headers and
should automatically apply different folding rules for those headers.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 29 Oct 2020 10:45:00 +1100

> On 29 Oct 2020, at 4:59 am, Nick Harper <[email-masked]> wrote:
> 
> Another option is to use a line-folding strategy from RFC 8792 instead of using the obsolete line folding from section 3.2.4 of RFC 7230.

That's a good point; I forgot about that option.

To do this, I think we'd have to say that it uses both obsolete HTTP line folding *and* 8792 folding (which would be applied first), because lots of folks already use obsolete line folding in their specs (and might even want to do so in the same example). 

See:
  https://github.com/httpwg/http-extensions/pull/1319/commits/fe6b4bc5
for an example of what this would look like in a spec.

This is now supported (and the above PR validates) in rfc-http-validate 0.1.3.

Thoughts? I'm particularly interested in the opinions of Annabelle and Justin (CC:ed), since this affects their spec most immediately.

Cheers,

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 28 Oct 2020 18:34:13 -0700

> On Oct 28, 2020, at 1:23 AM, Mark Nottingham <[email-masked]> wrote:
> 

That seems unnecessary to me. Just make the example very long lines
and change the spec rendering tools to render a wrap that fits
within the current view. The tools can provide a notation of
their own to indicate that the folding isn't part of the value.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 29 Oct 2020 13:56:01 +1100

Hey Roy,

Yes, that's effectively what RFC 8792 wrapping does -- see previous message (with proposal).

If folks are OK with that approach I think we can avoid changing Structured Fields.

Cheers,

> On 29 Oct 2020, at 12:34 pm, Roy T. Fielding <[email-masked]> wrote:
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 29 Oct 2020 05:04:40 +0100

> allowed, we'd feel fairly silly.

I agree. We don't adjust on-wire protocols to match a pretty-printer.
And doing just this could easily encourage lazy implementations to
bring line folding back in force after we've deprecated it and added
the "MUST NOT send" rule.

Better adopt RFC8792 as mentioned somewhere else in this thread!
I've seen Mark's patch making use of it and it looks safe and very
clear to me.

Willy