Subject: Re: #227: Encoding advice for new headers and parameters

From: Mark Nottingham <[email-masked]>
Date: Tue, 4 Oct 2016 12:59:07 +1100

> On 29 Sep. 2016, at 11:48 pm, Julian Reschke <[email-masked]> wrote:
> 
> On 2016-09-28 04:29, Mark Nottingham wrote:
>> [ "just me" hat on ]
>> 
>> <https://github.com/httpwg/http-extensions/issues/227>
>> 
>> After some discussion in Berlin and Stockholm, as well as experience with dealing with i18n in parameters for the Link header (see <https://github.com/mnot/I-D/issues/180>), I think we should give more definite advice about when RFC5987(bis) encoding should and should not be used.
>> 
>> In particular, flagging encoding by using a parameter name complicates extension processing (see the issue referenced above), and causes a lot of uncertainty about precedence, etc.
> 
> It complicates processing *slightly*.

I'm not taking about parsing overhead, I'm talking about complication to the model for headers that use it -- as we saw in Link.

> The issue of parameters potentially repeating, and the fact that you need to define what to do in that case, exists in any way. It is inherent in any format that supports name/value pairs.

Yes, but RFC5987 encoding complicates it further, because now you have the possibility of multiple parameters in two different encodings, and potentially different rules / precedences for each encoding.

>> I think it would be much simpler and more reliable to advise people minting new HTTP headers to *not* use RFC5987(bis) encoding, but instead advise that they mandate use of an encoding on the field (or a specified portion thereof).
> 
> RFC 5987 defines a way to deal with non-ASCII. It's not pretty, has a slightly bizarre syntax, but at least it's there, and it has been implemented successfully in all widely deployed user agents.

I agree with all of that.

> Defining *another* way to achieve this seems like a bad idea to me (insert XKCD reference here...).

I didn't say we needed to define another one; I just think we should stop promoting this one.

> (And yes, I'm all for working on a new common field syntax, which, as side effect, addresses non-ASCII, but that's a separate discussion)
> 
>> E.g., if the "foo" parameter on the "bar" header field might need to accept non-ascii content, it MUST be generated with those characters encoded, and MUST be parsed by first decoding that portion of the header.
> 
> ...which essentially *is* the format used RFC 5987, minus parameter naming and preamble.
> 
> Requiring it's use sounds attractive, but I have my doubts that the typical "producer" of field values will get it right; thus we might see "%" characters which are not meant to be percent-escapes in the wild.
> 
> The RFC 5987 format, as ugly it might be, at least has the property that the producer needs to make a conscious decision to choose the format, and thus hopefully will get it right according to spec.
> 
>> The actual encoding to be used need not be specified, ...
> 
> Here I disagree even more. Telling people not to use a standard format, but *not* to tell them what to use instead is strange.
> 
> > ...but the simplest approach would probably be to use RFC3986 %-encoding over a UTF-8 string.
> 
>> A more aggressive approach would be to also recommend that new parameters on existing fields (even if they specify use of 5987) SHOULD use such encoding.
> 
> -1 to mixing different escaping rules in the same field.

I tend to agree, although in the case of Link, it might be best to go that way, since we can't change title*.

>> Thoughts? I'm not going to lie down in the road for this, in that I suspect that most people will gravitate towards this kind of solution naturally, rather than use 5987, but it'd be nice to put clear advice out there.
> 
> I'm opposed to discourage use of RFC 5987 encoding until we have something better to recommend (and that includes a specification for it).

Fair enough. I look forward to bisbis, when we get to do that.

<chair hat> it sounds like we can close this issue with no action -- anyone else have thoughts?

> I'll also point out that in the meantime, at least one more specification uses this format (<https://tools.ietf.org/html/draft-ietf-httpauth-extension-09#section-4>), so if you are serious about discouraging it's use, you really should comment on that spec right now.

*sigh* Another mailing list.

Thanks,

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 4 Oct 2016 12:28:50 +0900

> <chair hat> it sounds like we can close this issue with no action -- anyone else have thoughts?

Well, this is in no way a new thought, but it is worth repeating: What 
about just using UTF-8, plain and simple.

This solution was available almost 20 years ago (see RFC 2277). There 
were certain backwards-compatibility issues, the worst of which 
according to my knowledge would have been that UTF-8 would have shown up 
as double-encoded garbage when interpreted as iso-8859-1 (as defined in 
HTTP at that time).

What I find extremely disappointing is that despite this group being 
composed of a lot of very intelligent people, it was impossible to move 
towards such a simple solution even at a slow pace. The good thing is 
that it's never too late. But that doesn't mean that we have to push 
this solution out another 20 or 50 years.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 4 Oct 2016 07:52:37 +0200

> I'm not taking about parsing overhead, I'm talking about complication to the model for headers that use it -- as we saw in Link.

Yes, that's what I'm talking about as well. Reasoning about

   title=x; title*=UTF-8''y

is not that different than about

   title=x; title=y

> Yes, but RFC5987 encoding complicates it further, because now you have the possibility of multiple parameters in two different encodings, and potentially different rules / precedences for each enco…

I don't see how that makes it significantly harder. Furthermore, the old 
spec already recommends precedence (and that' what is implemented), and 
we can talk about strengthening that requirement.

> I didn't say we needed to define another one; I just think we should stop promoting this one.

Which IMHO leaves spec authors in a worse position.

> ...

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 6 Oct 2016 14:44:10 +0200

> about just using UTF-8, plain and simple.

Well, that's an orthogonal discussion. It hasn't been UTF-8 in the past, 
and APIs assume otherwise, so it's not that simple. But it's certainly 
something we can consider in the future.

FWIW, I tried to better explain this in 
<https://github.com/httpwg/http-extensions/commit/5ec7eed4fa2c941a34e868117ef2fb634d8d506e>.

(We discussed this a bit in Stockholm, and my proposal was just to 
opt-in using a BOM in the first bytes of the field value...).

> this solution out another 20 or 50 years.

I agree that we need a better solution than RFC 5987.

However, the purpose of 5987bis is just to update a specification that's 
already in use.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 11 Oct 2016 16:46:18 +1100

> On 4 Oct. 2016, at 12:59 pm, Mark Nottingham <[email-masked]> wrote:
> 

OK, closing.