Subject: Re: New Version Notification for draft-kamp-httpbis-structure-01.txt (fwd)

From: Julian Reschke <[email-masked]>
Date: Fri, 11 Nov 2016 13:45:44 +0100

On 2016-10-30 19:59, Poul-Henning Kamp wrote:
> Updated in preparation for WG discussion in Seoul.
>
> Minor changes only.
>
> Github repo: 	https://github.com/bsdphk/id_common_structure

Thanks!

Find below some (unstructured) feedback:

    For textual formats, such as JSON, the format must first be neutered
    to not violate field-value's ABNF, and then workarounds added to
    reintroduce the features just lost, for instance UNICODE strings, and
    suddenly it is no longer JSON anymore.

That's misleading.

a) yes, it needs to be "neutered", but that's a simple character 
replacement operation, so it can be run on an existing JSON text.

b) the features that are lost are essentially line breaks and raw 
non-ASCII characters; however, the transmitted string would *still* be JSON.

        ascii_string = * %x20-7e
                # This is a "safe" string in the sense that it
                # contains no control characters or multi-byte
                # sequences.  If that is not fancy enough, use
                # unicode_string.

        unicode_string = * unicode_codepoint
                # XXX: Is there a place to import this from ?
                # Unrestricted unicode, because there is no sane
                # way to restrict or otherwise make unicode "safe".

It's not clear why there's even a distinction...

Also, it needs to be stated whether the grammar is octet or character 
based. For an abstract datamodel, the latter probably makes more sense.

        h1_common-structure-header =
                ( field-name ":" OWS ">" h1_common_structure "<" )
                        # Self-identifying HTTP headers
                ( field-name ":" OWS h1_common_structure ) /
                        # legacy HTTP headers on white-list, see {{iana}}

Do not mix message block ABNF with field value ABNF. Just define what's 
inside the field value.

        h1_element = identifier * (";" identifier ["=" h1_value])

Shouldn't the second "identifier" be "token"?

        h1_ascii_string = DQUOTE *(
                        ( "\" DQUOTE ) /
                        ( "\" "\" ) /
                        0x20-21 /
                        0x23-5B /
                        0x5D-7E
                        ) DQUOTE
                # This is a proper subset of h1_unicode_string
                # NB only allowed backslash escapes are \" and \\

        h1_unicode_string = DQUOTE *(
                        ( "\" DQUOTE )
                        ( "\" "\" ) /
                        ( "\" "u" 4*HEXDIG ) /
                        0x20-21 /
                        0x23-5B /
                        0x5D-7E /
                        UTF8-2 /
                        UTF8-3 /
                        UTF8-4
                        ) DQUOTE
                # This is UTF8 with HTTP1 unfriendly codepoints
                # (00-1f, 7f) neutered with \uXXXX escapes.

How would a generic recipient decide whether it needs to handle "\u"? 
What's the point of having different ABNF productions?

Also: this puts raw non-ASCII UTF-8 in the string value. It's not clear 
that this is a good idea for HTTP/1, but if it is, it could be done in 
JSON as well, which would eliminate one of the counter arguments in the 
Introduction.

        h1_common_structure = ">" h1_common_structure "<"

That's a bit too recursive (speaking of which: "_" isn't allowed in ABNF 
names)

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 11 Nov 2016 13:42:54 +0000

--------

>It's not clear why there's even a distinction...

To give designers of HTTP headers a trivial way to define strings
which are "safe" and free from needless complexity vs. strings where
the recipient should be prepared to deal with BOM, RLM and "MAN
IN BUSINESS SUIT LEVITATING".

>based. For an abstract datamodel, the latter probably makes more sense.

The abstract data model is abstract, so it is obviously neither.

For the h1 serialization, I don't see how it makes a difference, unless
somebody is running HTTP/1 in EBCDIC or Morse-code ?

>Shouldn't the second "identifier" be "token"?

yes, probably.

>What's the point of having different ABNF productions?

Based on the definition/data-dictionary of the header in question.

Remember:  This is only the data-model/h1-serialization, for each
HTTP header, it will (still) be necessary to define what the data
is/can be.

>that this is a good idea for HTTP/1, 

Neither is it obvious that it is going to cause any problems.

For reasons of transmission efficiency, I'm not keen on mandating
\uXXXX for all non-ascii unicode unless experimentation on the live
indicates that we have to, or if we decide for reasons of purity
that HTTP/1 can never have the high bit set.

Either way, another good reason to keep the "safe" string type.

>That's a bit too recursive

No, that is deliberately making recursion possible, as the simplest
possible way to define complex datastructures.

>(speaking of which: "_" isn't allowed in ABNF names)

Noted.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 17 Nov 2016 10:06:04 +0900

Hi,

Thank you for writing the draft.

Regarding the numbers, could we either exclude floating point from the
specification or state that an integral number MUST be encoded without
using a dot?

The reason I ask is because it is hard to correctly implement a parser
for floating point numbers, and a bug in the parser would likely lead
to a vulnerability [1]. Note that in some (if not most) of the
programming languages you would need to implement your own number
parser to meet the needs. For example, you cannot use sscanf in C,
because depending on the locale the function allows use of decimal
points other than '.'.

If we could exclude floating point numbers from the specification
entirely or have a restriction something like above, parser
implementors can refrain from implementing their own floating point
number parsers until the specification in which they are interested in
actually start using the notation.

Non-integral numbers are rarely used in the HTTP headers. The only one
I can recall is the q value of Accept-Encoding, but it is not a
floating-point but actually a fixed-point number (of three decimals
below the point), which could have been represented by using integral
numbers between 0 to 1000.

     weight = OWS ";" OWS "q=" qvalue
     qvalue = ( "0" [ "." 0*3DIGIT ] )
            / ( "1" [ "." 0*3("0") ] )

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0154.html

2016-10-31 3:59 GMT+09:00 Poul-Henning Kamp <[email-masked]>:
> ------- Forwarded Message
>
> A new version of I-D, draft-kamp-httpbis-structure-01.txt
> has been successfully submitted by Poul-Henning Kamp and posted to the
> IETF repository.
>
> Name:           draft-kamp-httpbis-structure
> Revision:       01
> Title:          HTTP header common structure
> Document date:  2016-10-30
> Group:          Individual Submission
> Pages:          13
> URL:            https://www.ietf.org/internet-drafts/draft-kamp-httpbis-structure-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-kamp-httpbis-structure/
> Htmlized:       https://tools.ietf.org/html/draft-kamp-httpbis-structure-01
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-kamp-httpbis-structure-01
>
> Abstract:
>   An abstract data model for HTTP headers, "Common Structure", and a
>   HTTP/1 serialization of it, generalized from current HTTP headers.
>
> ------- End of Forwarded Message
>
>
> --
> Never attribute to malice what can adequately be explained by incompetence..
>
>
>

-- 
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 17 Nov 2016 06:53:45 +0100

Hi Kazuho,

>      qvalue = ( "0" [ "." 0*3DIGIT ] )

I'd like to avoid FP as well. However it's important to note that fixed
point numbers is not exempt from similar issues due to the way they are
encoded, since everyone will store them in floats/doubles, but the error
is limited to the mantissa precision. For example 64-bit double numbers
contain a 53 bit mantissa so we can easily see a difference in the lower
bits. Example :

  #include <stdio.h>
  #include <stdlib.h>

  int main(int argc, char **argv)
  {
        double f = atof(argv[1]);
        printf("input=%s float=%f\n", argv[1], f);
        return 0;
  }

  $ ./a.out $((1<<32)).000001
  input=4294967296.000001 float=4294967296.000001
  $ ./a.out $((1<<33)).000001
  input=8589934592.000001 float=8589934592.000002

In my opinion we don't care here. And maybe we can document the expected
minimal precision (eg: minimum 53 bits to be able to store a 32-bit
integral range with a 1/1000000 fractional precision.

Also it's pretty certain that developers will use atof() on fixed point
numbers, but at least the input can be sanitized easily by ensuring that
only digits, dot and - are allowed in it.

Regards,
Willy

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 17 Nov 2016 16:17:07 +0900

Hi,

2016-11-17 14:53 GMT+09:00 Willy Tarreau <[email-masked]>:
> encoded, since everyone will store them in floats/doubles,

Yes. Therefore, I believe that the we should discourage people from
using fixed point numbers.

For example, if we want to define a quality value like the one found
in Accept-Encoding, we should not use three-digit fixed point numbers,
but instead use an integral value between 0 to 1000.

That way, we can totally avoid the issues introduced by a dot within a number.

> but the error

-- 
Kazuho Oku

---

From: Julian Reschke <[email-masked]>
Date: Thu, 17 Nov 2016 09:01:35 +0100

On 2016-11-17 08:17, Kazuho Oku wrote:
> ...

+1 in general, but that approach wouldn't work well in cases where we 
want to allow different levels of precision (such as timestamps that 
optionally would allow fractions of seconds)

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 08:25:52 +0000

--------

>integral range with a 1/1000000 fractional precision.

Uhm, guys:

       number = ["-"] 1*15 DIGIT
               # XXX: Not sure how to do this in ABNF:
               # XXX: A single "." allowed between any two digits
               # The range is limited is to ensure it can be
               # correctly represented in IEEE754 64 bit
               # binary floating point format.

Please notice:

	1) If there is a period, it _has_ to be between to digits.

	2) max 15 digits in total, so that people stuffing them in
	   ieee64bin aka double, don't get surprises.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 08:34:36 +0000

--------
In message <[email-masked]>

>using fixed point numbers.

That ship has already sailed since q=0.5 does not use integers.

But please note that this is a generic *data model*, and therefore
it should "deliver tools, not policies".

By this I mean that people may legitimately want to move non-integers,
deep structures and unicode strings, and it would be counterproductive
to force them to invent a parallel format to do that.

However, each individual header will have to define which _actual_
data you are allowed to put in it, and a general antipathy against
non-integers is probably wise there.

Summary: So if you don't like non-integers, don't use them in the
HTTP-headers you invent, but don't tell other people what data
types they are allowed to use.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 17 Nov 2016 10:01:10 +0100

On Thu, Nov 17, 2016 at 09:01:35AM +0100, Julian Reschke wrote:
> +1 in general, but that approach wouldn't work well in cases where we want
> to allow different levels of precision (such as timestamps that optionally
> would allow fractions of seconds)

Sure but that's one example where we know the 53-bit mantissa already fails
to represent sub-microsecond when the seconds represent the unix time since
the epoch, so a struct timespec is misrepresented there. On the other hand
most of the time we don't care about the sub-microsecond accuracy, it just has
to be known (hence documented) that writing timestamp 1479372852.095647253 to
represent what I'm seeing now may very well end up being read as
1479372852.095648125 for example.

Another option could be to state that some numbers can include a fractional
part and that this part has to be processed separately *if required*. That's
already OK for q= after all, which for many cases doesn't require accurate
processing :
  q=0           => always 0
  q=0.something => always >0 and <1
  q=1           => always 1

Or maybe Poul-Henning's 15-digit number can solve it as well. But we must not
use it for any number as it would remove the ability to pass 64-bit integers.

Regards,
Willy

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 17 Nov 2016 18:11:51 +0900

Hi,

Thank you for the response. I totally support your intention to
deliver tools, not policies.

My point is that the draft should not “enforce a policy” to implement
floating point parsers even in case you need to only handle integers.

For example, consider the case in which I am writing a draft that
passes an integer using Common Structure.

Since Common Structure allows an integral number to be sent with a
dot, a sender is permitted to send 100 as “100.00”, which means that
you would need to decode a floating point number even in the case
where you only need to deal with integers.

My understanding is that most (if not all) headers that will use
Common Structure would need to encode floating point numbers. And
therefore I am looking for a clause that makes decoders without a
floating point parser to be comformant to the specification (until
actually facing a need to decode floating points).

-- 
Kazuho Oku

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 09:48:16 +0000

--------
>On Thu, Nov 17, 2016 at 09:01:35AM +0100, Julian Reschke wrote:

(Sorry for going time-nut on you guys...)

>the epoch, so a struct timespec is misrepresented there.

Nobody outside a small group institutions, of which I am the founding
member[1], are able to generate such a timestamp with any level of
credibility as to its truthfullness.

>On the other hand most of the time we don't care about the sub-microsecond
>accuracy,

Trust me:  We will never care about *timestamps* with sub-microsecond
resolution in HTTP headers. They barely make sense in the first
place and transmission noise and and delays will totally swamp any
information they might carry.

We might care about time *intervals* with sub-nanosecond resolution,
for instance how long it took to find something in memory.  Eleven
digits behind the decimal point is all you need for that[2] and
they should be stacked on top of a time_t offset of much lower
precision.

>part and that this part has to be processed separately *if required*. 

We are trying to *communicate* the number.  How people decide to
process the number on their own computer is not for us to decide [3].

>use it for any number as it would remove the ability to pass 64-bit integers.

15 digits is enough for a Content-Length of a petabyte, I don't think
anybody, now or in the future, will find that a sensible thing to do.

If you want to move cryptographic bits, there is a dedicated "blob"
datatype for that.

Poul-Henning

[1] http://phk.freebsd.dk/pubs/timecounter.pdf

[2] This is phyics:  You have to get a certain number of electrons
    through an electrical circuit before the voltage crosses your
    triggering threshold, and they are swimming in thermal noise.

[3] But the 15 digit limitation is there to make it easier for them.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 17 Nov 2016 11:17:27 +0100

> information they might carry.

That's inexact, I've already seen people using timestamps as an easy
way to have unique IDs. Of course that's wrong but when the principle
is that once you're cycle-accurate you know you can't do two things
at once, it works as long as you don't have time jumps.

Note I'm *not* defending this misuse and not even advocating for
supporting it. I'm just saying that it's untrue that nobody cares.

> anybody, now or in the future, will find that a sensible thing to do.

Well it's only one day of traffic at 100 Gbps, it could conceivably be
used after the next decade for file system replication over HTTP. And
if you manage a flat filesystem, you possibly at least want to be able
to use more than this in the Range header to seek on your storage.
There's nothing wrong with asking for one GB of data starting at 1.23 PB.
Multi-petabyte filesystem are still not common but I do know people who
use some already, especially in web hosting (and I'm pretty sure you
know some as well). We clearly see that the boundary between web and
storage sometimes is very thin and we must not prevent solutions from
emerging just because of protocol limitations.

> If you want to move cryptographic bits, there is a dedicated "blob"

Agreed.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 10:21:01 +0000

--------
In message <[email-masked]>
, Kazuho Oku writes:

>dot, a sender is permitted to send 100 as '100.00'

So this is clearly something the draft needs to explain better.

CS does not decide what values are legal and which are not, it only
helps you transport values safely across the wire.

When you define your header you define the kind of values it accepts,
and if you say that "value X is an integer", it should be pretty
obvious to anybody that sending it as "100.0" is not OK.

If you think people will not get that fine detail right, you can
add text to your header definition which drives the point home.

>where you only need to deal with integers.

So this brings up an interesting efficiency detail related to the
HTTP1 serialization:

You cannot tell the difference between an identifier and a number
by looking at the data on the wire, because RFC7230::token allows
both DIGIT, "." and "-".

Your parser will therefore initially classify any "number" as
"identifier", and only once your application code tells it that it
expects an integer, will that string be interpreted as a number.

That means that no numeric processing needs to happen until it
is unavoidable.

(Any design I can imagine for a future HPACK/H3 semantic compression
will have the same property.)

As I hint in 7.2 as "Future work", having the individual headers
defined in a machine readble format (a "schema") would allow us to
use programs to generate combined CS parser+validator for such a
header.

However, history is particularly unkind to that concept, so I
left it in "Future work" and I think we should leave it there
for now.

>Common Structure would need to encode floating point numbers.

I don't think so.

First, just because they look like floating point numbers in the
H1 serialization, doesn't mean you have to process them as IEEE754
in your application.

If I recall right, the q= weights can be processed as integer
"milli-q", and still be compliant.  In reality comparison of q=
weights is very often done as strings.

Second, apart from q= and possibly future timestamps, I don't see
much call for fractional numbers anyhow[1].

All that said, it probably does makes good sense to have "integer"
as a special case of "number", like "ascii_string" is a subset of
"unicode_string", because it will make it easier for people
to specify their HTTP headers.

Poul-Henning

[1]  Somebody will undoubtedly use them for monetary values, but I
     have a hard time seeing that make it into a published RFC for
     many reasons.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 10:27:31 +0000

--------

>way to have unique IDs.

And I've seen people use ieee754 binary floating point for
currency values...

Remember that this is laregly a forward looking thing.

People who specify HTTP headers with CS should use "identifier"
for values which are supposed to be unique IDs, because that
allow smarter people to stuff an UUID in there.

>There's nothing wrong with asking for one GB of data starting at 1.23 PB.

That's true.

That is a good argument for having a 64 bit "integer" type.

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 17 Nov 2016 19:33:18 +0900

> use programs to generate combined CS parser+validator for such a

Once we start people using CS to define new headers, I think I'd be
likely writing a parser generator handle them, regardless of whether
if a schema language is defined as a specification.

The generated parser would just deserialize the properties that the
application care of, while skipping unknown parameters.

> to specify their HTTP headers.

Thank you for considering that. I think that would be a good move.

-- 
Kazuho Oku

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 17 Nov 2016 10:35:54 +0000

--------
In message <[email-masked]>

>application care of, while skipping unknown parameters.

... provided the definition of the header allows that.