Subject: Re: New Version Notification for draft-nottingham-structured-headers-00.txt

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Nov 2017 10:42:53 +1100

Just a thought; maybe we shouldn't be defining "numbers" here, but instead "i32" or similar. 

The intent of Structured Headers -- in my mind -- is not to address every conceivable use case for HTTP headers; rather it's to address the 80% case. There will always be people who need/want bespoke formats. If we can hit the 80% case (or more) with n bits, people who need more can use another format -- and if there's enough demand, we can add a different structure for that.

Cheers,

> On 1 Nov 2017, at 5:41 pm, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Kazuho,
> 
> On Wed, Nov 01, 2017 at 10:52:53AM +0900, Kazuho Oku wrote:
>> How long is the expected lifetime of Structured Headers? Assuming that
>> it would be used for 20 years (HTTP has been used for 20+ years, TCP
>> is used for 40+ years), there is fair chance that the 49¾ bits limit
>> is too small. Note that even if we switch to transferring headers in
>> binary-encoded forms, we might continue using Structured Headers for
>> textual representation.
>> 
>> Do we want to risk making _all_ our future implementations complex in
>> exchange of being friendly to _some_ programming languages without
>> 64-bit integers?
> 
> That's an interesting question that cannot be solved just by a Yes or a
> No. Making a language totally unable to implement a protocol (especially
> HTTP) is a no-go, and may even ignite the proposal of alternatives for
> some parts. So we must at least ensure that it is reasonably possible
> to implement the protocol even if that requires a little bit of efforts
> and if performance sucks, because people choosing such languages despite
> such limitatons will do it only for convenience and the languages will
> evolve to make their lifes easier in the future. What must really be
> avoided is everything requiring a full-range 64-bit internal
> representation all the time. But if 64-bit are needed only for large
> files, most developers will consider that their implementation is
> sufficient for *their* use cases (even if only 31 or 32 bits).
> 
> This is what the text-based integer representation has brought us over
> the last two decades : good interoperability between implementations with
> very different limits. The ESP8266 in my alarm clock with 50kB of RAM
> might very well be using 16-bit integers for content-length and despite
> this it's compatible with the rest of the world. Similarly haproxy's
> chunk size parser used to be limited to 28 bits for a while and was
> only recently raised to 32 after hitting this limit once.
> 
>> The other thing I would like to point out is that mandating support
>> for 64-bit integer fields does not necessary mean that you cannot
>> easily represent such kind of fields when using the programming
>> languages without 64-bit integers.
> 
> It only depends if all bits of the fields are always needed or not in
> general. If it's just a size, anyone can decide that limiting their
> implementation to 32-bit can be OK for their purpose.
> 
>> This is because there is no need to store an integer field using
>> integers. Decoders of Structured Headers can retain the representation
>> as a string (i.e. series of digits), and applications can convert them
>> to numbers when they want to use the value for calculation.
> 
> It can indeed be an option as well. A punishment I would say.
> 
>> Since the size of the files transmitted today do not exceed 1PB, such
>> approach will not have any issues today. As they start handling files
>> larger than 1PB, they will figure out how to support 64-bit integers
>> anyways. Otherwise they cannot access the file! Considering that, I
>> would argue that we are unlikely to see issues in the future as well,
>> with programming languages that do not support 64-bit integers _now_.
> 
> I totally agree with this. I like to optimize for valid use cases, and
> in general use cases vary with implementations. Similarly I want my code
> to be fast on fast machines (because people by fast machines for
> performance) and small on resource constrained machines. People adapt
> their hardware and software to their needs, the design must scale, not
> necessarily the implementations.
> 
>> To summarize, 49¾ bits limit is scary considering the expected
>> lifetime of a standard, and we can expect programming languages that
>> do not support 64-bit integers to start supporting them as we start
>> using files of petabyte size.
> 
> I think we can solve such issues by specifying some protocol limits
> depending on implementations. Not doing this is what has caused some
> issues in the past. Content-lengths larger than 2^32 causing some
> implementations to wrap for example have been used to cause request
> smuggling attacks. But by insisting on boundary checking for the
> critical parts of the protocol depending on the storage type (for
> well known types), we can at least help implementers remain safe.
> 
>>> If your 64 bit number is an identifier, the only valid operation
>>> on it is "check for identity", and taking the detour over a decimal
>>> representation is not only uncalled for, but also very inefficient
>>> in terms of CPU cycles.
>>> 
>>> The natural and most efficient format for such an identifier would
>>> be base64 binary, but if for some reason it has to be decimal, say
>>> convenience for human debuggers, one could prefix it with a "i" and
>>> send it as a label.
>> 
>> Requiring the use of base64 goes against the merit of using a textual
>> representation. The reason we use textual representation is because it
>> is easy for us to read and use. On most systems, 64-bit IDs are
>> represented as numbers. So people would want to transmit them in the
>> same representation over HTTP as well. So to me it seems that it is
>> whether we want 64-bit integers to be sent as numbers of strings (or
>> labels). That is the reason why I only compared the two options in my
>> previous mail.
> 
> There is also the option of considering such identifiers as arrays of
> 32-bit for implementers, since they don't need to perform operations
> on them. This is something we can explain in a spec (ie: how to parse
> identifiers in general).
> 
>> In this respect, another issue we should consider is that we can more
>> effectively compress the data if we know that it is a number
>> (comparing to compressing it as a text or a label).
> 
> Yep, I like the principle of variable length integers. It's not very
> efficient in CPU cycles but is still much more than any text-based
> representation when all code points are valid as it doesn't require
> syntax validation. But the benefits are huge for all small elements.
> We did this in haproxy's peers protocol (used to synchronize internal
> tables betweeen multiple nodes) because the most common types exchanged
> were server identifiers (typically values lower than 10 for 95% of
> deployments), and incremental counters (up to 64-bit byte counts) and
> we didn't want to use multiple types. By proceeding like this we can
> ensure that implementations are not very difficult and can accept
> limitations depending on their targetted use cases. And the bandwidth
> remains as small as possible.
> 
> By the way it is important to keep in mind that a data type is not
> necessarily related to the programming language's internal representation.
> IP addresses are not numbers, identifiers are not numbers, eventhough
> they can often be represented as such for convenience. Numbers have a
> fairly different distribution with far more small values than large ones.
> Identifiers (and addresses) on the opposite are more or less uniformly
> distributed and do not benefit from variable length compression.
> 
> Cheers,
> Willy

--
Mark Nottingham   https://www.mnot.net/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 01 Nov 2017 23:53:11 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
>instead "i32" or similar. 

The reason we have non-integers is the q=%f ordering parameter, and
that the pre-cursor draft tried to be a general purpose serialization.

Neither of those may not be a good enough reasons to keep number
under this new scope/goal.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 2 Nov 2017 10:22:58 +1000

On 2 November 2017 at 09:53, Poul-Henning Kamp <[email-masked]> wrote:

> ​
So what's the goal with this header format?  It feels like it's being moved
towards defining a rich set of types, where I thought it was aimed at
providing a simple set that provides good coverage (at the absolute
minimum: scalar, list, dictionary -- beyond that, scalar tokens, binary
blobs, quoted strings, and numbers).

If it's going toward richness, is there going to be an eventual need for,
for example, a "q-value" type?  Or a "timestamp"?  Those can look like
numbers, but that's an implementation detail and conceptually they are
different.

If the general definition of "number" were changed from "up to 15 digits"
to "guaranteed up to 15 digits, maybe more in other circumstances" and each
header field in question specified the minimum/maximum/etc. for its
particular case, what would that break?  I'm imagining the general
definition to include a standard reaction to numbers with too many digits
(beyond 15) for a particular implementation, and the individual header
fields could build on that general exception.  Am I missing something?  Is
this different from tokens/blobs/strings that are too long?

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Nov 2017 11:33:32 +1100

Hey Matthew,

> On 2 Nov 2017, at 11:22 am, Matthew Kerwin <[email-masked]> wrote:
> 
> 
> 
> So what's the goal with this header format?  It feels like it's being moved towards defining a rich set of types, where I thought it was aimed at providing a simple set that provides good coverage (at the absolute minimum: scalar, list, dictionary -- beyond that, scalar tokens, binary blobs, quoted strings, and numbers).
> 
> If it's going toward richness, is there going to be an eventual need for, for example, a "q-value" type?  Or a "timestamp"?  Those can look like numbers, but that's an implementation detail and conceptually they are different.

Personally, I don't think so. 

> If the general definition of "number" were changed from "up to 15 digits" to "guaranteed up to 15 digits, maybe more in other circumstances" and each header field in question specified the minimum/maximum/etc. for its particular case, what would that break?  I'm imagining the general definition to include a standard reaction to numbers with too many digits (beyond 15) for a particular implementation, and the individual header fields could build on that general exception.  Am I missing something?  Is this different from tokens/blobs/strings that are too long?

I'd like to see generic parsers for the types that enforce the syntax and raise errors consistently, so that we don't get into states where implementations behave differently or ambiguously. That would remove a major part of the burden for defining and implementing new header fields.

So while being *more* strict than the defined syntax is fine (i.e., placing different constraints on it), I don't think relaxing the constraints works.

Cheers,

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 2 Nov 2017 10:05:31 +0900

Hi Willy,

2017-11-01 15:41 GMT+09:00 Willy Tarreau <[email-masked]>:
>> is used for 40+ years), there is fair chance that the 49ž bits limit

I agree.

What I am arguing is that we should allow applications to send all
integers (especially all of those that fit into 64-bit) using series
of digits, rather than requiring use of strings, labels, or base64 for
storing them.

In my view, an application would not be enforced to handle every
number represented by a 64-bit number even if Structured Headers
defines handling of 64-bit numbers as a minimal requirement. For
example, you cannot download a file of 1EiB size, unless your
filesystem supports storing such large files.

> It can indeed be an option as well. A punishment I would say.

Actually I see it as an optimization.

Assuming that we would not be actually using all the numbers sent
using Structured Headers, it makes sense to delay converting them to
internal numeric representation (i.e. `int64_t`) until it becomes
necessary. In fact, many of us already have such kind of optimization.
For example, many of the HTTP clients keep the Last-Modified header in
string received as-is, since it is seldom required to make
calculations using the value. It is wise to keep them as strings (and
send them as part of the If-Modified-Sence header). What I am
suggesting is that the fields of Structured Headers can be handled the
same way.

Note that such optimization might not make sense for hyperscalar CPUs,
since they could validate the numeric representation (i.e. check if
the characters are digits) at the same time convert them to an
integral type. But still, it could be a good optimization for embedded
devices with less-complicated CPUs that we are trying to take care of
in this thread.

>> To summarize, 49ž bits limit is scary considering the expected

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 2 Nov 2017 10:23:20 +0900

Hi Mark,

Thank you for the response.

> Just a thought; maybe we shouldn't be defining "numbers" here, but instead "i32" or similar.

I am not sure if that is a good solution the issue, even in the case
the intent of Structured Headers is to address the 80% case.

IMO, we need to at least have a representation that can carry the size
of the file, which cannot always be represented as a i32 value. So the
introduction of a sized type (i.e. i32, i64) means that we would need
to have _two_ decoders for numbers, instead of one.

My question here is what the merit of having two decoders is.

Consider the case of a memory constrained HTTP client that can only
handle int32_t. When it sees a content-length value in i64 form (e.g:
`content-length: 1234567890123u64`), it would fail to handle the
response. That's exactly the same as what we see now with the use of
numbers without type specifiers (e.g. `content-length:
1234567890123`).

So, I do not see why you want to have multiple number types (with
different limits).

Am I missing something here; e.g., a possibility of having a more
graceful error handling, that can only be achieved through the
introduction of sized types?

-- 
Kazuho Oku

---

From: Loïc Hoguin <[email-masked]>
Date: Thu, 2 Nov 2017 01:32:32 +0000

On 11/01/2017 01:52 AM, Kazuho Oku wrote:
> Hi, PHK!
> 
> Thank you for your response.
> 
> 2017-10-31 17:26 GMT+09:00 Poul-Henning Kamp <[email-masked]>:
>> --------
>> In message <[email-masked]>
>> , Kazuho Oku writes:
>>
>>> So why not mandate support for 64-bit integers?
>>>
>>> [...]
>>>
>>> Let's not repeat the failure made by JSON.
>>
>> If we were designing a general-purpose data-carrier format, I would
>> be 100% with you there, but we are not.
>>
>> The goal here is to design a maximally robust data-carrier format,
>> and that means conservative choices and putting the inconvenience
>> on the end which packages the data.
> 
> In my view, current limit (15 digits at max.) is overly conservative.
> 
> Let me explain in response to your text below.
> 
>> The number format is intended for sending quantities on which
>> arithmetic makes sense, and the point of the restriction is
>> to reserve to the implementor the ability to use the most
>> efficient hardware native data type, without loss of precision.
>>
>> 15 digits is 49¾ bits, and while I'm not prepared to state
>> that "is enough for everybody" I think we can safely say that
>> it covers all uses of arithmetic seen in HTTP until now.
> 
> IMO, we should consider the future instead of optimizing against what
> we see now.
> 
> When Content-Length was defined in HTTP/1.0 back in May 1996, the
> largest file we used to transfer were CD-ROM images (650MB ~ 1GB ~
> 2^30 bytes). We are now after 20 years since that, and we are seeing
> SD cards of 512GB (~ 1TB ~ 2^40 bytes). Assuming that the increase
> will continue, we would be seeing a storage that can store 2^50 bytes
> (~1 PB) of data within 20 years.
> 

I will second that question for the opposite reason. The implementation 
I maintain currently supports integers of up to around 255^255 because 
the programming language's integers are implemented as bignums once they 
are above a certain value.

This means that theoretically I could accept or server content of size 
well exceeding the 15 digits, today. Having a new hard limit of 15 
digits would mean that, at least for content-length, I would have to 
start rejecting what worked perfectly before. (I don't think I've heard 
of anyone using values this large yet, but as pointed out this could 
well change in 20 years.)

Is that a real issue as far as content-length is concerned though? I 
think not. If the client tries to upload content that is too large, we 
have an error for that. If the server tries to send content that is 
larger than the client expects, the client can abort and figure out an 
alternative. This can happen not only because the number of digits is 
too large but also because the value is too large (not enough space on 
disk, file system not allowing files larger than X bytes, etc.) so I am 
not sure that a restriction on the number of digits helps in that case 
either.

It also won't help the smallest implementations that can sometimes be 
seen in embedded deployments; for many of the ones I've seen, even if 
they can parse a 15 digits content-length, they're not going to do 
anything with it because they don't have the memory. For them, it 
doesn't matter if the limit is 15 or 150 digits.

I understand wanting to nicely map textual representations to 
programming language types but that rarely plays out as expected. JSON 
is a great example of that, it has Javascript's numbers: floating point 
numbers that are made to look like integers. As a result it has proven 
unusable as-is for a variety of applications that need better precision 
(financial, for example). If JSON had used decimal numbers instead from 
the start there would have been no problem.

Cheers,

-- 
Loïc Hoguin
https://ninenines.eu

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Nov 2017 14:21:40 +1100

Hi Kazuho,

> On 2 Nov 2017, at 12:23 pm, Kazuho Oku <[email-masked]> wrote:
> 
> Hi Mark,
> 

If you need to support integers of such large size, I think we're already talking about two decoders, because we need to support floats too.

> numbers without type specifiers (e.g. `content-length:

It's an interesting question. We started down this path because there are interoperability problems in JSON with large numbers, and we wanted to avoid those. We're not JSON-based any more (although in some use cases, I suspect JSON will be used as an alternative serialisation). 

Most of the interop problems come around the edges of floats. Straw-man proposal: we could define float as 15 digits of precision (as we do now), and i64 for integers. That would give us good interop and a good range of capability. WDYT?

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 2 Nov 2017 13:42:53 +1000

On 2 November 2017 at 10:33, Mark Nottingham <[email-masked]> wrote:

> > So what's the goal with this header format?  It feels like it's being
> moved towards defining a rich set of types, where I thought it was aimed at
> > If it's going toward richness, is there going to be an eventual need
> for, for example, a "q-value" type?  Or a "timestamp"?  Those can look like
> > If the general definition of "number" were changed from "up to 15
> digits" to "guaranteed up to 15 digits, maybe more in other circumstances"
> and each header field in question specified the minimum/maximum/etc. for
> its particular case, what would that break?  I'm imagining the general
> I'd like to see generic parsers for the types that enforce the syntax and
> raise errors consistently, so that we don't get into states where
> implementations behave differently or ambiguously. That would remove a
> major part of the burden for defining and implementing new header fields.
>
> So while being *more* strict than the defined syntax is fine (i.e.,
> placing different constraints on it), I don't think relaxing the
> constraints works.
>
> Cheers,
>
> ​
I think this last point is where I trip up.  We define a generic structure
that covers most use-cases, with the most permissive version of each rule
(i.e. the generic parser can potentially accept more digits than a
header-specific parser would).  Looking at it from one perspective, each
individual header field spec defines a restricted profile, with its own
rules for how to deal with values that are ok generically, but not ok
specifically.

It seems weird, then, to pick a relatively-conservative upper limit for a
generic value, since it's meant to be the most permissive.  I don't know
about concrete use-cases for large numbers since the one people keep
talking about is Content-Length, which already exists and isn't defined
according to this generic structure;  but even without a use-case it still
feels a bit off to me.

I agree that calling it out specifically as a fifteen-decimal-digit type
(your '"i32" or similar') leaves it open in a better way.  One day we could
see an "i64", or even an arbitrary-length "number", if and when they're
needed, and having individual header field specifications describe their
uses will be a clear signal that an implementation does/doesn't need to
support them.

I'm trying to keep my personal feelings out of this about exactly how many
digits is good enough, and what sort of languages we care about optimising
towards.

Cheers

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 2 Nov 2017 14:18:23 +0900

Hi Mark,

> If you need to support integers of such large size, I think we're already talking about two decoders, because we need to support floats too.

I can understand your point that we would be having at least two
parsers, one for integers and one for floats (or three, if we consider
positive and negative integers as different types). But still, I am
not sure if the fact justifies having multiple parsers for different
sizes of integer types, since, as I stated, I do not see if there
would be any practical merit (e.g., interoperability improvement).

> It's an interesting question. We started down this path because there are interoperability problems in JSON with large numbers, and we wanted to avoid those. We're not JSON-based any more (although…

I can understand that the hesitation to require support for integer
values exceeding 2^52 due to the fact that cannot be handled correctly
by using a native type of JavaScript (or possibly with some other
programming languages; are there any?).

But as Alex pointed out [1], JavaScript is likely going to support
bigint. I would also assume that programming languages that lack
support for 64-bit ints now would start supporting 64-bit ints as (or
before) we start seeing files above 2^52 bits.

> Most of the interop problems come around the edges of floats. Straw-man proposal: we could define float as 15 digits of precision (as we do now), and i64 for integers. That would give us good inter…

I'd be fine if the limit for int becomes 64 bits (though I am not sure
if we even want a limit here).

For floats, first let me state that I do not know if we even need to
support it. To me it seems worth considering to drop it, as PHK
suggested.

That said, let me point out that you might not need to define the
maximum number of digits permitted in fraction part. You are anyways
expected to have some errors in floating point maths. The reason we
had issues in JSON Header Field Values is because it permitted the use
of exponents (i.e. "eNN") [3]. Without support for exponents, such
issue would not happen in Structured Headers.

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2017OctDec/0134.html
[2] https://lists.w3.org/Archives/Public/ietf-http-wg/2017OctDec/0140.html
[3] https://lists.w3.org/Archives/Public/ietf-http-wg/2016JulSep/0154.html

-- 
Kazuho Oku

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 2 Nov 2017 15:58:07 +1000

On 2 November 2017 at 15:18, Kazuho Oku <[email-masked]> wrote:

> ​[snip]​
​Not all numbers with dots in them are interpreted as IEEE 754 floats; for
example Ruby's BigDecimal [1] and Rational [2] classes. If precise
fractional numbers are required for a header value, there should definitely
be strong statements about the required precision.

That's a big "if", though; all the fractional numbers I can think of can
quite easily be served by fixed-point numbers/integers, either by changing
the units (seconds to microseconds, say), or adding a semantic constraint
to the parsed value (e.g. q-value * 1000).  So I wouldn't be opposed to it
being dropped.

Cheers

[1]:
https://ruby-doc.org/stdlib-2.4.2/libdoc/bigdecimal/rdoc/BigDecimal.html
[2]: https://ruby-doc.org/core-2.4.2/Rational.html

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 Nov 2017 06:54:59 +0100

Hi Kazuho,

> would be any practical merit (e.g., interoperability improvement).

In fact I think the root cause of the problem is that we must consider
the failure to parse as something normal, and not something which never
happens. Ie: a 32-bit content-length can be perfectly fine for many use
cases (probably 99.99%), but what is not acceptable is that the parser
silently fails to parse larger values and wraps or returns -1 or such a
thing.

Thus I'd rather suggest that number parsers MUST be able to correctly
parse all sizes supported by the implementation AND detect their own
failures to do so.

Personally I don't care if "q=0.33333" is rounded down to "q=0.25" or up
to "q=0.5" by an implementation which uses fixed floats with only 1 or 2
decimal bits, if that suits their needs. But integers must always be exact.

That's why I think that variable-length encoding can help (in fact ASCII
encoding actually *is* variable length encoding, just a very inefficient
format). All implementations can be compatible with their target use cases
without having to deal with difficult types.

(...)
> support it. To me it seems worth considering to drop it, as PHK

+1. We can define that "q=" is specified in thousands for example and
that only 3 decimals have to be parsed there.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 Nov 2017 07:06:00 +0100

> filesystem supports storing such large files.

... or you're forwarding it because you're an intermediary :-)

> suggesting is that the fields of Structured Headers can be handled the

That's a good point. In haproxy the only integers we always parse are
the content-length (which requires a lot of care including for duplicate
values etc), and the chunk sizes. The rest is optional. I suspect that
caches like Varnish and Squid have to deal a lot with dates. I'd still
be tempted to consider the difference between a "quantity" and an
identifier but as I mentionned in another e-mail, ASCII already provides
a variable length encoding which is moderately efficient for quantities
(hex in chunks is a bit better though). Probably that we'll find that
in the end, we can have "numbers" (in ASCII format) for everything, but
that we need a special case for file sizes and offsets (content-length,
ranges, chunk sizes) and a special case for dates.

> devices with less-complicated CPUs that we are trying to take care of

It's nice to consider this but it's also important to consider that most
implementers use high-level languages and consider such stuff useless.
How many times I've heard that I was playing with micro-optimizations
while in fact I was trying to protect against trivial DoS situations!

Thus I'd suggest to design with ease of optimization in mind but with
ease of *safe* implementation first.

Willy

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 2 Nov 2017 16:58:57 +0900

Hi Willy,

> parse all sizes supported by the implementation AND detect their own

I think that this is a powerful argument.

To rephrase it my way, the question is if (and how) we can expect
interoperability improvement by limiting the maximum of a integer
value. Unless there is a reason, we should not have limits.

As you have stated, each endpoint will have their own limits, and they
would not be able to handle values above that. So I basically agree
with you that having a limit does not improve interoperability, while
making things complicated.

The only case I could expect an improvement by having a hard limit is
when an intermediary is involved.

Consider the case in which we have a proxy that can only handle
integer values under 2^32. The proxy rejects forwarding messages with
a greater integer value included at any place within the message,
regardless of if it is actually using the value.

If we had limited the maximum to 2^32-1, the client and the server
trying to communicate a value greater than 2^32-1 will be forced to
use something other than an integer type. At the cost of complexity,
the two would be possible to communicate with each other through the
proxy.

But if we have no limits, the two will encode values greater than
2^32-1 as integers, and the proxy will refuse to forward the message.

Fortunately, I think that the problem can be made a non-issue by
clarifying that intermediaries must not refuse to forward messages
when it finds an integer value greater than it can parse in a header
field attribute that it does not use.

What do you think about adding such a requirement?

-- 
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 Nov 2017 09:14:11 +0100

Hi Kazuho,

> making things complicated.

Yep!

> when an intermediary is involved.

It can seem to be the case but in practice, intermediaries just have to
adapt to realistic use cases as well, because in their position, having
to consider performance is critically important as well. For example I
don't intend to change the 32-bit chunk parser in haproxy for a while
as it will severely hurt performance on 32-bit platforms. It will only
change the day I start to receive a bug report about it being too small.

> 2^32-1 as integers, and the proxy will refuse to forward the message.

Yes but in practice intermediaries are progressively exposed to changes
so it's really not a problem to deal with this. So this problem basically
happens a few times in a decade and is quickly addressed after someone
says "bah what a crappy code still using 32-bit when everyone is running
64" and then the implementer thinks it's the right time to switch.

> field attribute that it does not use.

I'd reformulate differently : as long as whatever it manipulates will not
affect the message's integrity nor subsequent messages' integrity, it can
simply pass as-is. Haproxy doesn't try to parse header fields it doesn't
use (the name, colon, value and CRLF are indexed but that's all). But
definitely content-length is critical to the message's integrity and the
next ones'. Similarly if it needs to rewrite a header field value based on
a previous value, it's important that the value is not truncated during
conversion (which is why our internal integers for application-level
processing are 64-bit, such processing is not performance critical).

> What do you think about adding such a requirement?

I think I'm fine with this.

Willy

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 2 Nov 2017 22:56:58 +1300

> ... or you're forwarding it because you're an intermediary :-)

Or its not a file size at all but some other value related to large 
files. ie frame counters on long-lived video feeds, Range etc.

Squid already is used on various academic networks where the datasets 
are so huge they have to be chunked by the TB. The Content-Range values 
on those messages can be amazing. I don't see those requirements getting 
any smaller in future.

> caches like Varnish and Squid have to deal a lot with dates.

Date are not so much of a problem for integer parsing. Age / max-age are 
more the worry as they are just on the brink of 32-bit being 
insufficient for edge cases involving archival content.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Nov 2017 09:30:08 +1100

> that only 3 decimals have to be parsed there.

I'm not so sure we can do that; it's not just QValues. E.g.,

http://httpwg.org/http-extensions/client-hints.html#dpr

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 2 Nov 2017 23:48:00 +0100

> http://httpwg.org/http-extensions/client-hints.html#dpr

Interesting one. With the proposed format there's no way to reproduce
the exact value (eg: 4000x3000 will send 1.3333 but after how many '3'
is it enough ?). Ratios could be useful there (eg: 4000/3000) but some
implementations will always suffer from div by zero or the funnier div
by -1 which nobody cares to check against and which produces the same
result. Maybe we should accept some limits to the precision here, such
as 16 bit integral value and 16 bit fractional one for all decimal
numbers (as an example of course).

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Nov 2017 10:30:43 +1100

Huh. 

Ilya, has there ever been discussion of using a format like "4:3" for DPR, to be more accurate?

Cheers,

> On 3 Nov 2017, at 9:48 am, Willy Tarreau <[email-masked]> wrote:
>

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 09:40:54 +1000

> numbers (as an example of course).
Or a simple parsing rule: 15 decimal digits, with an optional dot somewhere
in the middle.

That said, I don't think DPI is a great example; for one, because it
already exists (unless someone's planning to respecify it according to the
generic structure in future?), and for two, because it can be alternatively
stated as an integer number of dots per inch, which is the direction CSS
seems to be heading [1].

I'm not the most imaginative person, but I still can't come up with a
fractional number that isn't either a fixed-point number, an inverted
fraction, or bound to a unit, any of which can be turned into an integer by
restating the requirement (no hacking or kludging involved).  Which makes
me think that floats are convenient, but unnecessary.

Cheers

[1] https://drafts.csswg.org/mediaqueries-3/#resolution
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Nov 2017 10:52:39 +1100

> On 3 Nov 2017, at 10:40 am, Matthew Kerwin <[email-masked]> wrote:
> 
> That said, I don't think DPI is a great example; for one, because it already exists (unless someone's planning to respecify it according to the generic structure in future?), and for two, because it can be alternatively stated as an integer number of dots per inch, which is the direction CSS seems to be heading [1].

It's our spec, and it's still a draft :)

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Nov 2017 10:51:02 +1100

> making things complicated.

I think I'm going to push back on that. 

Having a limit defined in structured headers means that a header defined using them is guaranteed that receiving implementations will be able to handle a known set of values. If the recipient doesn't, it isn't properly implementing structured headers, so the only thing it can do is ignore the header -- the correct behaviour for any header that isn't implemented.

We *could* specify a minimum bar (e.g., "all implementations must be able to handle integers at least as big as 2**32"), but then larger integers would work with some implementations, while not work with others -- which is an interoperability problem.

A header that might use those larger values would then need to specify how to handle that error. Since the whole point of structured headers is to take as much of that work off of the shoulders of header authors as possible, that doesn't seem good.

Likewise, if we don't specify any limit, we'll get into the situation Willy outlines -- developers will choose values that they think are reasonable, until they get bug reports. Those bug reports *are* interoperability problems!

Again, I'll assert that not every numeric value that could possibly be transferred in HTTP needs to fit into structured headers; new headers don't have to use them, and we can add more structures later on if necessary. 

> What do you think about adding such a requirement?

If an intermediary finds an integer greater than 2**64 in a value, you're saying that it is required to forward it? What if it's e.g., 2**2048? 2**16384?

Intermediaries inspect header values all the time, and they need to be able to refuse to forward a message for any reason. E.g., Web Application Firewalls.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 10:08:00 +1000

> It's our spec, and it's still a draft :)
Oh, yeah.  Oops.  Only one reason, then.

Cheers

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 3 Nov 2017 06:53:40 +0100

On Fri, Nov 03, 2017 at 10:51:02AM +1100, Mark Nottingham wrote:
> Having a limit defined in structured headers means that a header defined
> using them is guaranteed that receiving implementations will be able to
> handle a known set of values. If the recipient doesn't, it isn't properly
> implementing structured headers, so the only thing it can do is ignore the
> header -- the correct behaviour for any header that isn't implemented.
> 
> We *could* specify a minimum bar (e.g., "all implementations must be able to
> handle integers at least as big as 2**32"), but then larger integers would
> work with some implementations, while not work with others -- which is an
> interoperability problem.
> 
> A header that might use those larger values would then need to specify how to
> handle that error. Since the whole point of structured headers is to take as
> much of that work off of the shoulders of header authors as possible, that
> doesn't seem good.

I'd rather proceed differently :
  - indicate a limit for sent values which serves as a minimum size for
    received value *to be compliant*. Eg: "if you send values no greater
    than 2^32-1, and can parse values as large as 2^32-1, you are expected
    to be interoperable with any compliant implementation

  - indicate how to safely detect overflows and how to deal with them

  - recommend to support larger values on receipt based on the fact that
    most machines where performance matters support 64 bits nowadays, so
    that you can be mostly compliant with non-compliant implementations

Then remains the problem of intermediaries which forward stuff they don't
have to analyse but it's not that much of a problem.

A possible solution derivating from the suggestions above could simply be
to define a few profiles for Structured Headers :
  - SH0 : the lowest level, implementations do not support receiving as
    large values as specified in next level (eg: my alarm clock's ESP8266
    possibly)

  - SH1 : sends and receives up to 2^32-1

  - SH1+: sends up to 2^32-1, receives up to 2^63-1

  - SH2 : sends and receives up to 2^63-1

This way we know that implementations within a same profile are interoperable,
that SH1 and SH1+ are interoperable, and that SH1+ and SH2 are as well. SH1
and SH2 are not, and SH0 is "good luck" (giving incentive to at least reach
SH1).

> If an intermediary finds an integer greater than 2**64 in a value, you're
> saying that it is required to forward it? What if it's e.g., 2**2048?
> 2**16384?

As long as it doesn't *need* it, yes. It's just passing the data as-is
without analyzing it.

> Intermediaries inspect header values all the time, and they need to be able
> to refuse to forward a message for any reason. E.g., Web Application
> Firewalls.

I disagree, intermediaries inspect almost nothing except what they need for
their own use. A WAF would need to have a rule to inspect that header, and
as such it becomes a recipient for that value and must parse it correctly.
If it fails to parse it, it cannot find the value and has to reject it. But
as long as it has no rule for that header, it has no reason for being
affected at all, that's just opaque data.

For me interoperability is really defined by who *consumes* the data, not
all those possibly on the path. Similarly the data flows through your router
and the router doesn't care about the contents in the TCP segments which
compose the IP packets it routes.

Willy

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 16:44:44 +1000

> and SH2 are not, and SH0 is "good luck" (giving incentive to at least reach
​Isn't that the same as Mark's suggestion of carving out "i32" as its own
numeric type?  (Assuming other numeric types are defined as they're found
to be valuable.)  Except that it moves the profile from the implementation
to the header field definition.

My thought process:

If 'Foo' is an integer known/defined to be within [1,3600], say, then its
spec can say: "the value MUST be a <theoretical-unsigned-4digit-type>
between 1 and 3600, respectively".

If 'Bar' can be between negative-a-lot and positive-a-lot, its spec would
say "...MUST be a <theoretical-signed-19digit-type>...".

So, 'Foo' is parsed against `"0" / NONZERO *3(DIGIT)`​ and 'Bar' against `[
"-" ] ( "0" / NONZERO *18(DIGIT) )` or whatever.

Implementations that want to understand messages with 'Foo' header fields
need to be able to parse and operate on, effectively, uint16 data.

Implementations that want to speak 'Bar' but don't have native int64
support deal with edge-cases either by using an integer-extending library,
or by being non-compliant.  But if 'Bar' can definitely, reasonably,
contain values beyond what the implementation can deal with, then that
implementation was never going to be interoperable anyway.

Cheers

---

From: Andy Green <[email-masked]>
Date: Fri, 3 Nov 2017 15:13:23 +0800

On 11/03/2017 02:44 PM, Matthew Kerwin wrote:
> 
> 
> On 3 November 2017 at 15:53, Willy Tarreau <[email-masked] <mailto:[email-masked]>> 
> wrote:
> 
> 

> support deal with edge-cases either by using an integer-extending 
> library, or by being non-compliant.  But if 'Bar' can definitely, 
> reasonably, contain values beyond what the implementation can deal with, 
> then that implementation was never going to be interoperable anyway.

I hesitate to bring it up but what about MPINT / BIGINT?  They're 
integers too.  If you don't want a limit, then a way to escape to a 
representation with a length + MSB payload will also be able to carry keys.

If a limit is ok, that the supported values doesn't fit conveniently in 
the natural type for the platform is unrelated IMHO.  After all ESP8266 
/ 32 can do RSA via BIGINT / MPINT.  And the compiler for ESP8266 / 
ESP32 supports a 64-bit long long, there's no barrier to using it.  As 
already mentioned filesizes long ago made a 32-bit limit not very 
useful.  So doesn't it make sense to just require 64-bit if there's to 
be a simple limit?

-Andy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Nov 2017 07:09:02 +0000

--------
In message <[email-masked]>

>between 1 and 3600, respectively".

The entire reason behind this draft is that the spec-writer can
write:

	"The value MUST be 1 <= value <= 3600, and serialized as a
	Common Structure Number [RFCXXXX]."

and *not* write any ABNF at all.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 17:40:26 +1000

On 3 November 2017 at 17:13, Andy Green <[email-masked]> wrote:

> I hesitate to bring it up but what about MPINT / BIGINT?  They're integers
> too.

​That's exactly what I meant by "integer-extending library".  I picked
19-digit-number and int64 because they're convenient for discussion, not
because I care about the particular numbers.​

>   If you don't want a limit, then a way to escape to a representation with
> a length + MSB payload will also be able to carry keys.
>
>
​Sorry, I can't parse that.  If you don't want a limit for your particular
header, don't use the common structure to define it.​

> the natural type for the platform is unrelated IMHO.

​Indeed.​

>   After all ESP8266 / 32 can do RSA via BIGINT / MPINT.  And the compiler
> for ESP8266 / ESP32 supports a 64-bit long long, there's no barrier to
> using it.  As already mentioned filesizes long ago made a 32-bit limit not
> very useful.  So doesn't it make sense to just require 64-bit if there's to
> be a simple limit?
>
> ​
If your implementation has trouble dealing with certain values (having to
rely on more-complicated data structures, algorithms, memory allocations,
etc), and you want to optimise for throughput/simplicity/etc., and the
header is clearly defined to have a limit within what you can handle
natively, why not be able to take advantage of that?  I'm thinking of
JavaScript (including Node.js) here, by the way, which stores all its
numbers in IEEE 754 floats.

Cheers

---

From: Andy Green <[email-masked]>
Date: Fri, 3 Nov 2017 15:53:57 +0800

On 11/03/2017 03:40 PM, Matthew Kerwin wrote:
> 
> 
> On 3 November 2017 at 17:13, Andy Green <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
> 
>         On 3 November 2017 at 15:53, Willy Tarreau <[email-masked]
>         <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
> 
> 
>                   large values as specified in next level (eg: my alarm
>         clock's
>              ESP8266
> 
> 
>         Implementations that want to speak 'Bar' but don't have native
>         int64 support deal with edge-cases either by using an
>         integer-extending library, or by being non-compliant.  But if
>         'Bar' can definitely, reasonably, contain values beyond what the
>         implementation can deal with, then that implementation was never
>         going to be interoperable anyway.
> 
> 
>     integers too.

...

>        If you don't want a limit, then a way to escape to a
> ​Sorry, I can't parse that.  If you don't want a limit for your 
> particular header, don't use the common structure to define it.​

... Willy is proposing some kind of capability profiles with fixed 
limits at 32-bit etc: that's what you had replied to.  I don't think 
that's going to give a nice result.

There seems to be a basic choice to have some strict integer limit, or 
to avoid it by having a "stretchy" way to have arbitrary-sized ints.  If 
people are OK with a strict limit, 64-bit is very widely supported at C 
level even down on "ESP8266" Willy mentioned.  I think a 32-bit limit is 
a non-starter.

>     If a limit is ok, that the supported values doesn't fit conveniently
>     in the natural type for the platform is unrelated IMHO.
> 
> 
> ​Indeed.​
> 
>        After all ESP8266 / 32 can do RSA via BIGINT / MPINT.  And the
>     compiler for ESP8266 / ESP32 supports a 64-bit long long, there's no
>     barrier to using it.  As already mentioned filesizes long ago made a
>     32-bit limit not very useful.  So doesn't it make sense to just
>     require 64-bit if there's to be a simple limit?
> 
> ​
> If your implementation has trouble dealing with certain values (having 
> to rely on more-complicated data structures, algorithms, memory 
> allocations, etc), and you want to optimise for 
> throughput/simplicity/etc., and the header is clearly defined to have a 
> limit within what you can handle natively, why not be able to take 
> advantage of that?  I'm thinking of JavaScript (including Node.js) here, 
> by the way, which stores all its numbers in IEEE 754 floats.

No, floats are not ints and cannot reliably be converted into them. 
They are a lossy approximation format.  For example you cannot put key 
BIGINTs into floats.

If there's demand for floats of course they can also be handled, but 
they can't be conflated with integers just 'cos node.

Also eg, ESP8266 Willy mentioned has compiler support for 64-bit long 
long.  It has no hw float or double.

-Andy

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 3 Nov 2017 16:31:51 +0900

Hi Mark,

> Likewise, if we don't specify any limit, we'll get into the situation Willy outlines -- developers will choose values that they think are reasonable, until they get bug reports. Those bug reports *…

I agree that the chunk size problem is an interoperability issue, and
that it would be better to have a bar for the chunk sizes.

OTOH, to me it seems that the issue is rather exceptional. In chunked
encoding, transferring a 16GiB file as 16000 chunks (1MiB each) or as
2 chunks (8GiB each) are semantically equivalent. So we can avoid the
issue by capping the maximum permitted value.

Consider Content-Length or Content-Ranges. An endpoint needs to send
the value in some way. If we cap the maximum in Structured Headers,
then applications would be forced into finding their own way of
encoding (possibly by splitting a integer value to an array of
integers; e.g. upper 32-bits and lower 32-bits).

I would anticipate that the majority of our use-case is to send a
specific value (i.e. like Content-Length or Content-Ranges) rather
than chunked encoding. Hence arguing for not having an upper limit.

Having a guaranteed minimum (e.g., every implementation should at
least support NN bits) might not be a bad idea, but my stance is that
if we choose NN it should be no smaller than 64 since we might see
files that do not fit into 2^52 in next 20 years.

> If an intermediary finds an integer greater than 2**64 in a value, you're saying that it is required to forward it? What if it's e.g., 2**2048? 2**16384?

Yes. Under the assumption that other limits (e.g. header field length)
exceed the configured maximum.

It's just a different way of thinking.

IMO, things become complicated only when we consider that the numbers
are values that get converted to internal type (e.g. int64_t) by the
decoder of Structured Headers.

If we consider the decoder to only validate the type (i.e. check that
all the characters being used are digits) and return that to the
application as a string (which might get converted to intNN_t by an
application when it needs to use it), it becomes a non-issue.

> Intermediaries inspect header values all the time, and they need to be able to refuse to forward a message for any reason. E.g., Web Application Firewalls.

I believe that it would be perfectly fine for a Web Application
Firewall to reject a HTTP message including some strange value. OTOH,
I do not see if a Firewall can determine if a value is suspiciously
large without knowning the meaning of the value. In other words, I
think that finding an unknown integer field of Structured Headers
would be no different to a Firewall to finding an unknown header
field. It does not impose any greater risk than observing an unknown
header field.

-- 
Kazuho Oku

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Nov 2017 06:55:03 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>Again, I'll assert that not every numeric value that could possibly be
>transferred in HTTP needs to fit into structured headers; new headers
>don't have to use them, and we can add more structures later on if
>necessary.

I'll add my voice to Mark here:   The point is not for this draft
to cover every imaginable scenario, but for it to cover as many of
the probable ones with as little complexity as necessary.

The acid test for me is this:

If we had this draft first, and then set about defining all the
headers we have to today, would any of them be impossible to express
inside the four corners of this draft ?

I belive the answer is no.

Some of them would look slightly different, but we could transmit
the information they contain just fine.

The one point where this draft comes up short-ish is timestamps.

I would be very easy to convince that we should add a type for
timestamps, expressed as a number counting POSIX time_t seconds.

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 18:17:18 +1000

On 3 Nov. 2017 5:55 pm, "Andy Green" <[email-masked]> wrote:

On 11/03/2017 03:40 PM, Matthew Kerwin wrote:

>
>
> On 3 November 2017 at 17:13, Andy Green <[email-masked] <mailto:
> [email-masked]>> wrote:​
>

... Willy is proposing some kind of capability profiles with fixed limits
at 32-bit etc: that's what you had replied to.  I don't think that's going
to give a nice result.

There seems to be a basic choice to have some strict integer limit, or to
avoid it by having a "stretchy" way to have arbitrary-sized ints.  If
people are OK with a strict limit, 64-bit is very widely supported at C
level even down on "ESP8266" Willy mentioned.  I think a 32-bit limit is a
non-starter.

> numbers in IEEE 754 floats.

No, floats are not ints and cannot reliably be converted into them. They
are a lossy approximation format.  For example you cannot put key BIGINTs
into floats.

That they are stored in floats is an implementation detail, the ES6 spec
says that safe JavaScript integers are limited to ~32 bit signed values (I
think, I haven't read it lately).

If there's demand for floats of course they can also be handled, but they
can't be conflated with integers just 'cos node.

The conflation was mine, the 32 bit limit is Node's.

Cheers
-- 
Matthew Kerwin

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 3 Nov 2017 17:23:19 +1000

> and *not* write any ABNF at all.
It might be my poor use of language, but I was trying to write almost
exactly what you did:  "The value MUST be 1<=value<=3600, and serialised as
a Common Structure
Whatever-We-Call-The-Short-Number-If-There-End-Up-Being-Multiple-Number-Types
[RFCXXXX]."

My angle brackets weren't meant to signal ABNF elements.

Cheers

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 03 Nov 2017 07:04:08 +0000

--------

>    to be interoperable with any compliant implementation

Remember:  This draft is only about text-processing.

No relevant computer can possibly have problems *parsing* a 64 bit
integer or 15 digit number.

They may not be able to deal with an object with that many bytes,
but the *semantics* of the numerical value parsed depends on the
header and limits and overflow handling should be defined in along
with the header.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 4 Nov 2017 08:17:00 +0100

On Fri, Nov 03, 2017 at 03:53:57PM +0800, Andy Green wrote:
> avoid it by having a "stretchy" way to have arbitrary-sized ints.  If people
> are OK with a strict limit, 64-bit is very widely supported at C level even
> down on "ESP8266" Willy mentioned.  I think a 32-bit limit is a non-starter.

The ESP in my alarm clock disagrees with you here :-)

  > print(32768*32768)
  1073741824
  > print(32768*65536)
  -2147483648
  > print(65536*65536)
  0
  > print(131072*65536)
  0
  > print(2147483648)
  2147483647
  > print(3333333333)
  2147483647

That's totally scary. The first 3 can be expected due to integer overflow
(2^30 is OK, 2^31 is negative, 2^32=0). The 4th one (2^33) is unexpected.
The last two ones shows that integer parsing uses saturation and reports
wrong values. That's precisely the type of issue I'd rather be sure to
address for sensitive protocol elements. But as long as we propose a
portable way to detect that a number is correctly parsed, I'm fine,
because as I mentionned earlier in this thread, such devices are not
going to be used to retrieve a 1 TB file.

But in C, code using atoi() to parse integers is very common, and when the
developers are told that atoi() is too short and unreliable and that they
must use strtol(), they end up using it naively causing such hard to detect
problems when they're not aware of the impacts :

  #include <stdio.h>
  #include <stdlib.h>

  int main(int argc, char **argv)
  {
        printf("atoi:   0x%16lx (%ld)\n", (long)atoi(argv[1]), (long)atoi(argv[1]));
        printf("strtol: 0x%16lx (%ld)\n", (long)strtol(argv[1], NULL, 0), (long)strtol(argv[1], NULL, 0));
        return 0;
  }

  $ ./a.out 2147483647
  atoi:   0x        7fffffff (2147483647)
  strtol: 0x        7fffffff (2147483647)

  $  ./a.out 2147483648
  atoi:   0xffffffff80000000 (-2147483648)
  strtol: 0x        80000000 (2147483648)

  $ ./a.out 4294967296
  atoi:   0x               0 (0)
  strtol: 0x       100000000 (4294967296)
 
  $ ./a.out 00003333
  atoi:   0x             d05 (3333)
  strtol: 0x             6db (1755)

That's why I think that we must really take care of 32-bit. Signed 32-bits
(hence unsigned 31 bits) are the only really portable integers. Most code
works pretty well above this, but naive implementations easily get caught
like this.

In fact I would probably be fine with unbounded integers if we :
  - explain how to safely parse an integer (and detect its correctness and
    validity for the type being used)
  - mention a warning about the risk of widespread code not capable of
    dealing with quantities larger than 2147483647.

(...)
> No, floats are not ints and cannot reliably be converted into them. They are
> a lossy approximation format.  For example you cannot put key BIGINTs into
> floats.

I agree, I really don't like floats to represent integers for discrete
values. Byte offsets and content-lengths are discrete values and must
absolutely be exact. Parsing "Content-length: 12345.5555" will lead to
funny things depending on the implementations. Some truncating to 12345
and others rounding up to 12346. It's even worse when such numbers have
to be converted to other quantities (eg: bits), because 12345.5555*8=
98764.4440 which differs from 12345*8=98760 possibly resulting in negative
values in subtracts (eg: remaining space in a buffer).

Willy

---

From: Andy Green <[email-masked]>
Date: Sat, 4 Nov 2017 15:59:00 +0800

> The ESP in my alarm clock disagrees with you here :-)

Well, the chip doesn't care :-) it's Willy that disagrees with me.  I 
don't claim to know the 64-bitness of BASIC or whatever you are testing 
with and you don't describe it.  So I think it is not relevant there is 
some code that runs on ESP platforms that just has 32-bit types.  That 
can be true on any platform.

It remains a fact that both ESP8266 and ESP32 recommended stock gcc 
toolchain from Espressif, that you would write http stuff with, supports 
proper 64-bit long long, making deploying it and all the operators using 
it trivial.

>    > print(32768*32768)

Here are the same trials done in C using gcc on ESP32 (because that is 
what I have to hand; but it's basically the same gcc toolchain + 32-bit 
Tensilica core as ESP32).

         lwsl_notice("32768 * 32768 = %lld\n", (long long)32768 * (long 
long)32768);
         lwsl_notice("32768 * 65536 = %lld\n", (long long)32768 * (long 
long)65536);
         lwsl_notice("65536 * 65536 = %lld\n", (long long)65536 * (long 
long)65536);
         lwsl_notice("131072 * 65536 = %lld\n", (long long)32768 * (long 
long)32768);
         lwsl_notice("2147483648 = %lld\n", (long long)2147483648);
         lwsl_notice("3333333333 = %lld\n", (long long)3333333333);

4: 32768 * 32768 = 1073741824
4: 32768 * 65536 = 2147483648
4: 65536 * 65536 = 4294967296
4: 131072 * 65536 = 1073741824
4: 2147483648 = 2147483648
4: 3333333333 = 3333333333

>    strtol: 0x       100000000 (4294967296)

Ehhhhhhhh that's **long** you are using there.  I am talking about long 
long.  You underestimate C programmers if you think they don't know the 
difference.

> works pretty well above this, but naive implementations easily get caught

This is why the C guys invented int64_t and friends (which are just 
typedefs into long long or whatever).  That is **thoroughly** portable, 
not just ESP8266 + ESP32 gcc but even windows has <stdint.h> with them.

So I dunno (because you don't tell) what you tested on, but the generic 
C toolchain for esp8266 + esp32 has 64-bit long long and uint64_t / 
int64_t, and support for all the C operators on those types, which is 
what matters for these platforms.

> values in subtracts (eg: remaining space in a buffer).

Yeah we completely agree on that, floats won't do for generic ints.

I don't have an opinion on whether the thing being discussed should deal 
with BIGINT, it just seemed to be missing from the discussion.  If it 
did, it would cover every other smaller limit case, but it would force 
people to deal with their length + MSB data format.  For anything other 
than MPINT / BIGINT, a 64-bit limit will cover anything related to 
dataset size for the foreseeable future.

-Andy

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 4 Nov 2017 19:21:08 +1000

On 4 November 2017 at 17:59, Andy Green <[email-masked]> wrote:

>>> are

​So, how about if I write this without using the word 'float' or 'int',
which carry specific connotations, and instead use 'number' to mean "a
rational number, on which arithmetic operations can be performed."  The set
of rational numbers can quite easily be separated into those with a
denominator of 1 (integers), and those with anything else (which I've been
calling "fractional numbers").  The integers and fractional numbers have
different conventions for being represented in ASCII sequences, and in this
particular sub-thread we're only discussing the representation (and thus
parsing) of integers, and how that is influenced by different environments'
abilities to operate on the values resulting from said parsing.  Yeah?

There is a pretty old ~(20 years), pretty stable (at least in respect to
its handling of numbers), pretty popular programming language out there
that states in its spec [1]:

    "...all the positive and negative integers whose magnitude is no
greater than 2^53 are representable in the Number type..."

and:

    "Some [...] operators deal only with integers in specific ranges such
as -2^31 through 2^31-1, inclusive, or in the range 0 through 2^16-1,
inclusive. These operators accept any value of the Number type but first
convert each such value to an integer value in the expected range. "

Reading the spec, notable examples of such operations are those related to
indexing, such as into arrays, and bitwise arithmetic operations (AND, OR,
NOT, SHIFT, etc.); in both cases values are limited to the range -2^31 to
2^31-1.

So in this language there is a specified hard upper limit on integer values
that can be handled natively (53 bits), and a softer but well-understood
practical limit on numbers that can be used in many common operations (32
bits).

Back to the question of parsing: if we define a limit for the number of
acceptable digits, it's a pretty strong judgement call propose a limit that
produces values that some languages can support natively, on the grounds
that they can support it natively, while ignoring such a popular language
that can't.

15 decimal digits (<50 bits) gets us the support of that language, and
covers a lot of use cases *.  9 decimal digits (<30 bits) would get us
close to universal support, but maybe be less useful.  Whichever concrete
maximum length is chosen, I think it's good to pick one that works for that
popular language (as well as all the others), but I don't think it's right
to call the parsing rule something as generic as "number".  I wouldn't mind
if it was 15 digits with a name of "common integer" or something, with the
view that we might, in future, find a need to define a 38-digit "large
integer" rule, or something like that.

* people keep talking about size of resources, but Content-Length and Range
are already defined and it'd probably be a while before they were respecced
to use this common header format.

Cheers

[1]:
http://www.ecma-international.org/ecma-262/8.0/index.html#sec-ecmascript-language-types-number-type

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 4 Nov 2017 10:29:06 +0100

On Sat, Nov 04, 2017 at 03:59:00PM +0800, Andy Green wrote:
> Well, the chip doesn't care :-) it's Willy that disagrees with me.

Sure :-)

> I don't
> claim to know the 64-bitness of BASIC or whatever you are testing with and
> you don't describe it.

Sorry, it's nodemcu, the Lua interpreter.

> So I think it is not relevant there is some code
> that runs on ESP platforms that just has 32-bit types.  That can be true on
> any platform.

Except that it's very popular, and just one example among others. I'm not
trying to find statistics on how many are OK and how many are broken, I'm
just saying that there are limitations at many places and that we have to
deal with these. I mentionned in another e-mail (I'm just realizing it was
off-list now) that a cleanly and carefully written HTTP server used as a
reference for nodemcu on this platform uses "tonumber()" to parse the
content-length, an unfortunately as you can see below, tonumber() has its
limits as well :

  > print(tonumber("3333333333"))
  2147483647
  > print(tonumber("0x123"))
  291
  > print(tonumber("000123"))
  123

Just an indication that developers are not always aware of the limits
they have to deal with nor the validity domain of the functions they're
using. The purpose of Structured Headers is to have safer and more
portable parsers, so we have to consider this above. Ideally I'd like
to see a set of HTTP number parsers safe for use progressively appear as
a replacement for tonumber(), atoi() and consorts athat web applications
should use over the long term.

> proper 64-bit long long, making deploying it and all the operators using it
> trivial.

Probably but not everyone uses C on such a platform when you have easy-to-use
alternatives like Lua, micropython and probably others. That's part of todays
web landscape unfortunately.

> >    > print(32768*32768)
> >    1073741824
> >    > print(32768*65536)
> >    -2147483648
> >    > print(65536*65536)
> >    0
> >    > print(131072*65536)
> >    0
> >    > print(2147483648)
> >    2147483647
> >    > print(3333333333)
> >    2147483647
> 
> Here are the same trials done in C using gcc on ESP32 (because that is what
> I have to hand; but it's basically the same gcc toolchain + 32-bit Tensilica
> core as ESP32).
> 

That's perfect and I'm not surprized.

> long.  You underestimate C programmers if you think they don't know the

Sorry, I forgot to mention that this was done on my 64-bit PC where long
and long-long are 64-bit. Look carefully you'll see that strtol() is safe
against overflow, but that with base set to zero as commonly found, it
parses hex as explained in the manual. I'm not making up this example,
I've seen such things used a lot of times. In fact people start with
atoi() until they're hit by a parsing issue, then switch to strtol() and
don't care about specifying the or even worse, purposely support this
because the same parser is used for content-length and for configuration.

> This is why the C guys invented int64_t and friends (which are just typedefs
> into long long or whatever).  That is **thoroughly** portable, not just
> ESP8266 + ESP32 gcc but even windows has <stdint.h> with them.

I agree. The web is just not only C (unfortunately as that's by far my
preferred language). It even used to be shell scripts for CGI at an era
where most shells were limited to 32-bit evaluation.

> with BIGINT, it just seemed to be missing from the discussion.  If it did,
> it would cover every other smaller limit case, but it would force people to
> deal with their length + MSB data format.  For anything other than MPINT /
> BIGINT, a 64-bit limit will cover anything related to dataset size for the
> foreseeable future.

I really think we should *encourage* 64-bit processing, *suggest* that
anything larger is possible provided it is handled with extreme care,
*remind* that maximum interoperability is achieved below 2^31, and
*enforce* strict parsing and detection of overflows in any case. If we
design with this, we should be able to make the best design choices for
most use cases and ensure that incompatibilities are safely covered.

Willy

---

From: Andy Green <[email-masked]>
Date: Sat, 4 Nov 2017 18:00:10 +0800

> Except that it's very popular, and just one example among others. I'm not

Mmm Lua is not that popular, and clearly it's the wrong tool to write a 
http server with if it dies at 32-bit.  Because for a long time people 
are serving files that are larger than 2GB routinely.

Bash is much more popular example that only has to support 32-bit, 
although modern implementations do support 64-bit.  Again, 32-bit bash 
isn't the right tool for serving files with then... no news there.  I 
don't think the larger web standards should be designed around these 
transitory and specific limitations.

>    > print(tonumber("3333333333"))

Okay... so that specific implementation is broken and no good for 
dealing with the >2GB reality we already live in for many years... there 
are many things that are no good for that task... these things can be 
forced into view by interoperability test suites...

> web landscape unfortunately.

Micropython... seems written in C

https://github.com/micropython/micropython

Lua... written in C

https://github.com/lua/lua

These broken 32-bit limited implementations are *already inadequate* for 
the >2GB world we have been living in for a long time now.  Their 
problem is just their specific implementation decisions to stick at 
32-bit internally... that's just their problem, not everybody's problem. 
  I don't think trying to restrict standards to meet their arbitrary and 
unnecessary (since the compiler underneath supports long long / int64_t 
just fine) limitations is a reasonable way forward.

> That's perfect and I'm not surprized.

Okie, so there is no problem with ESP devices using 64-bit types we agree.

> because the same parser is used for content-length and for configuration.

Well, that implementation is broken.  There are other broken C lib apis 
like gets(), but people still regard string processing with C as possible.

Lately there is a trend for tools like h2spec or Autobahn to confirm 
interoperation that will discover and flag this kind of issue just fine.

> where most shells were limited to 32-bit evaluation.

As we see though, underneath the lua and micropython examples, it is C. 
Yes those restricted implementations exist, but it's equally a clear 
reality the web does not stop at 2GB or 4GB for many years now, and 
neither to the C compilers.  So the problem is only these specific 
language implementations must either grow ways to represent larger ints 
or become irrelevant.

> most use cases and ensure that incompatibilities are safely covered.

I am afraid 32-bit only limit is basically useless for general web use 
for many years already.  Languages that don't even have a way to deal 
with >32-bit quantities are fundamentally broken and useless for a web 
with >2GB and >4GB objects.  That can't be the guide for standards, 
otherwise we would have inherited a web that coddles WIN16 / 8088 
limitations according to this logic.

-Andy

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 4 Nov 2017 21:09:02 +1000

On 4 November 2017 at 20:00, Andy Green <[email-masked]> wrote:

> http server with
> ​ [...]

​

​<snip>​

> Okay... so that specific implementation is broken and no good for dealing
> with the >2GB reality we already live in for many years... there are many
> things that are no good for that task... these things can be forced into
> view by interoperability test suites..

​​
<snip>
​

> ​​
>
> I am afraid 32-bit only limit is basically useless for general web use for
> many years already.  Languages that don't even have a way to deal with
> >32-bit quantities are fundamentally broken and useless for a web with >2GB
> and >4GB objects.  That can't be the guide for standards, otherwise we
> would have inherited a web that coddles WIN16 / 8088 limitations according
> to this logic.
>
>
​One last post from me, because I know I'm talking to the wind, but: I
would suggest that saying "this thing is bad or broken or wrong" doesn't
change the fact that someone will do it, on the open web, in a way that
causes Interesting™ breakages.

We aren't here to tell people what (not) to do; rather we're here to
describe how to get something done in a way that is useful and reliable,
hard to get wrong, and easy to sort out if/when it does eventually go
wrong.  That includes predicting and addressing ways we can foresee that it
could go wrong, or that similar things have gone wrong in the past.

And nobody (for a given definition of "nobody") cares about
interoperability test suites, apart from the really big one (i.e.
reality.)  Some of the errors that come up in that one can be rather...
Interesting™.

Cheers

---

From: Andy Green <[email-masked]>
Date: Sat, 4 Nov 2017 20:13:25 +0800

On 11/04/2017 07:09 PM, Matthew Kerwin wrote:
> On 4 November 2017 at 20:00, Andy Green <[email-masked] 
> <mailto:[email-masked]>>wrote:
> 
> 
>     Mmm Lua is not that popular, and clearly it's the wrong tool to
>     write a http server with
>     ​ [...]
> 
>     ​
> 
> 
> ​<snip>​
> 
> 
>     dealing with the >2GB reality we already live in for many years...
>     there are many things that are no good for that task... these things
>     can be forced into view by interoperability test suites.. 
> 
>     ​​
> 
> ​
> <snip>
> ​
> 
>     ​​
> 
>     I am afraid 32-bit only limit is basically useless for general web
>     use for many years already.  Languages that don't even have a way to
>     deal with >32-bit quantities are fundamentally broken and useless
>     for a web with >2GB and >4GB objects.  That can't be the guide for
>     standards, otherwise we would have inherited a web that coddles
>     WIN16 / 8088 limitations according to this logic.
> 
> 

The question is because some people on a 64-bit capable platform decided 
to use 32-bits internally, is that a problem that needs addressing in 
the generic standards?  Or is it the problem of the people who made 
those implementation decisions by themselves?

Nothing stops those servers processing the integers in question as 
strings and seeing if they exceed some implementation limit, and 
rejecting the request cleanly.  That would be fine for everyone I think.

If they don't do that, they don't even work properly serving large files 
today; that's their problem.

> wrong.  That includes predicting and addressing ways we can foresee that 
> it could go wrong, or that similar things have gone wrong in the past.

If you step back a bit, a simple, clear standard is the best way to get 
something "useful, reliable, hard to get wrong, easy to sort out". 
Piling on weird stuff - what was it, four "profiles" for integer 
sizes... is not making a better standard in those regards than just 
saying compliant implementations must handle 64-bit ints.  If it becomes 
so complex just to eat an int, you will create more "interesting" bugs 
you say you want to avoid.  As shown it is not an onerous requirement to 
say it should handle 64-bit even on weak platforms like ESP8266 / ESP32.

> reality.)  Some of the errors that come up in that one can be rather... 

Ehhh are you sure :-) I think you find many implementors really like 
having test suites.  Of course they don't "guarantee" there are no 
problems, but they will provoke expected problems and confirm adherence 
to the bits of the standard they check.  I have certainly had a lot of 
bugs and things I didn't realize from the standard I should have taken 
care about shown to me by h2spec.  And they are very useful as automated 
tests to confirm against regression too.

-Andy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 4 Nov 2017 13:38:12 +0100

On Sat, Nov 04, 2017 at 08:13:25PM +0800, Andy Green wrote:
> The question is because some people on a 64-bit capable platform decided to
> use 32-bits internally,

No, nobody decided to use 32-bit internally, they just used the integer of
the default size proposed by their language. "int" is signed 32-bit on the
vast majority of platforms and has even been used for decades to store IPv4
addresses. There's no reason for accusating anyone of purposely doing bad
stuff, the reality is that unsafe code exists all over the planet by lack
of knowledge and awareness. While we can hardly improve people's knowledge
using standards, we can at least improve their awareness of the issues.

> Nothing stops those servers processing the integers in question as strings
> and seeing if they exceed some implementation limit,

Honestly, checking for integer values using strings is complex and not
natural to anyone. Try to tell this to the guy who used tonumber(header)
in my previous example, where "tonumber()" is provided by default on his
system.

> today; that's their problem.

On small platforms nobody seeks to serve large files, they're dealing with
authentication pages, 10 settings on a page, and POST requests to change
such settings. Despite this these software do cause interoperability issues
right now by improperly parsing some valid responses (eg: when retrieving
the whether forecast from public services to adjust the heater). And when
you're unforunate enough to be responsible for the proxy in the middle that
blocks the so-called "valid" response that the device normally properly
deals with, it's a bit of a pain to have to argument that you're the only
one applying safe processing there.

> something "useful, reliable, hard to get wrong, easy to sort out". Piling on
> weird stuff - what was it, four "profiles" for integer sizes... is not
> making a better standard in those regards than just saying compliant
> implementations must handle 64-bit ints.  If it becomes so complex just to
> eat an int, you will create more "interesting" bugs you say you want to
> avoid.

I disagree on this one because we all know that we all code based on the
target use case. Do you add provisions against bit flips due to solar
eruptions in your code ? Most likely not. I don't either. But probably
if you were coding for a Mars probe you'd have to use "volatile" in front
of all your variables and perform such checks. It's just out of your scope
and you skip such checks, that's fine. People developing for platforms
adjusting their home temperature based on public forecast do the same,
no need to deal with complex code compatible with 64-bit quantities to
retrieve a JSON block.

> As shown it is not an onerous requirement to say it should handle
> 64-bit even on weak platforms like ESP8266 / ESP32.

As I shown it's the opposite. *existing* frameworks are not even able to
parse anything outside -2^31 and 2^31-1 nor have the types needed for this
and these ones are currently being used in various products.

> > Interesting(TM).
> Ehhh are you sure :-) I think you find many implementors really like having
> test suites.

I think you're only working in enterprise environments where some people do
care about this. While I too am fond of standards compliance, I'm disgusted
by what I see every other day, emitted by software supposed to be working
fine, or the type of issues certain software face. Just google for
"strstr cookie" or "atoi content-length" to see real-world horrors that *we*
can avoid by better specs. But it's not by asking HTTP implementers to have
to manually handle large integer arithmetics to comply with a spec that's out
of their usage scope that we'll see any progress made in this area!

Willy

---

From: Alex Rousskov <[email-masked]>
Date: Sat, 4 Nov 2017 09:30:09 -0600

On 11/04/2017 06:38 AM, Willy Tarreau wrote:
> *existing* frameworks are not even able to

Moreover, virtually all existing frameworks have bugs when processing
things _inside_ that 32-bit range, even though they have the types
needed for that range.

No matter what we define as Structured Headers, there will be many
implementation bugs. Our goal should be

* a standard that a good developer can implement correctly without major
hassles,

rather than

* a standard that an implementation by an average developer cannot screw up.

I have not heard arguments why the former cannot use 64-bit integers.
The latter is impossible to design IMO.

Alex.

---

From: Andy Green <[email-masked]>
Date: Sun, 5 Nov 2017 10:25:57 +0800

> the default size proposed by their language. "int" is signed 32-bit on the

So... those guys decided to use 32-bit internally.

...

> in my previous example, where "tonumber()" is provided by default on his

Yeah.  It needs some code to check a number string and see if it's 
bigger than what you can handle.  But it's not rocket science type code, 
is it?  What's the alternative, the server says "not my job to check if 
I can handle these numbers, mate" and just breaks or serves the first 
part of the file endlessly if some numbers come beyond what it can 
handle?  That's not really an OK situation for anyone.

I think it's OK if they want to say, "this server only deals with files 
smaller than 2G, that's what it is" or whatever, but it's the server's 
job to say no cleanly if it meets a situation eg, like Range: is beyond 
its limits to understand.  If the server doesn't take care about it, 
it's broken.

Putting it another way, more on the original topic, clearly there are 
two separate things here, practical limits on the numbers the internal 
server can express (which may be 64-bit), and judging if an expression 
of a number (which may exceed 64-bits, depending on what is decided) is 
inside the internal limits, without interpreting it.  Eg if it was 
length:MSB_data like bignum, if he see's a length coming > 4 he can 
judge it's too big without having to interpret anything else.  If it was 
==4 he can look at the top bit of the MSB.

If it was decimal ASCII numbers coming, eg, his limit is 2147483647 he 
can check how many digits and ban it if > 10, accept it if <=9, and 
progressively check the digits against "2147483647" if == 10 to judge it.

So maybe that should be recognized, that things wanting to use this new 
standard may choose not to be able to internalize the values that come, 
but they must all be able to judge them as able to be internalized or not.

> one applying safe processing there.

Well, you are far too brave saying small systems with, eg, a 32GB uSD 
connected, as ESP32 has the IO for, have no use for files > 2GB.  Even 
in the cases where they decide their limit, the implementation is 
responsible to cleanly enforce it.

> no need to deal with complex code compatible with 64-bit quantities to

My own code doesn't seem to lack bugs and people are using and shipping 
versions of it years old.  Imperfections and misunderstandings are 
everywhere.  But for guys looking at writing up a new way to express 
HTTP header values, which IIUI is the scope here, so what?  It has 
always been so and always will be so.  A spec that will be relevant 
perhaps for decades should be guiding everyone to a good clean place.

> care about this. While I too am fond of standards compliance, I'm disgusted

I agree there is a lot of that about around here, but it really isn't 
that I just don't understand where you are coming from... I likely 
understand where you are coming from at least as well as you do... eg, I 
wrote this and the related ESP32 stuff in lws from scratch, and it now 
supports H2 on ESP32

https://github.com/warmcat/lws-esp32-factory

> of their usage scope that we'll see any progress made in this area!

I think this business about being able to judge a number vs internalize 
it is a more fundamental and useful way to look at it.  Maybe there's 
some way to express the protection you hope for in the way that could 
eventually be defined.

-Andy

---

From: Matthew Kerwin <[email-masked]>
Date: Sun, 5 Nov 2017 14:15:00 +1000

On 5 Nov. 2017 12:27 pm, "Andy Green" <[email-masked]> wrote:

So maybe that should be recognized, that things wanting to use this new
standard may choose not to be able to internalize the values that come, but
they must all be able to judge them as able to be internalized or not.

Those who want to use this new standard are authors of new header field
values, not implementers (who have to honour the header field's definition
no matter how it's specified.)

And not all new header fields need to use this standard.

Cheers
-- 
Matthew Kerwin

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 5 Nov 2017 08:03:49 +0100

On Sun, Nov 05, 2017 at 10:25:57AM +0800, Andy Green wrote:
> Yeah.  It needs some code to check a number string and see if it's bigger
> than what you can handle.  But it's not rocket science type code, is it?

It's much worse than explaining in the spec how an integer must safely be
parsed so that these popular frameworks adopt and provide the method.
Typically they would provide the equivalent of this :

    /* returns <0 on parse error */
    int http_to_int(const char *str, int *value);

Then it's easier to deal with overflows the "normal" way in the parser
(ie: check for sum > 214748364 before multiplying then check for sum < prev
after adding the digit), which emphasizes the importance of having a single
parser for integers.

> What's the alternative, the server says "not my job to check if I can handle
> these numbers, mate" and just breaks or serves the first part of the file
> endlessly if some numbers come beyond what it can handle?  That's not really
> an OK situation for anyone.

I never said this at all, if you re-read what I wrote, I said that it's OK
to send this but that implementers must be aware that some implementations
will not support it, so when they have an alternative and are seeking for
interoperability, it's better to use the alternative. A good example is the
chunk size. Emitting chunks larger than 2GB doesn't provide much benefit.
Similarly, identifiers should not be specified as integers if they're going
to be larger than 2^31-1. And with just a few such careful measures we'll
quickly find that some implementations are totally interoperable on the
send side because they never manipulate large quantities at all, just like
other implementations will be totally safe on the receive side because they
will take specific measures to deal with the rare important values they care
for and will be OK. Just like my ESP doesn't need to parse large content
lengths advertised in POST since it will not be able to process them anyway.
However if it can detect the value is too large and report a 413 entity too
large, it suddenly becomes safe and interoperable.

> smaller than 2G, that's what it is" or whatever, but it's the server's job
> to say no cleanly if it meets a situation eg, like Range: is beyond its
> limits to understand.  If the server doesn't take care about it, it's
> broken.

I think we're saying the same thing.

> Putting it another way, more on the original topic, clearly there are two
> separate things here, practical limits on the numbers the internal server
> can express (which may be 64-bit), and judging if an expression of a number
> (which may exceed 64-bits, depending on what is decided) is inside the
> internal limits, without interpreting it.  Eg if it was length:MSB_data like
> bignum, if he see's a length coming > 4 he can judge it's too big without
> having to interpret anything else.  If it was ==4 he can look at the top bit
> of the MSB.

Yep.

> If it was decimal ASCII numbers coming, eg, his limit is 2147483647 he can
> check how many digits and ban it if > 10, accept it if <=9, and

That's exactly my point and what I'd like clearly stated in the spec. We
know that implementers do implement exactly what is specified when it's
easy, otherwise they seek something "surely good enough" which already
exists.

> I wrote this and the
> related ESP32 stuff in lws from scratch, and it now supports H2 on ESP32
> 

Oh nice, it's cool to know that H2 is ported to small devices since we've
been careful to keep this possible during the H2 design, eventhough there's
"a lot" of RAM on ESP32 (520 kB) :-)

> I think this business about being able to judge a number vs internalize it
> is a more fundamental and useful way to look at it.

I think we agree on this (and you probably formulate it better than I do).

> Maybe there's some way
> to express the protection you hope for in the way that could eventually be
> defined.

I think that describing a safe and portable integer parser, and a number of
very common limits that senders should avoid to cross whenever possible or
with a good reason will be enough for the spec.

Willy