Subject: [Acme] Proposed changes to make use of JSON in layered fashion.

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 19 Apr 2016 14:51:27 -0400

In the meeting, I proposed that we make the use of JSON in ACME
something that can be easily shared across multiple Web Services.

In a few years time we are quite likely to have multiple JSON Web
services that people want to use together. Implementing them as a
suite is going to be easiest if there is a consistent style. Some of
the encoding choices in ACME right now make implementation in
statically typed languages rather less convenient than they could be.
In particular it is really difficult to parse data streams when the
type descriptor for an object is a field inside the object that can
occur last rather than being first on the wire.

Looking at the impact of the changes in the spec, I don't think the
changes should delay things as I think they actually make the document
easier to read. At the moment there are things in sections 5 and 6
that seem in the wrong place. Promoting the "type" field to be the
name of the thing makes section 7 a lot clearer.

I can make the edits myself of course. But thought I would check up on
the nature of the edits first and see if my approach was acceptable.

I see the following areas needing change:

5.3.  Request URI Type Integrity

Reword to specify the mapping of requests onto JSON messages, each
request is a single object consisting of one field whose tag specifies
the request type and whose value is an object containing the request
data.

Move table from here into section 6. Because section 5 should specify
the layering, it should be ACME agnostic. Later on, this is the
section we will pull out and build on as a re-usable element in other
protocols (e.g. LURK).

5.X  Discuss GET vs POST

While we are about it, there should be some grounding in section 5 for
the GET/POST distinction made in section 6.

I think the document as a whole would be cleaner if these were set out
as two options in section 5 then section 6 describes which are
assigned to which.

6.  Certificate Management

One of the things I found very confusing when trying to implement was
that there are tables in 5.3 and 6.1 that seem to have the same
information but obviously don't as one has eight entries and the other
has six.

I think these need to be merged.

Example now becomes:

{ "new-reg"  {
     "contact": [
       "mailto:[email-masked]",
       "tel:+12025551212"
     ],
     "agreement": "https://example.com/acme/terms",
     "authorizations": "https://example.com/acme/reg/1/authz",
     "certificates": "https://example.com/acme/reg/1/cert",
   }}

6.1.2 Authorization Objects

   The structure of an ACME authorization resource is a JSON
object with exactly one field that specifies resource type and whose
value is an object containing the resource parameters.

   {
     "status": "valid",
     "expires": "2015-03-01T14:09:00Z",

     "identifier": {
       "dns" : {
          "value": "example.org"}
     },

     "challenges": [
       {
         "http-01" : {
         "status": "valid",
         "validated": "2014-12-01T12:05:00Z",
         "keyAuthorization": "SXQe-2XODaDxNR...vb29HhjjLPSggwiE"
       }}
     ],
   }

7.  Identifier Validation Challenges

Here, just delete the 'type' fields and instead put the type value in
the section headings.

---

From: Ron <[email-masked]>
Date: Thu, 21 Apr 2016 07:55:51 +0930

Hi Phillip,

> occur last rather than being first on the wire.

I was one of the remote people who thought several of the things you
said at the meeting had merit and were worth discussing further here,
so thanks for writing this up.

But I also thought you were talking about something more than what you
have proposed here now - so could you perhaps clarify a little - did
some of us read more into what you were saying than you actually had
in mind, or have you backed away from some of those suggestions after
further discussion after the meeting?

In particular one of the things that I thought seemed important was
moving _more_ of the protocol into the JSON.  Right now we have a
bit of a mish-mash where part of the data you need to complete some
task is in the JSON, and part of it is squirrelled away in the HTTP
headers.  And there's a few places at least where that creates real
problems (like where an issued certificate might have more than one
valid issuer chain, just to pick one that JSHA and I had looked at).

I got the impression you were hoping to make this more transport
agnostic (which seems like a useful and fairly easy thing to do if
we are going to change things to the degree you are talking here).

> name of the thing makes section 7 a lot clearer.

That's not without its own cost though, but more on that below ...

> the request type and whose value is an object containing the request

I understand the idea of what you are aiming for with this sort of
change - but the idea of a protocol full of JSON objects that are
mandated to strictly have only one field worries me a bit.

It seems more like an admission that "JSON is wrong way to encode
this" (or perhaps "I'm using the wrong tool to decode JSON") than
a genuine advantage to the encoding.  And I'm not of the opinion
that JSON is the wrong format to encode this in.

JSON objects can have multiple fields, and those field are unordered.
I don't think you can really escape that, and I don't think we can
reasonably "fix" that, without inventing something that "Isn't JSON".

There may be smarter ways we can arrange some of these structures,
for good reasons, but I'm not sure that reinventing ASN.1 in JSON is
a plan I'd happily endorse.

And I do think there *are* ways that we need to rearrange some of
these request structures.  In particular, overloading "reg" to both
make requests and request changes is troublesome.  I think it needs
to be split into separate query and change resources.

The obvious problem case that evidenced this is in testing of a real
implementation is that if you send:

 { "resource": "reg", "delete": true }

to the current Boulder implementation, it will happily return a 202
status, with the current registration resource as if you'd submitted
a "trivial payload" to actually query that state - with no indication
that something went wrong and the request you actually issued had
failed.

I think that's a serious impediment to ever being able to extend the
registration resource functions in any backward compatible way and
we should address that now while we still have the chance.  A request
to change something should never be misinterpreted as a request to do
something entirely different to that "successfully".

>      "certificates": "https://example.com/acme/reg/1/cert",

I agree there's several places in the text where you either need to go
back and forth to get the full description of something, or where the
examples appear (or are) conflicting.  I think I noted some of those
in the nits I posted earlier, and agree they should be fixed, but that's
mostly editorial, not an actual change to the spec.

> Here, just delete the 'type' fields and instead put the type value in

The example you give here highlights the concern I expressed above.
In:

  "identifier": { "dns": { "value": "example.org"} },

by getting rid of the "type" field, you may have made things easier
for your parser (while it is hardcoded with *all* of the valid field
names that might appear in an "identifier" object) but you've in turn
made it harder for everyone else to know what to do if they don't
recognise that field name, and difficult or impossible for any future
extension to that structure to occur.

My code can no longer just go 'there's a value in the type field that
I don't recognise, but I know what "type" means and I can safely just
ignore that'.  Now I have a field name that I don't recognise, or have
any idea what to do with, and have no "critical" flag to tell me if
I can safely ignore that or not.

And I have a structure to parse with lots of redundancy.  Why have
"dns" be an object that itself only has one field in it?  Why bother
to wrap it in "identifier" if "dns" defines it as an identifier ...

It wouldn't be impossible to fix those problems, but if the fix for
them is to define lots of objects that may only have one field, then
I don't really see that as being an improvement, and this isn't
really JSON anymore, it's more like Yet Another Text Representation
Of ASN.1, which I think would burn bridges for future extension that
we'd be better off not burning.

There may be a way to get what you want and have this still be JSON
with all the nice properties that come with that, but I don't think
the above example is it.

You still *have* to parse the JSON text to ensure it is valid JSON
before you can make any assumption about what "dns" actually means
or whether the data that follows it can safely be stuffed into a
static data type.  And once you've done that, you have the "type"
field to know what the static type is ...

I do think there is some scope (and real need) to tweak some of
these structures, and I have a lot of natural sympathy for anything
which would make them easier to parse correctly and safely in a
broad range of code, but I think the above change just trades one
set of problems for a different (and larger) set of them and pushes
the problem from being a relatively widely solved one ("how do I
parse JSON") to being one that every implementor will need to code
logic for ("how do I handle unknown _fields_, and not just unknown
values in them").

But maybe I'm still just missing something about everything you have
in mind here, so I'm keeping an open mind still, and do think there
are other very good reasons we might tweak some of these structures.

  Cheers,
  Ron

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 21 Apr 2016 17:55:44 -0400

> further discussion after the meeting?

Yes and no. When I originally got the speaking slot, 'submit to last
call' was not on the agenda. So the proposal I am making is slightly
more modest.

If we were 6 months out from last call then I would absolutely propose
splitting the draft into two documents, a platform document and an
ACME protocol document. In fact I already have such a draft:

https://tools.ietf.org/html/draft-hallambaker-json-web-service-02

But I don't want my proposal to be critical path for last call. And so
what I have done instead is to implement my version of ACME according
to the layered model I describe and work out what the minimal changes
that would be necessary to make ACME consistent.

What I would like to avoid is what hit use with SAML where the people
who insisted on one approach to layering in Version 1.0 discovered it
didn't really work and it forced us to a Version 2.0. Ooops.

> headers.

I am trying to work out where things are happening, right now it looks
as if we have things happening in the HTTP headers that are not
actually covered by the authentication at all. Or that could just be
me misreading the draft, either way it is confusing.

> And there's a few places at least where that creates real

That is still my objective. Though I agree that the first cut might
actually be more of a 'discover bits of the spec that were assumed a
bit too much'.

> that JSON is the wrong format to encode this in.

I don't see that as being the case. Rather we have a series of
protocol messages, each protocol message is of a specific type. We
should state the type in JSON. So my ideal would be:

"tag" : { ... object ... }

But it seems to be easier to interface these to JSON support code
wrapped up as an object.

{ "tag" : { ... object ... } }

Now in a future version of the spec, I could well imagine that we
might want to relax that one field requirement so that one
authentication could cover two or more messages or we might have
arrays of objects instead of singletons. So say we were requesting a
thousand device certs at a time, we could do that.

So I am saying that for purposes of ACME/1.0 we only need singleton
requests and responses.

> reasonably "fix" that, without inventing something that "Isn't JSON".

Or you can use nesting because the tag always precedes the value.

> a plan I'd happily endorse.

I see ASN.1 as a terrible implementation of a good idea. The idea of
having one data encoding is good. The one they chose is horrible and
DER worst of all.

> something entirely different to that "successfully".

I think that is spot on and comes up to 'criticality', one of those
issues that people hate to raise because people have misunderstood it
so badly in the past. I am pretty sure that a large fraction of those
BULLRUN dollars were spent muddying the water with chatter about
criticality.

There are two ways that you might want a legacy application to react
when faced with an extension:

1) Ignore the additional fields and continue as normal
2) Reject the request, do not attempt to process it.

This is of course what the X.509 'CRITICAL' bit is intended to
achieve. But people misinterpreted 'CRITICAL' and meaning 'important'
and have as a result come up with idiotic specs that require CAs to
cause legacy systems to break just for the hell of it. And probably
not coincidentally, that idiot spec delayed deployment of a scheme
that could have blocked the FLAME attack on Microsoft's internal CA.

When we faced this issue in SAML, I had to disguise the criticality
bit as 'Conditions' so that folk didn't complain.

Sometimes the desired legacy behavior is to have it reject the
request, sometimes it is to carry on. This maps very nicely to the
nested JSON scheme:

* To cause additional fields to be ignored, keep the object type
identifier tag the same.
* To cause legacy software to reject, change the object type identifier tag

> extension to that structure to occur.

The rule 'ignore field names you don't understand' seems to fit the
JSON approach. If a field is mandatory, you need to understand it.

Lets say that we want to introduce a new identifier type.

"identifier": { "new": { "value": "example:id"} },

The service can't understand the identifier type that it needs to
understand to process the request, so the response here is 'I don't
support that request'.

If however we wanted to add some additional information into the
identifier, such as a DNSSEC trust root (no, I don't know why you
would, pretend), we have:

 "identifier": { "dns": { "value": "example.org",
     "dnssec" : "wh2iiy24iy2iuwrywiuy3riu2h3riuh=="} },

Legacy services can simply ignore the additional information.

I think this works very nicely with the challenge types. I can imagine
us adding lots of additional fields over time that are not intended to
break backwards compatibility. But we do have that option if it is
needed.

> I can safely ignore that or not.

So we need to add the above explanation into section 5 and then
identify the areas where we might want the option of breaking legacy
systems rather than suffer their best efforts.

> to wrap it in "identifier" if "dns" defines it as an identifier ...

Question is whether this is an extension point or not. If it isn't
then we can remove the hierarchy.

  { "blah" : "....",

becomes

  { "blah" : "....",
    "dns": { "value": "example.org"} },

> we'd be better off not burning.

I think we just need to understand where we want to put extension
points. Right now, anywhere where we have a "type" tag approach seems
right.

ASN.1 certainly has a terrible approach to extensibility. The scheme I
outline looks to me as if maintains the JSON approach of 'add fields
at will' while still having enough mechanism there to be able to tell
legacy services when they should not attempt to do their best with
stuff they do not understand.

---

From: Ron <[email-masked]>
Date: Tue, 26 Apr 2016 09:12:00 +0930

> that would be necessary to make ACME consistent.

I'm less sure about splitting the document at this stage (if there are
real reuse points to be had elsewhere a separate document can still
be produced later) - but I also don't think this is the biggest blocker
for moving to last call any time soon ...

Right now we have exactly zero implementations of the -02 draft (I have
an almost complete client implementation, but there is no server that
I can test it with nor am I aware of other client implementations), and
we do have a number of known troublesome issues with the current draft,
some of which moving more of this into the JSON would help to address.

(Some of which I've already raised here, some which I still have queued
up since there's no point flooding this list with them now while the
first lot are still unresponded to and while I'm still running into more
of them while polishing and testing the implementation I have).

> didn't really work and it forced us to a Version 2.0. Ooops.

Even without the 'layering' changes you are proposing here, we already
have a number of "oops it doesn't really work" things to address still.

I'd love to see this go to last call as soon as possible.  But rushing
that doesn't seem wise when we still have both implementation and
security model issues shaking out that are yet to be resolved.

> me misreading the draft, either way it is confusing.

Well, they are covered by the TLS connection to the server.  And the
server currently does not sign any of the other content it returns
separately to that.  Only client submissions contain protected content.

The more immediate problem is we have things like Link headers being
used to indicate where to obtain issuer certificates from, which are
both underspecified, and it would seem, insufficient for that task in
general.

> actually be more of a 'discover bits of the spec that were assumed a

Right now, boulder only implements part of the -01 draft, and some
things differently to how the -01 draft says they should be ...

Since migrating it to -02 (let alone -03) is going to be a notable
transition (and will surely break many of the extant clients that
don't actually implement the spec at all but rather some approximation
of it RE'd from what the LE client did), I think it's worth spending
the time to get as much of this right as we already know is troublesome
before it starts that transition, otherwise we're just going to force
two (or more) transitions on the existing users ...

I'm certainly happy to propose and write up things based on what I've
learned from my implementation of this - but it would be nice to get
a little more feedback on some of the things I'd already raised here ...

I don't want to spend a lot of time on that unless we have some at
least in principle agreement about the problem(s) and preferred
solutions from the document authors.

> thousand device certs at a time, we could do that.

The problem I see with what you've proposed here (based on the specific
examples you gave), is that you're just shuffling the unknowns around,
and not in a way that I see improving the general robustness.

If I have something like:

 "field": { "type": "x", "whatever": ... }

Then the spec can concretely define what "type" means, what the
allowable grammar for its value may be, and what I should do if my
code doesn't recognise a value it has.  It likewise knows that if
I don't see the mandatory fields then the object is invalid.

If I instead have something like:

 "field: { "randomtag1": {...}, "randomtag2": ... }

Then I have no context at all to know what I should do with tags
that I don't already have hardcoded.  And the solution to that
requires either only have strictly one member of "field", so that
I know "randomtag1" is implicitly the "type" field, or having a
separate 'critical' flag (which experience says nobody will respect
or implement correctly anyway) and makes most parsers more complex.

> So I am saying that for purposes of ACME/1.0 we only need singleton

I think if we _require_ that, then this is no longer JSON and we
lose a lot of the nice advantages using JSON gives us.  And if we
don't require that, then we're introducing nastier problems than
it it solving ...

> having one data encoding is good. The one they chose is horrible and

Don't get me wrong, ASN.1 and TLV in general is fine for what they
were designed for, carrying extensible binary data where bandwidth
is limited.

But this isn't ISDN, and we're using a text protocol, which means
you *have* to parse and validate the text *before* you can even think
about deciding whether what is in it is valid or what it means.

We don't need to compress implicit meaning into the field names,
because we can have explicit fields, that all parsers must understand,
which make any extensions explicitly distinguishable from transmission
errors.

I think I'd actually like to understand why your system has trouble
with parsing this, because I'm sure there's an interesting clue there
somewhere, but I'm less sure it's a sign of trouble in the parts of
this protocol you gave as examples.

> bit as 'Conditions' so that folk didn't complain.

I think in the example I gave above, the problem is much simpler,
(and one that I worry your syntax proposal might make worse :) ...

We've overloaded "reg" to mean more than one thing (both query and
modify), and the extension made which meaning was intended ambiguous.

I don't think we need a critical flag to fix that case, we just need
the request to do One Thing, unambiguously.

If "reg" only meant "modify this registration object", then the case
above could have just returned an "I didn't modify anything" status
even if it didn't recognise "delete", which wouldn't be misleading
at the client as to what happened.

I think we can actually build "criticality" into the protocol quite
reasonably here without needing a separate flag.  For instance:

A "modify reg" request should quite reasonably be rejected if there
are fields which are not understood.

An unrecognised identifier "type" value can safely be "ignored" -
which may still lead to failure of the operation if no recognised
type is also included - but explicitly marks the mandatory field(s)
where optional extensions may be declared.

> * To cause legacy software to reject, change the object type identifier tag

I think we can do that without the nesting, and I don't think the
nesting adds anything in general to avoiding this problem.

In many languages, JSON maps neatly to native data structures,
so a JSON object is trivially imported into a hash (or dictionary
or whatever your pet language calls them).

This makes looking for mandatory fields with well known names (like
"type") and deciding whether you recognise the value both trivial
to do and trivial to audit.

If we start telling people you have to iterate over every hash key
of every structure you receive, and reject if there is an element
that you don't recognise - then we're probably going to fail at
this, because nobody is going to do that in practice.  Their code
is just never going to notice an element they aren't expecting.

We might get away with insisting on that for one or two critical
structures, but in general, if we want this to be widely implemented,
then if we want to extend something that can't be safely ignored, we
need to use a new indicator value in a well known mandatory field
like "type" or "resource".

That's the only way it will be noticed by "the simplest code", which
is what (some/many/most) people will write whatever we do.

> JSON approach. If a field is mandatory, you need to understand it.

I think where we disagree here is on the "bootstrap problem".

You are proposing that adding any new type immediately becomes a
field name that existing implementations won't recognise (and in
practice will probably never even notice is there).  That makes
it harder than it needs to be to ever add new mandatory elements
because the "natural" behaviour of existing code will be to just
ignore them.

I'm thinking more along the lines of, we start with a fixed set
of mandatory fields, and a defined set of values they can have.
In that model, when we can define which of the mandatory field
values an implementation MUST recognise (like "resource"), and
which of them may contain values that MAY be ignored (like the
identifier "type") -- which is basically what the existing spec
does, modulo a few glitches we should iron out in it.

> break backwards compatibility. But we do have that option if it is

I don't really see the advantage of that over, say:

 "identifier": { "type": "dns", "value": "example.org",
                 "dnssec": "wh2iiy24iy2iuwrywiuy3riu2h3riuh==" },

You still need to parse all of the *text* of "identifier" before you
can assume anything contained in it is valid or means what you think
it means.  Once you've done that, you have "type" and can use the
content to fill your static data types, without needing to recurse
through multiple objects to get at it.

The difference is, we can mandate up front that "type" MUST exist,
and how you must deal with any particular value in it (known or
unknown).

>     "dns": { "value": "example.org"} },

I'd even be thinking:

 { "blah": "....", "dns": "example.org" }

If you are going to mandate the "one field" thing and that you must
understand what "dns" is, then "value" alone in an object adds nothing.

But this is why I think we're better off mandating "identifier" as a
well known and well defined field, with "type" as the mandatory field
that lets you recognise this is a valid identifier object and lets you
know if you support that type of identifier or not.

In your example above, if the new dnssec field was optional, then it
could still use "type": "dns" - and if it was mandatory, we'd add a
new "type": "dnssec" and define that requirement in the IANA spec for
that new type.

> points. Right now, anywhere where we have a "type" tag approach seems

Yes, I think we can handle 'criticality' by defining whether an
unrecognised type can be ignored or should be cause for rejection
for each object where that is needed.

> stuff they do not understand.

I might still be missing something that you're _thinking_ (though I
think I understand well enough what you've _said_ and the examples
you have given) - but my understanding of the problem you are trying
to solve with this syntax change is that your code has trouble with
deciding the type of something unless that is encoded implicitly in
its field name.

My biggest problem with the solution to that you are suggesting is
that it takes away the ability in general to know the type of something
(and whether it's a valid object in the general case), by replacing
well known defined fields which carry that information explicitly, with
arbitrary field names that may or may not contain it implicitly,
depending on whether the field name is valid but unrecognised, or just
line noise corruption ...

If I see a "type" I don't recognise in an otherwise valid object, then
I can report to the user "Unknown identifier type 'blah'" and give them
some useful indication of what to go look for.

If I can't distinguish that from a simply corrupted structure, then all
I can tell them is something more like "Error 184276, corrupt data", and
OpenSSL already inflicts enough of that sort of thing on them to want
more of it where it is easily avoidable.

I would like us to have a good look at the current JSON objects, and
what in them is and isn't mandatory and the failure modes, and what is
still underspecified, and what other data belongs in the JSON rather
than in the transport metadata.  I think that's an important step
(along with getting more actual implementation experience) before we
can serious consider pushing this spec to last call.

I'm not seeing the concrete changes you propose above directly address
that (yet) though.  I think it would be helpful if you more directly
directly describe the *problem* your code has with the existing syntax
though.  So far I'm seeing a proposed solution to it, and some
rationalisation for that, but I'm not seeing the compelling benefit
that outweighs the problems it in turn introduces in other aspects.

This isn't a problem in general with parsing JSON into static data
types, so there's some other more specific issue if this is giving
you headaches trying to implement it in your framework, and it would
be nice to identify that clearly if we're going to try to accommodate
it better.

Basically, I don't think you can win at the sort of "layering to
define type" methodology you are proposing in this case, there is
always going to be some point in parsing JSON where you don't really
know the high level 'type' of something until you've first parsed the
text (and I say this as someone who is a fan of static typing and has
written code to mmap TIFFs into static structures to parse them ... ).
That isn't a model JSON fits into well, and I think the real choice
here is either "Don't use JSON", or "Work to the strengths of JSON"
and the latter option seems the better one to me at present.  That's
the part where I see we are falling short by splitting some of the
protocol into the transport layer.

  Cheers,
  Ron