Subject: [httpapi] Discussion about JSON payloads and code generation

From: Christoph Kappestein <[email-masked]>
Date: Sat, 16 Jan 2021 14:14:19 +0100

Hi all,

I would like to gauge whether there is some interest in discussing ways to
describe JSON payloads with a focus on code generation on this WG. Also it
would be interesting to gather feedback from other developers who have
experiences in this space.

I have wrote some thoughts about this topic in a separate blog post, if you
are interested you can checkout the post at:
https://chriskapp.medium.com/discussion-about-json-payloads-and-code-generation-8d60bc8fd94e

Please let me know what you think.

best regards
Christoph

---

From: Darrel Miller <[email-masked]>
Date: Sat, 16 Jan 2021 22:12:35 +0000

Hi Christoph,

From: httpapi <[email-masked]> on behalf of Christoph Kappestein <[email-masked]>

I would like to gauge whether there is some interest in discussing ways to describe JSON payloads with a focus on code generation on this WG. Also it would be interesting to gather feedback from other developers who have experiences in this space.

As an editor of the OpenAPI spec, that relies heavily on JSON Schema, consider my perspective to be heavily biased here. But I do agree wholeheartedly that JSON Schema has challenges when it comes to describing types for code generation.  However, I would be reluctant to go down a path of defining a new specification for describing types.

The recent Postman developer survey https://www.postman.com/state-of-api/ found that in their 13,000 respondents, 75% of them claimed to be users of JSON Schema.  Convincing developers to move to a different format is going to require a compelling alternative.

What would be awesome to see is if a community could form around creating a new JSON Schema vocabulary that targets type description.  JSON Schema itself is now defined by a set of vocabularies, one of which is the validation vocabulary.  I know the JSON Schema core team have been looking for someone to spearhead this effort.   https://github.com/json-schema-org/json-schema-vocabularies

By taking this approach, the existing JSON Schema specifications and artifacts can be leveraged, and the new type definition keywords could be used alongside the validation keywords.

Darrel

---

From: Christoph Kappestein <[email-masked]>
Date: Sun, 17 Jan 2021 12:46:21 +0100

Hi Darrel,

Thank you for the feedback. I understand that it is not easy possible to
change the schema specification but I had the hope that the OpenAPI spec
will be in future versions more JSON schema independent. I.e. that a user
could use XSD, JTD, Protobuf or any other specification to describe the
request/response payload, also for payloads in the future which are not
JSON. So that a user is not forced to use JSON schema and that an
alternative could emerge.

A core editor from the JSON schema spec has already approached me regarding
a vocabulary, but from my point of view I think that those vocabularies
inherit many properties from JSON schema which are not ideal for code
generation. In the spirit of "do one thing and do it well" we could have
different specifications for specific use cases, which would be a much more
flexible framework than forcing every schema to be a vocabulary. A
generator could then support only specific schema types. If I remember
correctly the RAML specification solved this also in a similar way. This
approach would be also great for the OpenAPI since they do not have to
force a different schema spec but the users can decide which is the optimal
solution.

Thanks to the link of the API report, I was not aware of it and it looks
really interesting. Regarding the OpenAPI spec, for me it would be really
interesting to know whether there is such a plan to make the schema
description part more JSON schema independent?

best regards
Christoph

Am Sa., 16. Jan. 2021 um 23:12 Uhr schrieb Darrel Miller <
[email-masked]>:

> Hi Christoph,
>
> *From:* httpapi <[email-masked]> on behalf of Christoph
> Kappestein <[email-masked]>
>
> As an editor of the OpenAPI spec, that relies heavily on JSON Schema,
> consider my perspective to be heavily biased here. But I do agree
> wholeheartedly that JSON Schema has challenges when it comes to describing
> types for code generation.  However, I would be reluctant to go down a path
> of defining a new specification for describing types.
>
> The recent Postman developer survey https://www.postman.com/state-of-api/ found
> that in their 13,000 respondents, 75% of them claimed to be users of JSON
> Schema.  Convincing developers to move to a different format is going to
> require a compelling alternative.
>
> What would be awesome to see is if a community could form around creating
> a new JSON Schema vocabulary that targets type description.  JSON Schema
> itself is now defined by a set of vocabularies, one of which is the
> validation vocabulary.  I know the JSON Schema core team have been looking
> for someone to spearhead this effort.
> https://github.com/json-schema-org/json-schema-vocabularies
>
> By taking this approach, the existing JSON Schema specifications and
> artifacts can be leveraged, and the new type definition keywords could be
> used alongside the validation keywords.
>
> Darrel
>
>
>

---

From: Jason Desrosiers <[email-masked]>
Date: Sun, 17 Jan 2021 12:40:50 -0800

Hi Christoph,

Even if/when OpenAPI allows you to use something other than JSON Schema, I
think you still have a serious problem. The problem with "do one thing and
do it well" in this case is that you end up with a lot of duplication. I
don't want to have to maintain separate and almost identical sets of
schemas for code generation, validation, documentation, etc. Ideally, you
would start with a simple and flexible core that other specifications can
build off of to progressively enhance the core. That's what we're trying to
enable with JSON Schema vocabularies.

I believe it's possible to define a JSON Schema vocabulary for code
generation that progressively enhances standard JSON Schema so only one
schema needs to be maintained. I say that with all due humility and
willingness to wrong. I have never attempted to implement a JSON Schema
based code generator, but I have written other tooling with this kind of
thing in mind. In our Github Issue, I'll share more details about how I
think the problems you identified could be solved without breaking JSON
Schema validation.

Full disclosure: I am a member of the JSON Schema core team, so I should be
considered even more biased than Darrel :-). But, even if you decide not to
align with JSON Schema, I'd be happy to share lessons we've learned from
developing JSON Schema that might apply to TypeSchema as well.

Jason Desrosiers

On Sun, Jan 17, 2021 at 3:46 AM Christoph Kappestein <
[email-masked]> wrote:

> A core editor from the JSON schema spec has already approached me
> regarding a vocabulary, but from my point of view I think that those
> vocabularies inherit many properties from JSON schema which are not ideal
> for code generation. In the spirit of "do one thing and do it well" we
> could have different specifications for specific use cases, which would be
> a much more flexible framework than forcing every schema to be a
> vocabulary. A generator could then support only specific schema types. If I
> remember correctly the RAML specification solved this also in a similar
> way. This approach would be also great for the OpenAPI since they do not
> have to force a different schema spec but the users can decide which is the
> optimal solution.
>
>> I would like to gauge whether there is some interest in discussing ways
>> to describe JSON payloads with a focus on code generation on this WG. Also
>> it would be interesting to gather feedback from other developers who have
>> --
> httpapi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/httpapi
>

---

From: Darrel Miller <[email-masked]>
Date: Sun, 17 Jan 2021 21:17:28 +0000

Christoph,

From: Christoph Kappestein <[email-masked]>

Thank you for the feedback. I understand that it is not easy possible to change the schema specification but I had the hope that the OpenAPI spec will be in future versions more JSON schema independent. I.e. that a user could use XSD, JTD, Protobuf or any other specification to describe the request/response payload, also for payloads in the future which are not JSON. So that a user is not forced to use JSON schema and that an alternative could emerge.
There is a proposal for supporting an "Alternative Schema" https://github.com/OAI/OpenAPI-Specification/blob/master/proposals/001_Alternative%20Schema%20Proposal.md

Implementation feedback was that that the current proposal is too complex to implement, so we likely will be revisiting it once OpenAPI 3.1 is released.

A core editor from the JSON schema spec has already approached me regarding a vocabulary, but from my point of view I think that those vocabularies inherit many properties from JSON schema which are not ideal for code generation. In the spirit of "do one thing and do it well" we could have different specifications for specific use cases, which would be a much more flexible framework than forcing every schema to be a vocabulary. A generator could then support only specific schema types.

Speaking purely as an individual, I think extending JSON Schema would be a better first step.  Getting traction in this space is hard, and successfully mapping to the type systems of multiple platforms is a problem that people have been failing at for decades, no matter how focused the effort was.

Having said that.  I wonder if this working group would be the right place to create a specification that describes an extended set of primitive data types and corresponding text-based wire representations.  JSON Schema completely punts on anything beyond what JSON defines for type, and format is left as implementation dependent.

I could imagine this kind of standardization would be helpful for query parameter values, header values as well as a range of media types used as payloads.

I'm thinking of a single RFC that describes a set of serializations used in HTTP APIs. Things like date-time, times, durations, currency, coordinates, language identifiers, etc. It could also provide a standardized type identifier to use in complex type definition/validation specifications

I think it would be useful when writing specifications to be able to reference a single spec and say primitive values are serialized as per RFCXXXX.   Looking at the structured headers spec, they have done this work for a few core data types that are common in headers.  https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#section-4.1.3.1

So, bringing this back to TypeSchema.  Do you think that it would be useful for TypeSchema if there existed a standardized rich set of primitive data types for HTTP APIs that were platform independent and optimized for wire representations?

I'm pretty sure the JSON Schema folks would be delighted if there were some well known values that could be used with format keyword.  The OpenAPI specification defines a few formats (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.1.0.md#data-types) and we continue to get requests for more, but I'm not convinced the OpenAPI specification is the right place to do that. OData defines a bunch of their own https://docs.oasis-open.org/odata/odata-csdl-json/v4.01/odata-csdl-json-v4.01.html#_Toc38466396

The feels like something that if properly scoped and standardized could make life easier for a bunch of different efforts.

Thoughts?

Darrel

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Mon, 18 Jan 2021 11:16:48 +0100

søn. 17. jan. 2021 kl. 22:17 skrev Darrel Miller
<Darrel.Miller=[email-masked]>:

> In the spirit of "do one thing and do it well" we could have different
> specifications for specific use cases, which would be a much more flexible
> framework than forcing every schema to be a vocabulary. A generator could then
> support only specific schema types.

Ulysse Carion has already created a schema language specifically
tailored for code generation with RFC 8927.

https://www.rfc-editor.org/rfc/rfc8927.html

-- 
Asbjørn Ulsberg           -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: Christoph Kappestein <[email-masked]>
Date: Sun, 24 Jan 2021 10:46:02 +0100

Hi,

Thanks for the feedback.

@Jason this is true and I totally understand your concern. I think the
difficult
part is to define a "core" which works for every use case i.e. code
generation,
validation, documentation, etc. We should ask what is the common ground of
all
these specs? In JSON Schema this is probably only the part how documents and
types are referenced (JSON Path). But thanks for the offer and the great
feedback which you have provided, I will also give you an answer at our
github
issue, if I find the time finally ;)

@Darrel the idea sounds great to define the representation of data types. At
TypeSchema we have defined the following primitive types:
- date (rfc3339 full-date)
- date-time (rfc3339 date-time)
- time (rfc3339 full-time)
- duration (ISO 8601)
- binary (base64 encoded)

We have found that most programming languages have internal object
representations for these kinds of types. If we would have an official list
with
a fitting identifier for a type, then we could also use this identifier at
the
"format" in JSON Schema/TypeSchema to describe the type.

I am wondering whether it would be also possible to describe complex types
in such a specification, i.e. that we describe how a list, map or
object/struct
type could look and is defined through keywords. So I would totally support
this
effort, since this is basically the reason why we have developed TypeSchema
and
we would be happy to build upon such a specification.

best regards
Christoph

---

From: Darrel Miller <[email-masked]>
Date: Tue, 26 Jan 2021 03:31:37 +0000

Hi Christoph,

From: Christoph Kappestein <[email-masked]>
I am wondering whether it would be also possible to describe complex types
in such a specification, i.e. that we describe how a list, map or object/struct
type could look and is defined through keywords. So I would totally support this
effort, since this is basically the reason why we have developed TypeSchema and
we would be happy to build upon such a specification.

best regards
Christoph

If we added the constraint that complex types must be described using JSON primitives, then I could imagine enabling a registry of "well known" types, including complex types and their corresponding JSON serializations.

One thing that appeals to me about a registry of well known types, is that no particular serialization needs to win.  People can just pick the one they want to use.

Going back through the short history of this working group, we have the Linkset draft https://datatracker.ietf.org/doc/html/draft-wilde-linkset-07#section-4.2.3 that serializes a link as a map entry and Evert Pot's https://tools.ietf.org/html/draft-pot-json-link-02#section-2 serialization as an object.  I've seen scenarios where both formats make sense.  There is no reason that "link-relation-map-entry" and "link-relation-object" could not both be registered.

Darrel