Subject: [Geojson] Processing model

From: Allan Doyle <[email-masked]>
Date: Wed, 25 Nov 2015 14:37:17 -0500

I’m starting a new thread to talk about the GeoJSON processing model and my dislike of the language that’s trying to creep in.

GeoJSON is a format.

You can decide if you have valid GeoJSON by either reading the spec and mentally testing it, or someone can write a parser that reads a file and says “yes” or “no” to the question of whether it is GeoJSON or not. Better still, a linter that can give you hints about why it might not be.

If there is any language at all in the spec about a process needing to consume extension content and produce a file with the same content, how do you test that? You can’t run the same test as above. At best you have to run it twice, once on the input and once on the output. Then you have to compare the non-GeoJSON parts (the parts the test program doesn't even know how to parse because it’s not GeoJSON and the parser just knows about GeoJSON).

How do you do that?

And, in my world, for a format, *why* would you do that? There are times you want a pass-through and times you want a filter. Why preclude filters?

Give the format to the world, and let the world decide what to do with it. 

	Allan

---

From: Martin Daly <[email-masked]>
Date: Wed, 25 Nov 2015 19:52:08 +0000

> I’m starting a new thread to talk about the GeoJSON processing model and my
> dislike of the language that’s trying to creep in.
> 
> GeoJSON is a format.
> 
> You can decide if you have valid GeoJSON by either reading the spec and
> mentally testing it, or someone can write a parser that reads a file and says
> “yes” or “no” to the question of whether it is GeoJSON or not. Better still, a
> linter that can give you hints about why it might not be.
> 
> If there is any language at all in the spec about a process needing to consume
> extension content and produce a file with the same content, how do you test
> that? You can’t run the same test as above. At best you have to run it twice,
> once on the input and once on the output. Then you have to compare the non-
> GeoJSON parts (the parts the test program doesn't even know how to parse
> because it’s not GeoJSON and the parser just knows about GeoJSON).
> 
> How do you do that?
> 
> And, in my world, for a format, *why* would you do that? There are times you
> want a pass-through and times you want a filter. Why preclude filters?
> 

Word.

Martin

---

From: Sean Gillies <[email-masked]>
Date: Wed, 25 Nov 2015 14:18:36 -0700

Allan,

We can't assert that GeoJSON exists outside of any processing model because
GeoJSON is based on JSON and JSON has a processing model already. It's a
very minimalist one and I like it for that reason. This is it:
https://tools.ietf.org/html/rfc7159#section-9

9.  Parsers

   A JSON parser transforms a JSON text into another representation.  A
   JSON parser MUST accept all texts that conform to the JSON grammar.
   A JSON parser MAY accept non-JSON forms or extensions.

   An implementation may set limits on the size of texts that it
   accepts.  An implementation may set limits on the maximum depth of
   nesting.  An implementation may set limits on the range and precision
   of numbers.  An implementation may set limits on the length and
   character contents of strings.

10.  Generators

   A JSON generator produces JSON text.  The resulting text MUST
   strictly conform to the JSON grammar.

The I-JSON message format standard adds this (
https://tools.ietf.org/html/rfc7493#section-4.2):

4.2.  Must-Ignore Policy

   It is frequently the case that changes to protocols are required
   after they have been put in production.  Protocols that allow the
   introduction of new protocol elements in a way that does not disrupt
   the operation of existing software have proven advantageous in
   practice.

   This can be referred to as a "Must-Ignore" policy, meaning that when
   an implementation encounters a protocol element that it does not
   recognize, it should treat the rest of the protocol transaction as if
   the new element simply did not appear, and in particular, the
   implementation MUST NOT treat this as an error condition.  The
   converse "Must-Understand" policy does not tolerate the introduction

Bray                         Standards Track                    [Page 4]

________________________________

RFC 7493                The I-JSON Message Format             March 2015

   of new protocol elements, and while this has proven necessary in
   certain protocol designs, in general it has been found to be overly
   restrictive and brittle.

   A good way to support the use of Must-Ignore in I-JSON protocol
   designs is to require that top-level protocol elements must be JSON
   objects, and to specify that members whose names are unrecognized
   MUST be ignored.

I see this as nothing but positive and I want this expressed in the GeoJSON
draft. What objections do you have to that language?

I don't see any need to expand GeoJSON's processing model beyond the above,
or even to have a section of the spec titled "processing model".

On Wed, Nov 25, 2015 at 12:37 PM, Allan Doyle <[email-masked]>
wrote:

> I’m starting a new thread to talk about the GeoJSON processing model and
> my dislike of the language that’s trying to creep in.
>
> GeoJSON is a format.
>
> mentally testing it, or someone can write a parser that reads a file and
> says “yes” or “no” to the question of whether it is GeoJSON or not. Better
> still, a linter that can give you hints about why it might not be.
>
> If there is any language at all in the spec about a process needing to
> consume extension content and produce a file with the same content, how do
> you test that? You can’t run the same test as above. At best you have to
> run it twice, once on the input and once on the output. Then you have to
> compare the non-GeoJSON parts (the parts the test program doesn't even know
> how to parse because it’s not GeoJSON and the parser just knows about
> GeoJSON).
>
> How do you do that?
>
> And, in my world, for a format, *why* would you do that? There are times
> you want a pass-through and times you want a filter. Why preclude filters?
>

-- 
Sean Gillies

---

From: Martin Daly <[email-masked]>
Date: Wed, 25 Nov 2015 21:29:15 +0000

I’ll note, snarkily, that in neither case are the unknown parts expected to be *kept*. That makes them comparable to my wording in https://github.com/geojson/draft-geojson/pull/117, not any others.

Martin

From: GeoJSON [mailto:[email-masked]] On Behalf Of Sean Gillies
Sent: 25 November 2015 21:19
To: Allan Doyle <[email-masked]>
Cc: [email-masked]
Subject: Re: [Geojson] Processing model

Allan,

We can't assert that GeoJSON exists outside of any processing model because GeoJSON is based on JSON and JSON has a processing model already. It's a very minimalist one and I like it for that reason. This is it: https://tools.ietf.org/html/rfc7159#section-9
9.  Parsers

10.  Generators

The I-JSON message format standard adds this (https://tools.ietf.org/html/rfc7493#section-4.2):
4.2.  Must-Ignore Policy

I see this as nothing but positive and I want this expressed in the GeoJSON draft. What objections do you have to that language?
I don't see any need to expand GeoJSON's processing model beyond the above, or even to have a section of the spec titled "processing model".

On Wed, Nov 25, 2015 at 12:37 PM, Allan Doyle <[email-masked]<mailto:[email-masked]>> wrote:
I’m starting a new thread to talk about the GeoJSON processing model and my dislike of the language that’s trying to creep in.

GeoJSON is a format.

How do you do that?

        Allan

--
Sean Gillies

---

From: Allan Doyle <[email-masked]>
Date: Wed, 25 Nov 2015 16:34:10 -0500

> On Nov 25, 2015, at 4:18 PM, Sean Gillies <[email-masked]> wrote:
> 
> Allan, 
> 
> We can't assert that GeoJSON exists outside of any processing model because GeoJSON is based on JSON and JSON has a processing model already. It's a very minimalist one and I like it for that reason. This is it: https://tools.ietf.org/html/rfc7159#section-9 <https://tools.ietf.org/html/rfc7159#section-9>
> 9.  Parsers
> 

That’s ok. It says nothing about a relationship between an given input and a given output. 

> 
> The I-JSON message format standard adds this (https://tools.ietf.org/html/rfc7493#section-4.2 <https://tools.ietf.org/html/rfc7493#section-4.2>):
> 
> 4.2.  Must-Ignore Policy
> 

That language appears under a heading called “4. Recommendations for Protocol Design”. GeoJSON already references RFC7493 in section 6.1. The text in 4.2 is wrapped by wording that says “I-JSON is designed for use in Internet protocols.  The following recommendations apply to the use of I-JSON in such protocols.”

That makes 4.2 informative.

There is no such “recommendation” wrapper in the proposed GeoJSON text. Thus RFC7493 was incompletely channeled into the GeoJSON draft. It would be better to apply the Don’t Repeat Yourself principle and refer to RFC7493, not lift some text from it.

> 
> I don't see any need to expand GeoJSON's processing model beyond the above, or even to have a section of the spec titled "processing model”.

I guess I’m against what I’d maybe call “gratuitous handholding”.

> 
> On Wed, Nov 25, 2015 at 12:37 PM, Allan Doyle <[email-masked] <mailto:[email-masked]>> wrote:

---

From: Erik Wilde <[email-masked]>
Date: Wed, 25 Nov 2015 22:48:14 +0100

On 2015-11-25 22:18, Sean Gillies wrote:
> We can't assert that GeoJSON exists outside of any processing model
> because GeoJSON is based on JSON and JSON has a processing model
> already. It's a very minimalist one and I like it for that reason. This
> is it: https://tools.ietf.org/html/rfc7159#section-9

thanks, @sgillies. i am not quite sure that a "format" is something so 
absolutely different from a "protocol". both are intended to establish 
some mutual understanding of a limited domain of the world. protocols 
also have interaction rules, which are not required for formats. but 
apart from that, it's basically the same, since (pretty much) every 
protocol has rules for formats. how parties engaging in interactions 
should behave (accepting/rejecting content) is essential to protocols, 
and that applies to formats as well.

even if the processing models stays very minimalist along the lines of 
what sean pointed to, that's extremely valuable. it tells implementers 
what kind of processing they are supposed to implement, and at which 
stage they can throw up their arms and claim that whatever it is they're 
looking at, it certainly isn't GeoJSON.

same for producers: what rules do they have to follow so that anybody 
they are interacting with will have to accept that what they do is 
within the limits of what they're allowed to do.

i am not quite sure how saying that "people will do whatever they want 
to do anyway" is helping here. sure some will and always will. but 
others are looking for guidance how to do things properly, and any code 
that is written should have a fighting chance to implement the same 
assumptions about what can be expected. if the spec refuses to say what 
is and isn't expected to happen, then achieving interop gets harder.

[[ tldr here is my anecdote about the great geojsonlint which did not 
implement a constraint i ran into while experimenting 
(https://github.com/JasonSanford/geojsonlint.com/issues/19). one cause 
may have been that in the old spec, these notes were scattered across 
the spec and a little bit hard to find. just getting them into one place 
already would be an improvement. if there is a problem with the headings 
"extensibility" and/or "processing model", then there are others to 
choose from. ]]

the spec could have a processing model section that is backed by a test 
suite. implementers then have a good starting point to see what they 
should and shouldn't be able to deal with. such a test suite is a good 
way to explain what a spec is capable of, and where the limits are. of 
course there will always be implementations that ignore this. but being 
helpful to those that want to be interoperable can be a good thing.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked] |
            | http://dret.net/netdret    |
            | http://twitter.com/dret    |

---

From: Sean Gillies <[email-masked]>
Date: Wed, 25 Nov 2015 15:22:21 -0700

Allan,

On Wed, Nov 25, 2015 at 2:34 PM, Allan Doyle <[email-masked]>
wrote:

> because GeoJSON is based on JSON and JSON has a processing model already.
> It's a very minimalist one and I like it for that reason. This is it:

Agreed, it does not. But there is without a doubt a processing model.
Elements of that model include parsers, generators, texts, and other
representations. I'm satisfied with this and don't seek to add things like
infosets or feature types.

> I see this as nothing but positive and I want this expressed in the
> GeoJSON draft. What objections do you have to that language?
>
>
> That language appears under a heading called “4. Recommendations for
> Protocol Design”. GeoJSON already references RFC7493 in section 6.1. The
> text in 4.2 is wrapped by wording that says “I-JSON is designed for use in
> Internet protocols.  The following recommendations apply to the use of
> I-JSON in such protocols.”
>
> There is no such “recommendation” wrapper in the proposed GeoJSON text.
> Thus RFC7493 was incompletely channeled into the GeoJSON draft. It would be
> better to apply the Don’t Repeat Yourself principle and refer to RFC7493,
> not lift some text from it.
>

Given that GeoJSON currently says this:

The GeoJSON object MAY have any number of other members.

– in other words, that you are well within proper GeoJSON when you add
non-standard members – I feel like we could modestly increase confidence on
all sides by insisting that implementers ignore unrecognized members and
never halt and catch fire on encountering them. If we could agree that RFC
7159's

covers this already, then the language from RFC 7493 may be superfluous.

> I don't see any need to expand GeoJSON's processing model beyond the
> above, or even to have a section of the spec titled "processing model”.
>
>

I don't understand why you would call it this.

-- 
Sean Gillies

---

From: Allan Doyle <[email-masked]>
Date: Wed, 25 Nov 2015 20:11:55 -0500

> On Nov 25, 2015, at 5:22 PM, Sean Gillies <[email-masked]> wrote:
> 
> Allan,
> 
> On Wed, Nov 25, 2015 at 2:34 PM, Allan Doyle <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Nov 25, 2015, at 4:18 PM, Sean Gillies <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Allan, 
>> 
> Agreed, it does not. But there is without a doubt a processing model. Elements of that model include parsers, generators, texts, and other representations. I'm satisfied with this and don't seek to add things like infosets or feature types.

As long as we’re away from prescribing what to do with an input other than parse it or an output other than generate a correct one, I’m ok.

> – in other words, that you are well within proper GeoJSON when you add non-standard members – I feel like we could modestly increase confidence on all sides by insisting that implementers ignore unrecognized members and never halt and catch fire on encountering them. If we could agree that RFC 7159's
> 

The GeoJSON draft incorporates 7159 and 7493 by reference. Unless it diverts from those specs, then the language in those specs should be sufficient. The GeoJSON draft should leave the text in those specs and refer to it. Copying it in will lead to confusion. It already has. I would have spent far less of today on this if I had realized the proposed extension wording was an incompletely lifted part of 7493.

> I don't understand why you would call it this.

Some specs can get too far into trying to protect people from themselves. I don’t want ours to be one of them.

---

From: Alexandre Petrescu <[email-masked]>
Date: Thu, 26 Nov 2015 11:07:18 +0100

Hello,

Allow me a comment here.

Le 25/11/2015 22:48, Erik Wilde a écrit :
>> We can't assert that GeoJSON exists outside of any processing
>> model because GeoJSON is based on JSON and JSON has a processing
>> model already. It's a very minimalist one and I like it for that
>> reason. This is it: https://tools.ietf.org/html/rfc7159#section-9
>
> thanks, @sgillies. i am not quite sure that a "format" is something
> so absolutely different from a "protocol". both are intended to
> establish some mutual understanding of a limited domain of the world.
> protocols also have interaction rules, which are not required for
> formats. but apart from that, it's basically the same, since (pretty
> much) every protocol has rules for formats. how parties engaging in
> interactions should behave (accepting/rejecting content) is essential
> to protocols, and that applies to formats as well.

Indeed protocols rely on message formats all over.  As such it's
difficult to distinguish 'protocol', or 'message exchange' if I may,
from 'format'.

However, it's worth noting that whereas a message exchange can not work
w/o a precisely defined message format, message formats independent of
the way they are transported do exist.  Designers of these formats
rightfully prefer to not worry about how they are transported - it's
just payload data.

In GeoJSON case I think the assumed protocol is HTTP.  If so it is worth
stating.  Because it gives a very clear picture.

If on another hand, at a node there is a constant relationship between a
field in a received message and field in a message to be sent then one
is specifying protocol behaviour.  (e.g. "if receiver receives a message
with the field Request==1 then it must set the coordinates in the
generated reply" - is pure protocol specification).

Alex

>
> even if the processing models stays very minimalist along the lines
> of what sean pointed to, that's extremely valuable. it tells
> implementers what kind of processing they are supposed to implement,
> and at which stage they can throw up their arms and claim that
> whatever it is they're looking at, it certainly isn't GeoJSON.
>
> i am not quite sure how saying that "people will do whatever they
> want to do anyway" is helping here. sure some will and always will.
> but others are looking for guidance how to do things properly, and
> any code that is written should have a fighting chance to implement
> the same assumptions about what can be expected. if the spec refuses
> to say what is and isn't expected to happen, then achieving interop
> gets harder.
>
> (https://github.com/JasonSanford/geojsonlint.com/issues/19). one
> cause may have been that in the old spec, these notes were scattered
> across the spec and a little bit hard to find. just getting them into
> one place already would be an improvement. if there is a problem with
> the headings "extensibility" and/or "processing model", then there
> are others to choose from. ]]
>
> the spec could have a processing model section that is backed by a
> test suite. implementers then have a good starting point to see what
> they should and shouldn't be able to deal with. such a test suite is
> a good way to explain what a spec is capable of, and where the limits
> are. of course there will always be implementations that ignore this.
> but being helpful to those that want to be interoperable can be a
> good thing.
>
> cheers,
>
> dret.
>