Subject: [fdt] Fwd: [Webtransport] Standards for protocol headers?

From: Scott Morgan <[email-masked]>
Date: Fri, 12 Jun 2020 19:53:04 -0500

Hi Marc,

   Thanks, I'm not trying to be intentionally confusing :)  In an attempt
to clarify, I'm trying to create a new structured PDU format that could be
used to define PDU description languages (not a primary concern) as well as
larger data structures like files or bytes in datagrams.
https://en.wikipedia.org/wiki/Protocol_data_unit

Lets continue @ [email-masked] !

Cheers,
Scott

---------- Forwarded message ---------
From: Marc Petit-Huguenin <[email-masked]>
Date: Fri, Jun 12, 2020 at 10:03 AM
Subject: Re: [Webtransport] Standards for protocol headers?
To: Scott Morgan <[email-masked]>, <[email-masked]>

Hi Scott,

I still find your terminology confusing.  I am not sure if you are
proposing to design a new PDU description language (ala ASN.1, ABNF, RBNF,
CDDL, XML RELAX, etc..) or a new structured PDU format (ala DER, BER, XML,
JSON, CBOR, etc...).  If it is the former, I would suggest to resend your
email to [email-masked], where we discuss this kind of things.

Thanks.

On 6/12/20 7:25 AM, Scott Morgan wrote:
> Hi Marc,
>
>    Thanks for responding!  I meant 'protocol header' in the broadest
> sense.  I have implemented some protocols in the past (Web Socket v13,
SMTP
> etc) and work a lot with JSON and XML as I mentioned previously.  Here are
> some general frustrations (i.e. problem statement) that I'm looking to
> overcome;
>
> 1) JSON and XML generally lack support for binary data (there are some
> workarounds but nothing is super great).
> 2) JSON and XML both now have the concept of a Schema
>          Note I think the concept of a Schema is a good thing but also
feel
> that Schemas often don't cover all
>      Metadata that needs to be covered.  Perhaps the concept of a Schema
> needs to be enhanced by UsageMetadata information.
>          Also note I am frustrated with JSON's inability to provide human
> readable schemas (multi line text in the .json file (not /n)).
>          I also have frustrations with XML schemas and XML (like everyone
> else who now favors JSON over XML)
> 3) Protocol's often have Headers and other information which act in a
> similar fashion to Schemas (i.e. they describe / specify data structures)
>
>   I'm wondering if it's time for a new markup language that can support
> binary data and assist in specifying protocol byte segments in addition to
> more general purpose transport and storage data (like JSON & XML do).
>   I'm thinking about calling it 'Classification Markup Notation'.
>
>   I messaged this to the group just to throw the idea out there, to get
> feedback and also to find collaborators.
>
>
> On Thu, Jun 11, 2020 at 7:53 AM Marc Petit-Huguenin <
[email-masked]>
> wrote:
>
>> Hi,
>>
>> On 6/10/20 5:14 PM, Scott Morgan wrote:
>>> Hi All,
>>>
>>>    I have been working with JSON, XML and some protocol headers recently
>>> and I am wondering...
>>> Why are there no real standard text formats for protocol headers, or am
I
>>> missing something?
>>>
>>> It seems to me that a slightly more streamlined/optimal binary/markup
>>> standard could help protocols attain POCs and adoption much quicker and
>>> help reduce some of the work defining protocols.
>>>
>>> Any thoughts on this topic?
>>>
>>>
>>
>> Can you please elaborate on what you mean by "protocol header"?  My
>> understanding of that term is the fixed part at the beginning of a PDU,
PDU
>> that also contains a variable length part after it (as in UDP Header, TCP
>> Header, RTP Header, HTTP, SIP, SMTP etc...), but that definition does not
>> seem to apply to JSON or XML (as a whole).
>>

-- 
Marc Petit-Huguenin
Email: [email-masked]
Blog: https://marc.petit-huguenin.org
Profile: https://www.linkedin.com/in/petithug

-- 
Regards,
Scott Morgan
President & CEO
Adligo Inc
http://www.adligo.com
https://www.linkedin.com/in/scott-morgan-21739415
A+ Better Business Bureau Rating
<https://www.bbb.org/chicago/business-reviews/computer-software-publishers-and-developers/adligo-inc-in-chicago-il-88381256>
https://github.com/adligo

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]
skype:adligo1?call
Send Me Files Securely:
*https://www.sendthisfile.com/f.jsp?id=ewOnyeFQM18IDRf7MMIdolfI
<https://www.sendthisfile.com/f.jsp?id=ewOnyeFQM18IDRf7MMIdolfI>*

---

From: Stephen McQuistin <[email-masked]>
Date: Sat, 13 Jun 2020 12:07:55 +0000

Hi Scott,

We have a draft (draft-mcquistin-augmented-ascii-diagrams) that is looking to add tooling (e.g., to generate parser code) around “traditional” packet header diagrams, by using a consistent format for both the diagram and the text that accompanies it.

In terms of other PDU description languages, the recent QUIC drafts have replaced packet diagrams with a bespoke description language: https://tools.ietf.org/html/draft-ietf-quic-transport-29#section-1.3.

These are both much simpler than the languages that Marc mentioned, and only deal with PDU description rather than data layout description more generally, but might be worth a look.

Thanks,

Stephen

On 13 Jun 2020, at 01:53, Scott Morgan <[email-masked]<mailto:[email-masked]>> wrote:

Hi Marc,

   Thanks, I'm not trying to be intentionally confusing :)  In an attempt to clarify, I'm trying to create a new structured PDU format that could be used to define PDU description languages (not a primary concern) as well as larger data structures like files or bytes in datagrams.
https://en.wikipedia.org/wiki/Protocol_data_unit

Lets continue @ [email-masked]<mailto:[email-masked]> !

Cheers,
Scott

---------- Forwarded message ---------
From: Marc Petit-Huguenin <[email-masked]<mailto:[email-masked]>>
Date: Fri, Jun 12, 2020 at 10:03 AM
Subject: Re: [Webtransport] Standards for protocol headers?
To: Scott Morgan <[email-masked]<mailto:[email-masked]>>, <[email-masked]<mailto:[email-masked]>>

Hi Scott,

I still find your terminology confusing.  I am not sure if you are proposing to design a new PDU description language (ala ASN.1, ABNF, RBNF, CDDL, XML RELAX, etc..) or a new structured PDU format (ala DER, BER, XML, JSON, CBOR, etc...).  If it is the former, I would suggest to resend your email to [email-masked]<mailto:[email-masked]>, where we discuss this kind of things.

Thanks.

On 6/12/20 7:25 AM, Scott Morgan wrote:
> sense.  I have implemented some protocols in the past (Web Socket v13, SMTP
>          Note I think the concept of a Schema is a good thing but also feel
> On Thu, Jun 11, 2020 at 7:53 AM Marc Petit-Huguenin <[email-masked]<mailto:[email-masked]>>
> wrote:
>
>> Hi,
>>
>>> Why are there no real standard text formats for protocol headers, or am I
>>> missing something?
>>>
>> understanding of that term is the fixed part at the beginning of a PDU, PDU

--
Marc Petit-Huguenin
Email: [email-masked]<mailto:[email-masked]>
Blog: https://marc.petit-huguenin.org<https://marc.petit-huguenin.org/>
Profile: https://www.linkedin.com/in/petithug

--
Regards,
Scott Morgan
President & CEO
Adligo Inc
http://www.adligo.com<http://www.adligo.com/>
https://www.linkedin.com/in/scott-morgan-21739415
A+ Better Business Bureau Rating<https://www.bbb.org/chicago/business-reviews/computer-software-publishers-and-developers/adligo-inc-in-chicago-il-88381256>
https://github.com/adligo

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]<mailto:[email-masked]>
skype:adligo1?call
Send Me Files Securely:
https://www.sendthisfile.com/f.jsp?id=ewOnyeFQM18IDRf7MMIdolfI

<signature.asc>--
FDT mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/fdt

---

From: Patrick McManus <[email-masked]>
Date: Mon, 15 Jun 2020 14:53:45 -0400

you might be interested in structured field values for http
https://datatracker.ietf.org/doc/draft-ietf-httpbis-header-structure/

-P

On Sat, Jun 13, 2020 at 8:08 AM Stephen McQuistin <
[email-masked]> wrote:

> Hi Scott,
>
> We have a draft (draft-mcquistin-augmented-ascii-diagrams) that is looking
> to add tooling (e.g., to generate parser code) around “traditional” packet
> header diagrams, by using a consistent format for both the diagram and the
> text that accompanies it.
>
> In terms of other PDU description languages, the recent QUIC drafts have
> replaced packet diagrams with a bespoke description language:
> https://tools.ietf.org/html/draft-ietf-quic-transport-29#section-1.3.
>
> These are both much simpler than the languages that Marc mentioned, and
> only deal with PDU description rather than data layout description more
> generally, but might be worth a look.
>
> Thanks,
>
> Stephen
>
> On 13 Jun 2020, at 01:53, Scott Morgan <[email-masked]> wrote:
>
> Hi Marc,
>
> [email-masked]
> Webtransport mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/webtransport
>

---

From: Scott Morgan <[email-masked]>
Date: Mon, 15 Jun 2020 20:09:34 -0500

Hi All,

  Thanks for your suggestions!  I am really trying to scope my effort as;
***
How to interpret bytes!
***
ASCII & in particular # 10 seems significant! A single byte that splits
lines (and could spit other data up);
https://www.petefreitag.com/item/863.cfm#:~:text=The%20ASCII%20character%20code%2010,a%20line%20in%20a%20file.

I'm looking to replace JSON & XML as the dominant transfer format for REST
(and other styles/protocols) that would also be a good standard for
protocol level information (bits and bytes).

The ASCII newline byte and A8 (intersection of ASCII & UTF-8) seems to be
logical spit (binary vs character markup data).

Keep the RFCs coming my way!

On Mon, Jun 15, 2020 at 1:53 PM Patrick McManus <[email-masked]>
wrote:

>> We have a draft (draft-mcquistin-augmented-ascii-diagrams) that is
>> looking to add tooling (e.g., to generate parser code) around “traditional”
>> packet header diagrams, by using a consistent format for both the diagram
>> and the text that accompanies it.
>>

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]
skype:adligo1?call
Send Me Files Securely:

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 16 Jun 2020 04:46:19 -0700

> protocol level information (bits and bytes).

The next logical step in the evolution (or is it devolution?) of XML then JSON is SExp.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 16 Jun 2020 17:03:27 +0200

Hi Marc,

This is actually a quite profound observation.

We used S-Expressions in the work that led up to RFC 3259, and everything went downhill since then.

The important part of an S-Expression design is, of course, what kind of data goes in there; the nice thing about S-Expressions is that, to a certain extent, you get to design that in a bespoke way for each case, but that also is its downfall as it cannot be standardized.

JSON (or, preferably, CBOR) actually makes nice S-Expressions if you just don’t use those maps (“JSON Objects”); see https://www.ietf.org/id/draft-bormann-cbor-cddl-freezer-04.html#name-alternative-representations for a current example (yes, that is the entirety of CDDL in that figure).  Not being able to separate strings and symbols does hurt, but can be worked around.

Grüße, Carsten

---

From: Scott Morgan <[email-masked]>
Date: Tue, 16 Jun 2020 21:34:50 -0500

Please provide the latest RFC for SExp :)

On Tue, Jun 16, 2020 at 10:03 AM Carsten Bormann <[email-masked]> wrote:

> The important part of an S-Expression design is, of course, what kind of
> data goes in there; the nice thing about S-Expressions is that, to a
> certain extent, you get to design that in a bespoke way for each case, but
> that also is its downfall as it cannot be standardized.
>
> JSON (or, preferably, CBOR) actually makes nice S-Expressions if you just
> don’t use those maps (“JSON Objects”); see
> https://www.ietf.org/id/draft-bormann-cbor-cddl-freezer-04.html#name-alternative-representations
> for a current example (yes, that is the entirety of CDDL in that figure).
> Not being able to separate strings and symbols does hurt, but can be worked
> around.
>
> Grüße, Carsten
>
>

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]
skype:adligo1?call
Send Me Files Securely:

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Wed, 17 Jun 2020 00:29:55 -0700

> Please provide the latest RFC for SExp :)

S-Expressions predates the RFC series, see section 3 of:

McCarthy J. Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I. :34. 

There has been a few efforts since to standardize it:

https://tools.ietf.org/html/draft-hedberg-spocp-sexp-00
https://people.csail.mit.edu/rivest/Sexp.txt

I especially like the canonical S-Expressions in the Rivest proposal.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 17 Jun 2020 14:46:37 +0200

> https://people.csail.mit.edu/rivest/Sexp.txt

Hmm.  This spends a lot of time discussing six different representations for the same thing, apparently a byte string, plus two in-memory forms that are essentially a seventh and an eight form.  Not something I’d like to use in a protocol.

The byte strings can be annotated by a “display hint”, which seems to be a kind of type system (and, yes, the thing is old enough to default to Latin 1 :-).

You seem to be interested in the canonical form, which is closer to what one would use today.

SExp = [* SExp] / Atom
Atom = bytes / TBD([Type, bytes])
Type = bytes

Where TBD would be a tag that is not defined today.

I believe we are now (2020) in a position that we have genuinely obsoleted this kind of presentation-oriented representation as an interchange format.

But the idea to describe data items using the above as a generic data model still makes sense.
I don’t quite understand why only atoms can be type-tagged, so I would simplify to

SExp = [* SExp] / TBD(Type, SExp) / bytes
Type = bytes

If you don’t have tags, maybe use

SExp = [* SExp] / {Type => SExp} / bytes
Type = bytes

Special accommodation would probably be needed in JSON, which doesn’t have “bytes”; just mix in some base64url.

Grüße, Carsten

---

From: Scott Morgan <[email-masked]>
Date: Tue, 23 Jun 2020 21:27:29 -0500

Hi All,

  Thanks for the thoughts!  I'm looking to create more of a pluggable
delegating approach to interpreting bytes, this would allow craziness like
embedding AIFF, JPEG, PNG, JSON, CSV and XML and other 'ClasifiedBytes'
directly inside the 'Bytes'.
  It would start with a root Bytes class that had an optional max (bytes)
class attribute (protocol DoS defense) and optional size classified
attribute.  The RootParser would spin through looking only for the unix
newline byte (i.e. /n).   Each of these LineOfBytes would be processed,
then similar to the lexicon of many programming languages fields (B8 bytes
(ASCII intersection with UTF-8)) would be assigned to bytes delimited by
('=','<,>') i.e.;

text=SomeText<,>jpeg<size="3">=123<,>

Also basic XML character escapes would be used in the Bytes classified as
Text (aka UTF-8), but custom encoding could easily be added.

text=Some&lt;Text&gt;<,>jpeg<size="3">=123<,>

The sub Classification (aka Schema) of the Bytes might look something like
this;

<space>
  name=com.example<,>
  <class>
    name=NameId<,>
    fields=<{>
      name=TextLine<,>
      id=Integer
    <}><,>
  </class>
  <class>
    name=Point<,>
    fields=<{>
      x=SignedInteger<,>
      y=SignedInteger
    <}><,>
  </class>
  <class>
    name=PointList<,>
    fields=<{>
      key=NameId<,>
      values=List<Point>
    <}><,>
  </class>
</space>

Then the following bytes could be parsed as a com.example.PointList (not an
efficient example!)

key=<{>
  name=twoPoints<,>
  id=123<,>
<}><,>
values=<{>
  <{>x=1<,>y=2<}><,>
  <{>x=-1<,>y=-5<}>
 <}>

Although this format seems quite heavy (due to wrapping JSON like B8
special character bytes in XML like B8 byte special character bytes), each
classification would allow for better processing based on the need, for
example if the above example used *JSON* for the *values* it might look
like;

key=<{>
  name=twoPoints<,>
  id=123
<}><,>
values=
*  {*
*    "xs":[1,-1],*
*    "ys":[2,-5]*
*   }*

  I used this weird union of XML and JSON above, but it's a very rough
idea.  The space would be quite similar to the Java namespace and each line
of bytes would be limited to 2,147,483,647 bytes by default (a Java byte[])
limitation.

Thoughts?

On Wed, Jun 17, 2020 at 7:46 AM Carsten Bormann <[email-masked]> wrote:

> Hmm.  This spends a lot of time discussing six different representations
> for the same thing, apparently a byte string, plus two in-memory forms that
> are essentially a seventh and an eight form.  Not something I’d like to use
> in a protocol.
>
> The byte strings can be annotated by a “display hint”, which seems to be a
> kind of type system (and, yes, the thing is old enough to default to Latin
> 1 :-).
>
> I believe we are now (2020) in a position that we have genuinely obsoleted
> this kind of presentation-oriented representation as an interchange format.
>

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]
skype:adligo1?call
Send Me Files Securely:

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 24 Jun 2020 06:56:26 +0200

> Thoughts?

Well, you seem to want to invent another lexical syntax.

2020, we don’t do that any more.

For things that are text and need to be exchanged as text, JSON is the lexical syntax now.
Where you need more flexibility (e.g., include JPEG) and don’t have a need for the encoding to be text, use CBOR.

This turns the question “what is the lexical syntax” (something which you would express in ABNF) into the question “what is the structure of the data” (something that you express in CDDL).  For a theoretician, of course, both are syntaxes, but there is a lot of benefit in handling the lexical syntax with a known good software component (many bugs that turn into attack vectors are in parsers for lexical syntax).

[That doesn’t mean you cannot have fatal bugs in the component that handles the structure of the data — we have just reduced the attack surface a lot, and we can use an optimized component for the lexical work.]

This doesn’t mean there isn’t a place for highly refined encodings such as HDF5, or legacy structural encodings such as RIFF (underlying WAV etc.) or JPEG’s byte stuffing based delimiters.  There are even new legacy encodings of this kind being defined :-) (e.g., Extensible Binary Meta Language, draft-ietf-cellar-ebml).  But for 98 % of all new work, JSON and CBOR are it.

Grüße, Carsten

---

From: Scott Morgan <[email-masked]>
Date: Wed, 24 Jun 2020 08:54:16 -0500

Hi Again,

  Thanks for responding, I have glanced at these and here are my notes;

https://tools.ietf.org/html/rfc7049
I want to avoid being bound by something like this (i.e. JSON);

The underlying data model is an
   extended version of the JSON data model [RFC4627
<https://tools.ietf.org/html/rfc4627>].

https://tools.ietf.org/html/rfc5234
Yep putting my idea in this form would probably help and be a lot of work.
So far the Bytes Class would simply look like this;
*(OCTET) / *(LF)
Also the LineOfBytes Class would look like this;
*(OCTET) LF
:)

https://www.ietf.org/id/draft-ietf-cellar-ebml-17.txt
This seems similar to what I'm trying to do but different enough to be a
separate idea, I like the part about VINTs.  However since I'm trying to
parse OCTETs distinguishing the 7 bit ascii from other OCTETs I would need
(7 bit VINTs) to allow for that compression.

Please let me know if anyone is interested in collaborating on this idea!

On Tue, Jun 23, 2020 at 11:56 PM Carsten Bormann <[email-masked]> wrote:

> This turns the question “what is the lexical syntax” (something which you
> would express in ABNF) into the question “what is the structure of the
> data” (something that you express in CDDL).  For a theoretician, of course,
> both are syntaxes, but there is a lot of benefit in handling the lexical
> syntax with a known good software component (many bugs that turn into
> attack vectors are in parsers for lexical syntax).
>
> [That doesn’t mean you cannot have fatal bugs in the component that
> handles the structure of the data — we have just reduced the attack surface
> a lot, and we can use an optimized component for the lexical work.]
>
> This doesn’t mean there isn’t a place for highly refined encodings such as
> HDF5, or legacy structural encodings such as RIFF (underlying WAV etc.) or
> JPEG’s byte stuffing based delimiters.  There are even new legacy encodings
> of this kind being defined :-) (e.g., Extensible Binary Meta Language,
> draft-ietf-cellar-ebml).  But for 98 % of all new work, JSON and CBOR are
> it.
>
> Grüße, Carsten
>
>

By Appointment Only:
1-866-968-1893 Ex 101
[email-masked]
skype:adligo1?call
Send Me Files Securely: