Subject: [Cbor] MIME tag 257 vs 36

From: Laurence Lundblade <[email-masked]>
Date: Tue, 15 Sep 2020 13:17:03 -0700

Does this seem right?

MIME tag 257 can carry MIME messages that have any content transfer encoding. It is not necessary to use tag 36 for content transfer encoding 7bit, base64 and quoted printable. You cannot however use tag 36 with content transfer encoding binary. You probably can use tag 36 with content transfer encoding 8bit.

It is thus recommend protocols use tag 257 when encoding MIME messages. Use of tag 36 should be avoided. It’s primary use is in protocols that were defined before tag 257 was defined.

I don’t think this is an issue, but I have little bit of worry that tag 36 might give you some end-of-line canonicalization because it is UTF-8 that tag 257 won’t.

LL

---

From: Jim Schaad <[email-masked]>
Date: Tue, 15 Sep 2020 14:21:23 -0700

-----Original Message-----
From: CBOR <[email-masked]> On Behalf Of Laurence Lundblade
Sent: Tuesday, September 15, 2020 1:17 PM
To: [email-masked]
Subject: [Cbor] MIME tag 257 vs 36

Does this seem right?

[JLS] I would disagree.  You cannot use tag 36 for 8bit encoding unless the charset of the message is either US-ASCII or UTF8.  If this is encoding using the code page 932 (Shift JIS) then while it is 8bit it is not UNICODE and thus requires tag 257.

[JLS]  I can't see that EOL canonicalization would be an issue here.
Jim

LL

_______________________________________________
CBOR mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/cbor

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 16 Sep 2020 18:22:45 +0200

> On 2020-09-15, at 22:17, Laurence Lundblade <[email-masked]> wrote:
> 
> Does this seem right?
> 
> MIME tag 257 can carry MIME messages that have any content transfer encoding.

Yes, except we don’t really care about the content transfer encoding (that is internal to the MIME message), we care about whether the result is UTF-8 or not (and tag 257 can carry UTF-8 and non-UTF-8 MIME messages).

> It is not necessary to use tag 36 for content transfer encoding 7bit, base64 and quoted printable. You cannot however use tag 36 with content transfer encoding binary. You probably can use tag 36 with content transfer encoding 8bit.

In all these cases, if the encoded message happens to be UTF-8, you can still use tag 36.  Whether you should, is maybe another question.

> It is thus recommend protocols use tag 257 when encoding MIME messages. Use of tag 36 should be avoided. It’s primary use is in protocols that were defined before tag 257 was defined.

That is a permissible conclusion.

> I don’t think this is an issue, but I have little bit of worry that tag 36 might give you some end-of-line canonicalization because it is UTF-8 that tag 257 won’t.

I don’t think CBOR decoders can do end—of-line processing willy-nilly in their UTF-8 strings, so I don’t see an issue.

Note that I just did PR#210 for the tag 35 issues; I don’t see a need to apply any further changes to the text on tag 36.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 16 Sep 2020 11:52:35 -0700

> On Sep 16, 2020, at 9:22 AM, Carsten Bormann <[email-masked]> wrote:
>> 
> 

MIME has lots of internal mechanisms for variation and extension. Regular expressions seems to me a moving target not really oriented protocol interoperation. Tag 36 seems fine as is.

Thanks for the response.

LL

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 17 Sep 2020 23:20:54 -0700

Shouldn’t both these MIME tags require canonical CRLF line endings? How else are the line endings going to end up correctly?

Makes me wonder about major type 3 too.  My understanding is that Unicode handles line endings like ASCII, it is neutral and happy to have CR, LF or CRLF so type 3 is neutral.

Text files are somewhat less common these days, but git still does conversion. CBOR could have a tag that indicates something is text with canonical CRLF line endings.

LL

> On Sep 16, 2020, at 11:52 AM, Laurence Lundblade <[email-masked]> wrote:
> 
>

---

From: Jim Schaad <[email-masked]>
Date: Sun, 20 Sep 2020 10:54:58 -0700

-----Original Message-----
From: CBOR <[email-masked]> On Behalf Of Laurence Lundblade
Sent: Thursday, September 17, 2020 11:21 PM
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]
Subject: Re: [Cbor] MIME tag 257 vs 36

[JLS] The only place that I know of where canonical line endings is required is for S/MIME.  MIME itself uses LF or CRLF as a line ending just fine.

[JLS] There are a large number of source control systems that do CRLF <=> LF conversions during upload and download.  There are also a large number of other programs (such as browsers) that will do it in some cases as well.  This is due to the artifact that Windows and Linix do not create the same line endings by default and thus doing things like text file comparisons because difficult because every line is different.  I don't see this as a CBOR problem at all.

Jim

LL

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 20 Sep 2020 21:23:46 +0200

On 2020-09-20, at 19:54, Jim Schaad <[email-masked]> wrote:
> 
> I don't see this as a CBOR problem at all.

I agree.  Please see https://tools.ietf.org/html/draft-bormann-dispatch-modern-network-unicode-02 for my last year’s attempt to get some progress on the issue.  I got some great feedback and need to submit the -03 based on this feedback.  When a few CBOR drafts are out of the way…

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 24 Sep 2020 22:24:53 -0700

I did some more reading:
- CRLF is required for MIME headers, q-p, b64 CTEs (RFC 2045)
- CRLF is required for text/xxx types (RFC 2046)
- Unicode Format for Network Interchange requires CRLF (RFC 5198)
- HTTP headers require CRLF
- Git stores LF and translates to local line ending

These text formats choose a canonical line ending for on-the-wire messages so they can know to correctly translate to the local text format. These days it is just CRLF for Windows and LF for Linux/MacOS, it used to be CR for MacOS and in theory any local representation could be used including line lengths or a database of records.

It seems clear that tag 257 and 36 do require CRLF per the MIME standard. That’s all fine and there’s nothing more to do other than reference the MIME standard.

You all may have though about this more than me, but it also seems there is room for the definition of some tags for major type 3 used with line-oriented text since major type 3 is silent on the line ending convention (as it should be). Maybe:

   Tag xx for IETF CRLF line endings (means translate CRLF to local; perhaps error on CR)
   Tag yy fo MNU (means translate LF or CRLF to local; perhaps error on CR)

CBOR protocols that use line-oriented data can specify which on-the-wire line ending they use, so the tags are not absolutely required, but they may be useful. They may also slightly help CBOR protocol designers using line-oriented text realize they need to say what the line ending is.

LL

> On Sep 20, 2020, at 12:23 PM, Carsten Bormann <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 25 Sep 2020 07:59:54 +0200

Hi Laurence,

> - HTTP headers require CRLF

This is one of the big areas where the letter of the standards and the reality can differ a lot.  Newer versions of the standards tend to contain a larger dose of reality; e.g., RFC 7230:

   Although the line terminator for the start-line and header fields is
   the sequence CRLF, a recipient MAY recognize a single LF as a line
   terminator and ignore any preceding CR.

RFC 2616 was somewhat more weasely:

   RFC 2046 requires that content with a type of "text" represent
   line breaks as CRLF and forbids the use of CR or LF outside of line
   break sequences. HTTP allows CRLF, bare CR, and bare LF to indicate a
   line break within text content when a message is transmitted over
   HTTP.

   Where it is possible, a proxy or gateway from HTTP to a strict MIME
   environment SHOULD translate all line breaks within the text media
   types described in section 3.7.1 of this document to the RFC 2049
   canonical form of CRLF. 

Since HTTP is what happens for most interchange, expect the strict CRLF requirement of RFC 2046 to be ignored.

> - Git stores LF and translates to local line ending

Git is rooted in reality.
(It gained CRLF support only after some gyrations.)

> These text formats choose a canonical line ending for on-the-wire messages so they can know to correctly translate to the local text format. These days it is just CRLF for Windows and LF for Linux/MacOS,

Right.

> it used to be CR for MacOS and in theory any local representation could be used including line lengths or a database of records.

I wish people would stop talking about CR line ends; these used to exist, but they no longer do in reality (outside very weird closed environments such as certain PDF files).  RFC 7230 reflects what the reality is, today, which is no longer very tolerant of pure-CR line endings (it’s been 20 years now since those became obsolete).

> It seems clear that tag 257 and 36 do require CRLF per the MIME standard. That’s all fine and there’s nothing more to do other than reference the MIME standard.

That’s what they say.
Any implementation that doesn’t accept LF line endings is pretty much broken for the real world.

>   Tag yy fo MNU (means translate LF or CRLF to local; perhaps error on CR)

It may not be necessary to tag something for what it already is.
But it may be useful to be able to say in CDDL what a text string should look like  ; that may be something that could be added to the MNU specification or to a companion CDDL extension.

> CBOR protocols that use line-oriented data can specify which on-the-wire line ending they use, so the tags are not absolutely required, but they may be useful. They may also slightly help CBOR prot…

In reality, the line ending on the Internet is [CR] LF.  There is no other place CR can be used outside playing teleprinter (RFC 5198), so most software can simply treat CR as pure noise, which it typically does.

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 25 Sep 2020 10:11:00 -0700

Hi Carsten,

Thanks for the detailed explanation.

Is this a good summary of the upshot?

Text lines in Internet protocols (on the wire) are delimited by either a CRLF or just an LF. Officially many protocols specify CRLF, but implementations often work with either. CBOR type 3 text can be either line ending, even a mixture of both.

Operating systems usually have a line end convention. Windows uses CRLF. Linux and MacOS use LF. Some applications on a given OS may work with either and some may prefer the OS’s line ending convention.

The majority of use cases and CBOR protocols using type 3 text will work with either line ending. However, some use cases or protocols may not work with either in which case translation to and/or from the local line end convention, typically that of the OS, is necessary.

Are you proposing that CDDL be able to specify a line end convention for type 3 items?

LL

> On Sep 24, 2020, at 10:59 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Laurence,
>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 26 Sep 2020 08:57:00 +0200

Hi Laurence,

> On 2020-09-25, at 19:11, Laurence Lundblade <[email-masked]> wrote:
> 
> Hi Carsten,
> 
> Text lines in Internet protocols (on the wire) are delimited by either a CRLF or just an LF. Officially many protocols specify CRLF, but implementations often work with either.

Indeed.

> CBOR type 3 text can be either line ending, even a mixture of both.

CBOR text strings can contain any UTF-8, so they are agnostic to line endings.
(“CBOR type 3 text” is a pleonasm, but also leaving out the word “major”.)

Right.

> The majority of use cases and CBOR protocols using type 3 text will work with either line ending. However, some use cases or protocols may not work with either in which case translation to and/or f…

I think the important observation is that most of the limitations to one type of line ending stem from using line endings as part of the encoding/framing of a TCP-based protocol.  E.g., SMTP is less tolerant of modern line endings because it needs to react to CRLF dot CRLF (unless some SMTP framing is used).  That is not a problem with CBOR-based protocols, as the framing is coming from the self-delimiting CBOR encoding, not from parsing text streams.

> Are you proposing that CDDL be able to specify a line end convention for type 3 items?

I think that might be useful.  Most text strings in CBOR protocols are “1-D”, i.e., they don’t employ line endings.  Where “2-D” text is intended, indicating a preference for a form of line ending, or a restriction to a form of line ending, might be useful.  Of course, restricting text content can today be done with regexes (and soon with ABNF); there is no .feature in regex/ABNF though…

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 26 Sep 2020 11:11:02 -0700

> On Sep 25, 2020, at 11:57 PM, Carsten Bormann <[email-masked]> wrote:
> 

I was thinking more of CBOR-based protocols that need to carry 2-D text than protocol framing like SMTP needs. Needs to work with text editors like git. A query protocol against a database that stores text that cares about the line ending. Some text-based error log recorder. The protocol designer should specify what to do. The CDDL would help.

LL