Subject: [apps-discuss] I-D Action: draft-ietf-appsawg-text-markdown-01.txt

From: [email-masked]
Date: Tue, 09 Sep 2014 09:17:48 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Applications Area Working Group Working Group of the IETF.

        Title           : The text/markdown Media Type
        Author          : Sean Leonard
	Filename        : draft-ietf-appsawg-text-markdown-01.txt
	Pages           : 14
	Date            : 2014-09-09

Abstract:
   This document registers the text/markdown media type for use with
   Markdown, a family of plain text formatting syntaxes that optionally
   can be converted to formal markup languages such as HTML.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-appsawg-text-markdown/

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-appsawg-text-markdown-01

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-appsawg-text-markdown-01

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 9 Sep 2014 09:33:54 -0700

For the record, I think the changes Sean made in the -01 make the situation significantly worse. Please: let's let the Markdown "community" settle down before we go barging in with a media type, particularly one that puts *other people's names* in the registry.

--Paul Hoffman

---

From: Dave Crocker <[email-masked]>
Date: Tue, 09 Sep 2014 10:15:48 -0700

> For the record, I think the changes Sean made in the -01 make the situation significantly worse. Please: let's let the Markdown "community" settle down before we go barging in with a media type, pa…

What is the formal basis in IETF process for slow-rolling a media type
request?

I'm not challenging your concern, just the question of following our own
rules.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 9 Sep 2014 10:26:01 -0700

> What is the formal basis in IETF process for slow-rolling a media type

Why does there need to be a "formal basis"? It's about being polite to the people outside the IETF, not about formal processes. We remember how to do the former, I hope.

--Paul Hoffman

---

From: Dave Crocker <[email-masked]>
Date: Tue, 09 Sep 2014 10:28:27 -0700

On 9/9/2014 10:26 AM, Paul Hoffman wrote:
> Why does there need to be a "formal basis"? It's about being polite
> to the people outside the IETF, not about formal processes. We remember
how> > to do the former, I hope.

Being polite in a way that results in differential handling of
formally-legitimate requests can also be interpreted as unfair
treatment. It's not enough that your or we "mean well".

d/

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 13:32:39 -0500

> I'm not challenging your concern, just the question of following our own

BTW, the I-D uses a MIME type parameter to distinguish between sub-types
of Markdown, including the original.

A bigger concern for me than the name (is there a trademark to worry
about?  I think there isn't) is the fact that the I-D doesn't describe
the format except via informative references to unstable documents.  Of
course, for specific extensions that might be unavoidable, but at least
for the original it ought to be possible to include a normative[*]
description.

[*] Yes, I know, the I-D targets publication on the Informational track.

Nico
--

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 9 Sep 2014 20:50:57 +0200

On Tuesday, September 9, 2014 8:32:39 PM CEST, Nico Williams wrote:
> ... the I-D doesn't describe

The original is a brief description and a perl implementation, and the two 
mostly agree but have many minor differences. So which of the two would you 
reference?

Arnt

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Sep 2014 13:56:56 -0500

On Tue, Sep 09, 2014 at 08:50:57PM +0200, Arnt Gulbrandsen wrote:
> The original is a brief description and a perl implementation, and
> the two mostly agree but have many minor differences. So which of
> the two would you reference?

I don't know yet.  But we could certainly study the matter and pick or
(or a subset).

Some things that the author (and reviewers) could do to resolve any
disagreements between the two original "specifications":

 - ask the original author which of those two should be normative

 - identify a common subset, and if it's useful enough consider using
   that as normative

 - review running code to see which of the two is closest to what's
   deployed and pick that

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 9 Sep 2014 20:33:35 -0000

>>> For the record, I think the changes Sean made in the -01 make the situation significantly worse. Please: let's let
>the Markdown "community" settle down before we go barging in with a media type, particularly one that puts *other
>people's names* in the registry.

Agreed.  It seems to me that the point of a text/markdown media type
is to identify whatever format it is that markdown users expect to
find.

If, as in this case, it's not obvious which of multiple versions of
that format is most widely used, or where it's defined, it's more than
reasonable to delay until the smoke clears.  As Paul noted, it's been
around for a decade, and there's nothing urgent about defining the
media type right this minute.

R's,
John

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 9 Sep 2014 13:57:35 -0700

>  - review running code to see which of the two is closest to what's

As I read it, the mini-charter submitted for this document specifically
declares as out-of-scope any discussion of what markdown format elements we
might want to choose as standard or normative, or making any definitions of
such on our own.  Thus, all of those suggestions can't be followed without
renegotiating with the WG.

What is agreed is that we would create an IANA registry and registration
process for assigning token words to specific versions of markdown and
their defining documents.  This "standard" version might be just another
entry in such a registry.

That said, I concur that there's no hurry to expedite this work, and also
that we should not do this in a vacuum.  As much as possible, the markdown
community should be made aware of the work proposed here, and participate
in the selection and definition of at least the initial set of registry
entries and their corresponding reference materials.  I'm willing to wait a
reasonable period for that to happen.

-MSK, with half of the APPSAWG chair hat on

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 10 Sep 2014 08:49:35 -0700

> reasonable period for that to happen.
If I were to prepare something that looks like a liaison statement from
APPSAWG to the markdown community asking them for feedback on what we're
proposing, is there a mailing list or a specific set of people to which I
should send it?

-MSK

---

From: Dave Crocker <[email-masked]>
Date: Wed, 10 Sep 2014 08:59:59 -0700

On 9/9/2014 1:57 PM, Murray S. Kucherawy wrote:
> declares as out-of-scope any discussion of what markdown format elements
> we might want to choose as standard or normative, or making any
> definitions of such on our own.  Thus, all of those suggestions can't be
> followed without renegotiating with the WG.
> 

The emergence of some sort of competitive situation for markdown for me
encourages two points:

1. For any label registered under markdown, require a real and useful
specification, not a handwave.

2. Do /not/ register 'standard', since it implies a preferred choice;
especially in the midst of rancorous competition, that merely offends
whoever does not get the label.

d/

---

From: Steve Atkins <[email-masked]>
Date: Wed, 10 Sep 2014 09:50:05 -0700

On Sep 10, 2014, at 8:49 AM, Murray S. Kucherawy <[email-masked]> wrote:

> On Tue, Sep 9, 2014 at 1:57 PM, Murray S. Kucherawy <[email-masked]> wrote:
> 
> That said, I concur that there's no hurry to expedite this work, and also that we should not do this in a vacuum.  As much as possible, the markdown community should be made aware of the work proposed here, and participate in the selection and definition of at least the initial set of registry entries and their corresponding reference materials.  I'm willing to wait a reasonable period for that to happen.
> 
> 
> If I were to prepare something that looks like a liaison statement from APPSAWG to the markdown community asking them for feedback on what we're proposing, is there a mailing list or a specific set of people to which I should send it?

The six people listed at http://commonmark.org + John Grubor are probably a decent place to start. The recent rancor between them seems to have calmed down a bit, but it's probably worth avoiding the word "standard" anyway.

Cheers,
  Steve

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 10 Sep 2014 16:57:41 +0000

One more thing... I would say that the lack of a specified flavour does not mean standard or original, but rather unknown. For many senders it is difficult to find out what kind of markdown a particular document is.

Army

---

From: Sean Leonard <[email-masked]>
Date: Wed, 10 Sep 2014 13:38:38 -0700

On 9/10/2014 9:57 AM, Arnt Gulbrandsen wrote:
> One more thing... I would say that the lack of a specified flavour 
> does not mean standard or original, but rather unknown. For many 
> senders it is difficult to find out what kind of markdown a particular 
> document is.
>

Right; this is reflected in the paragraph on page 5 of draft-01:

        When this parameter conveyed (even if empty), the implicit first
        rule is "Original", namely, the original Markdown rules provided
        in John Gruber's Markdown Syntax from 2004 [MDSYNTAX  <https://tools.ietf.org/html/draft-ietf-appsawg-text-markdown-01#ref-MDSYNTAX>]. When this
        parameter is not conveyed, the author does not express any intent
        about which rules apply: [MDSYNTAX  <https://tools.ietf.org/html/draft-ietf-appsawg-text-markdown-01#ref-MDSYNTAX>] may not necessarily be the
        author's intent.

-Sean

---

From: Sean Leonard <[email-masked]>
Date: Fri, 12 Sep 2014 14:37:25 -0700

Hi all,

First, thanks for your comments, both publicly and privately.

The Markdown community is fairly distributed. A significant part of the 
community lurks on the [email-masked] list 
<http://six.pairlist.net/mailman/listinfo/markdown-discuss>. There are 
other groups on the Internet as well--I have reached out to a few. I 
also asked some implementers to participate directly. I even asked John 
Gruber, back when the work started. He did not have objections.

Most of the ideas that informed this draft-01, were already discussed in 
July 2014-September 2014 on that list:
http://six.pairlist.net/pipermail/markdown-discuss/2014-July/thread.html

In addition to those conversations, several of the volunteer reviewers 
have provided very detailed feedback. I specifically asked some 
implementers to be reviewers, and they are graciously volunteering their 
time. You can see the list on the APPSAWG wiki.

One point of feedback in particular is that the introduction lacks a 
disucssion of the goals and use cases for a) Markdown, and b) the 
text/markdown media type. Some respondents have said "nobody is going to 
use such-and-such thing" when in fact the matter was discussed, and 
there are people who have come forward who really want to see 
such-and-such thing for well-defined purposes. If these purposes are 
spelled out clearly and concisely in the draft, it would help explain 
why such-and-such thing exists.

To this end I created a GoalsAndUses wiki:
http://trac.tools.ietf.org/wg/appsawg/trac/wiki/TextMarkdown/GoalsAndUses

To everyone who is using Markdown...please put your use cases on that 
webpage. It centralizes it and allows everyone to see them, so I can 
(potentially) incorporate them into future drafts.

Thanks!

Sean

---

From: Sean Leonard <[email-masked]>
Date: Mon, 15 Sep 2014 22:48:04 -0700

Hi all,

Looks like I have the required 5 reviewers for the Markdown draft. 
However the reviewer set is starting to skew more heavily to non-IETFers 
than IETFers, as I assembled a set of reviewers from around the Markdown 
community (including developers of Markdown tools, and users in 
industry-specific domains). It would be nice if the draft could have one 
additional reviewer who has more experience in the IETF, particularly 
relating to media types and protocol issues associated with them. Any 
takers?

Thanks,

Sean

---

From: Claudio Allocchio <[email-masked]>
Date: Tue, 16 Sep 2014 08:04:17 +0200 (CEST)

On Mon, 15 Sep 2014, Sean Leonard wrote:

> Hi all,
>
> Looks like I have the required 5 reviewers for the Markdown draft. However 
> the reviewer set is starting to skew more heavily to non-IETFers than 
> IETFers, as I assembled a set of reviewers from around the Markdown community 
> (including developers of Markdown tools, and users in industry-specific 
> domains). It would be nice if the draft could have one additional reviewer 
> who has more experience in the IETF, particularly relating to media types and 
> protocol issues associated with them. Any takers?

Hello Sean,

we can ask the APSDIR for a review...

to hich draft are you explicitly referring to?

   draft-ietf-appsawg-text-markdown-01.txt
   draft-seantek-text-markdown-00.txt

both?

>
> Thanks,
>
> Sean
>
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: Sean Leonard <[email-masked]>
Date: Mon, 15 Sep 2014 23:15:21 -0700

On 9/15/2014 11:04 PM, Claudio Allocchio wrote:
>> However the reviewer set is starting to skew more heavily to 
>> non-IETFers than IETFers, as I assembled a set of reviewers from 
>> around the Markdown community (including developers of Markdown 
>> tools, and users in industry-specific domains). It would be nice if 
>> the draft could have one additional reviewer who has more experience 
>> in the IETF, particularly relating to media types and protocol issues 
>> associated with them. Any takers?
>
> Hello Sean,
>

-Sean

---

From: Graham Klyne <[email-masked]>
Date: Tue, 16 Sep 2014 11:31:34 +0100

> draft-ietf-appsawg-text-markdown-01.txt

Reviewing https://tools.ietf.org/html/draft-ietf-appsawg-text-markdown-01

# 1. Introduction

First para (Nit):

This seems a bit bloated.  I don't think anything relevant is lost by deleting 
from "Compare with [RFC6838] Section 4.2.1." to the end of the paragraph.

Para starting "Markdown specifically is a family of syntaxes..." (nit):

I would be inclined to remove the text

   "Fed
    up with the complexity and security pitfalls of formal markup
    languages (e.g., HTML5) and proprietary binary formats (e.g.,
    commercial word processing software), yet unwilling to be confined to
    the restrictions of plain text,"

and leave just "Many users have turned to Markdown for document processing"

# 2. Markdown Media Type Registration Applications

General:

I have my doubts about creating a registry of processors.  Could the required 
information for interoperability not be captured by capturing the processor 
capabilities as rules?

For comparison, consider the example of HTTP feature negotiation (type, 
language, encoding, etc.) vs UA string testing and/or user-agent sniffing, which 
is frequently regarded as a poor way to do content matching.  This specification 
appears to be blessing an approach analogous to UA string testing.

Maybe this was discussed and I missed it?

The description of "processor-args" seems odd to me - it seems to tie the media 
type string to a particular form of implementation (posix commands).  Would it 
not be more flexible to use some kind of attribute/value list (e.g. similar to 
media type parameters themselves), and let the application turn them into 
command line options or environment variables or whatever is needed?  (As you 
plan to allow references to web resources here, maybe use something like encoded 
JSON, which can be a common representation for direct or indirect values?)

I think such an approach could also sidestep some of the unresolved security 
concerns in the draft (e.g. [[TODO: discuss the implications of processor-args, 
and safeguards.]], etc.)

Para "Interoperability considerations":

Contains the text: "When it is desirable to reflect the author's intent in the 
output, stick with the flavor identified in the flavor parameter."  What is this 
"flavor" parameter?  I'm not seeing it.

[later: looks like left over from a previous incarnation - maybe worth a global 
search for changed names?]

# 4.  IANA Considerations

Is it really necessary to have "expert review" for these registries?  That 
requires a volunteer and may impose some additional overhead on IANA.  Would 
"First come first served" not work here?

What are the requirements for updating a registry entry?  I'd suggest including 
an "escape" clause that allows IESG or an IETF-stream RFC to update any entry. 
(I'd trust the community to not do this capriciously).

Rather than reserve some names for future use, why not just pre-register them 
(even if the descriptions are vague for now, allowing that they can be updated 
later)?

#g

---

From: Graham Klyne <[email-masked]>
Date: Tue, 16 Sep 2014 15:10:11 +0100

Further to my previous review, I just noticed another thing.   The draft 
indicates it is intended to be informational.  Is it allowed in IETF process for 
an informational RFC to create an IANA registry?  I don't know, just asking.  (I 
usually associate such actions with standards track or BCP documents.)

Now that I've had a little time to reflect on this, Ill say:

1. That I think it's a reasonable idea to define this media type, and that the 
broad approach is OK, but ...

2. I think the spec could do with some trimming and polishing, and

3. I'd prefer to see the 'processor' and 'processor-args' parameters dropped 
completely, and focus on creating a tabulation of rules to capture information
about the varieties of Markdown that are actually used.

(And a thought: rather than requiring the rule definitions to resolve any 
potential conflicts between themselves, have a simple left-to-right processing 
of parameters such that where there is a conflict, parameters appearing later in 
the list override earlier ones.  That would provide a well-defined way to say 
something like "github flavoured markdown, except that newlines are not rendered 
as line breaks in the output".)

#g
--

---

From: "Fletcher T. Penney" <[email-masked]>
Date: Tue, 16 Sep 2014 12:43:41 -0400

In the interest of furthering discussion...

I actually prefer the opposite (well -- keep `processor`, but get rid of 
`processor-args` and the feature list) .

 From a practical standpoint, are there more than 10 people in the world 
who are going to define lists of arguments to ask for specific features 
from Markdown-like processors, when those specific features aren't even 
clearly defined anywhere?  Who's going to write software to take an 
arbitrary feature list, and then try to match it up with the 
ever-growing number of Markdown-like implementations, each with their 
own slightly different take on features and syntax?

For example, you request "footnote" support.  Do you mean MultiMarkdown 
footnotes?  PHP Markdown Extra Footnotes?  Pandoc footnotes?  Something 
else?  As for MMD, do you mean the more recent footnotes that include 
"inline" footnotes, or just the old-school reference style footnotes? 
Which HTML formatting do you mean?  Wrapped in a `<div>`?  Mixed in with 
citations, or as two separate lists?

You think CommonMark sparked a ####storm?  I imagine that trying to 
define a canonical feature list is likely to cause just as big of 
one....  Who gets to keep "footnote" and who gets stuck with 
"footnote-brand-x"?

Standardizing certain features across implementations would be nice.  It 
has been brought up countless times on the discussion list without any 
traction.  IMO, this effort is not the forum to try to sneak that in 
through the back door.

I think this is a very deep rabbit hole that is unlikely to lead to 
anything concretely useful.

On the other hand, if you say "Pandoc", then I have a rough idea of what 
is likely to work and what isn't when I use MultiMarkdown instead.  I 
also have an obvious solution handed to me and giftwrapped -- use pandoc 
if the document isn't parsing as expected.  As a user, a list of 
expected features would not be particularly helpful in troubleshooting.

Fletcher

On 9/16/14, 10:10 AM, Graham Klyne wrote:
<snip>
> 3. I'd prefer to see the 'processor' and 'processor-args' parameters
> dropped completely, and focus on creating a tabulation of rules to
> capture information
> (And a thought: rather than requiring the rule definitions to resolve
> any potential conflicts between themselves, have a simple left-to-right
> processing of parameters such that where there is a conflict, parameters
> appearing later in the list override earlier ones.  That would provide a
> well-defined way to say something like "github flavoured markdown,
> except that newlines are not rendered as line breaks in the output".)
>
> #g
> --
>
>

-- 
Fletcher T. Penney
[email-masked]

---

From: Graham Klyne <[email-masked]>
Date: Wed, 17 Sep 2014 09:29:31 +0100

On 16/09/2014 17:43, Fletcher T. Penney wrote:
>  From a practical standpoint, are there more than 10 people in the world who are
> going to define lists of arguments to ask for specific features from
> Markdown-like processors, when those specific features aren't even clearly
> defined anywhere?  Who's going to write software to take an arbitrary feature
> list, and then try to match it up with the ever-growing number of Markdown-like
> implementations, each with their own slightly different take on features and
> syntax?
>
>
> footnotes?  PHP Markdown Extra Footnotes?  Pandoc footnotes?  Something else?
> As for MMD, do you mean the more recent footnotes that include "inline"
> footnotes, or just the old-school reference style footnotes? Which HTML
> formatting do you mean?  Wrapped in a `<div>`?  Mixed in with citations, or as
> two separate lists?
>
> You think CommonMark sparked a ####storm?  I imagine that trying to define a
> canonical feature list is likely to cause just as big of one....  Who gets to
> keep "footnote" and who gets stuck with "footnote-brand-x"?
>
> Standardizing certain features across implementations would be nice.  It has
> been brought up countless times on the discussion list without any traction.
> IMO, this effort is not the forum to try to sneak that in through the back door.
>
> I think this is a very deep rabbit hole that is unlikely to lead to anything
> concretely useful.

It was not my intent in suggesting to stick with rules rather than processors to 
define a One True Way for every feature.  It would be perfectly reasonable in my 
view to introduce a rule that corresponds to a set of features supported by a 
particular processor (I mentioned GitHub as an example previously).

But I think it's also useful to be able to refer to particular features in 
isolation from a processor, as then there's a possibility for a community of use 
to converge around some set of features.  E.g. Scholarly Markdown [1] might be 
expressed in such terms.

[1] http://blog.martinfenner.org/2013/06/17/what-is-scholarly-markdown/

>
> On the other hand, if you say "Pandoc", then I have a rough idea of what is
> likely to work and what isn't when I use MultiMarkdown instead.  I also have an
> obvious solution handed to me and giftwrapped -- use pandoc if the document
> isn't parsing as expected.  As a user, a list of expected features would not be
> particularly helpful in troubleshooting.

Hmmm, but specifying just "Pandoc" wouldn't tell you what Pandoc options should 
be used when rendering a document.  Rules that can specify individual features 
could encompass that, IMO.

My suggested starting point for populating a table of rules (I'm skirting around 
the term "registry" here) would be to look at the various features that can be 
enabled in Pandoc by command line switches, plus a few common feature sets like 
'original', 'github', 'pandoc-default', etc.

We don't currently have enough experience to know how to best describe Markdown 
flavours, particularly in a way that maximizes interoperability, so I'd argue 
for simplicity and flexibility, which I think a list of "rules" can be.

#g
--

---

From: "Fletcher T. Penney" <[email-masked]>
Date: Mon, 22 Sep 2014 09:52:49 -0400

On 9/17/14, 4:29 AM, Graham Klyne wrote:

> We don't currently have enough experience to know how to best describe
> Markdown flavours, particularly in a way that maximizes
> interoperability, so I'd argue for simplicity and flexibility, which I
> think a list of "rules" can be.

I absolutely agree with this sentiment, but I believe you and I have 
different definitions of the word simplicity.  ;)

For example:

	Content-Type: text/markdown; charset=UTF-8; flavor=GitHub;
		processor="MultiMarkdown-3.0 -c"

In my view, this is bordering on the edge of simplicity vs complexity. 
I'm already wondering, did you mean GFM or MMD?  Why specify both? (This 
is a rhetorical question, BTW)  But to add anything else (e.g. a list of 
which options are enabled, which are disabled, etc.) is really getting 
too complicated, IMHO.

The reality of the Markdown ecosystem is complex, messy, and still in 
flux.  To try to create a complex schema to accurately classify the 
author's exact intention with this level of specificity seems to be a 
bit of a fool's errand at this time.  Stick to the 50,000 foot view -- 
specify a "flavor/processor/whatever" and leave it at that.  If the 
processor is allowed to include command line arguments as a single 
string (as above), then that might be a necessary evil. But at least it 
should take care of allowing identification the author's intent.

FTP

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 22 Sep 2014 10:42:08 -0700

On Mon, Sep 22, 2014 at 6:52 AM, Fletcher T. Penney <
[email-masked]> wrote:

> author's exact intention with this level of specificity seems to be a bit
> of a fool's errand at this time.  Stick to the 50,000 foot view -- specify
> a "flavor/processor/whatever" and leave it at that.  If the processor is
> allowed to include command line arguments as a single string (as above),
> then that might be a necessary evil. But at least it should take care of
> allowing identification the author's intent.

+1.  Moreover, providing more direction than that, especially down to the
level of command line options and minimum package versions, smells a lot
like a security problem waiting to happen.

-MSK, just participatin'

---

From: Michel Fortin <[email-masked]>
Date: Mon, 22 Sep 2014 14:34:25 -0400

Le 22-sept.-2014 à 9:52, Fletcher T. Penney <[email-masked]> a écrit :

> If the processor is allowed to include command line arguments as a single string (as above), then that might be a necessary evil. But at least it should take care of allowing identification the author's intent.

Well, command line arguments might for some implementations of Markdown. I can't claim to have surveyed them all, but many Markdown processors out there are actually libraries that often come with no command line syntax, but with a programatic API meant to be called in a specific language. This is the case of PHP Markdown for instance. To me, the whole part that deals with processor arguments just seems out of place.

-- 
Michel Fortin
[email-masked]
http://michelf.ca

---

From: "Fletcher T. Penney" <[email-masked]>
Date: Mon, 22 Sep 2014 14:37:52 -0400

I would suggest that this is an argument for my original position of 
keeping this really simple.  Specify a desired flavor of Markdown, and 
leave it at that.

I suspect that practically speaking, if any further information is 
needed to correctly process a piece of text, an automated system of rule 
parsing is not going to get it right anyways....

F-

---

From: Sean Leonard <[email-masked]>
Date: Mon, 22 Sep 2014 12:19:54 -0700

On 9/22/2014 6:52 AM, Fletcher T. Penney wrote:
> I'm already wondering, did you mean GFM or MMD?  Why specify both? 
> (This is a rhetorical question, BTW)

[Yes, I know it's a rhetorical question, but...you asked... ;) ]

In that particular case I wanted "real world" examples, and was not 
inclined to provide two distinct examples in the draft, so I crammed 
both into one header.

A more real-world example (based on draft-02, which is forthcoming), is:
Content-Type: text/markdown; flavor=Original;
  processor="Markdown.pl-1.0.2b8 --html4tags"

That is a real case. It is apparently widely known in the Markdown 
community that Gruber's own implementations are at variance with his 
syntax rules. Markdown.pl version 1.0.1 and 1.0.2b8 produce meaningfully 
different results; in fact in a few cases the results are exactly 
opposite by design. The Readme docs basically say that Gruber decided 
some other way was better, so he changed it and bumped the version.

Then there is the issue of --html4tags. With that argument you get <br>; 
without it you get <br/> (and other things). For people attempting to 
output some kind of XHTML, this is a meaningful difference, even if the 
implementation doesn't guarantee strict XHTML output.

Fortunately in the case of Markdown.pl, there is only one argument that 
matters, and it's a binary yes/no option. Thus the whole thing can be 
documented and encapsulated without needing to deal with m x n x o 
permutations.

Sean

---

From: Ned Freed <[email-masked]>
Date: Mon, 22 Sep 2014 12:52:43 -0700 (PDT)

> like a security problem waiting to happen.

+1. The point of registering things is to define an appropriate type label
and registration data, including documenting any security issues.

Cleaning up the media type itself in any sense is not a prerequisite to
registration and in this case is not something the IETF should be doing.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Mon, 22 Sep 2014 13:19:37 -0700

>>> Stick to the 50,000 foot view 

Yes, but...

The 50,000 foot level rarely permits interoperability.

My concern is that what gets registered needs to contain enough
information for the recipient of the media type to be able to know what
is needed to process what the author put there.

To the extent that the world of this media type involves heuristics and
other imprecisions, that's unfortunate; to the extent that that world
suffers those hassles successfully, fine.  No, it's not our job to fix
their world.

However the writeup needs to be clear/explicit about any lack of clarity
in this...

And to the extent that one use of the media type fits one version of the
world and another fits a different one, and heuristics used in the field
aren't generally successful, and the label doesn't distinguish, then we
/added/ to the imprecision.

It has sounded as if recent developments in that community have created
some conflicting cases.  At the least what we define for registration
needs to permit easily distinguishing between them.

d/

---

From: Michel Fortin <[email-masked]>
Date: Mon, 22 Sep 2014 16:32:36 -0400

Le 22-sept.-2014 à 15:19, Sean Leonard <[email-masked]> a écrit :

> Then there is the issue of --html4tags. With that argument you get <br>; without it you get <br/> (and other things). For people attempting to output some kind of XHTML, this is a meaningful difference, even if the implementation doesn't guarantee strict XHTML output.

I'd question whether it is a good idea to say anything about the output. Has the desired output format anything to do with the type of the Markdown content?

If you're going to specify output arguments for the sake of reproducing the exact same output, wouldn't it be simpler and more robust to just pass the HTML output around instead of the Markdown source? If you really need to keep the Markdown source around, can't you standardize on a workflow using multipart/alternative that includes both the Markdown source and the resulting HTML?

I can see why specify the flavor because this has something to do with how the Markdown syntax is parsed. A small set of configuration variables could be relevant if they affect how the document is interpreted by the Markdown processor. But details in the output such as XHTML/HTML termination of tags has little to do with the document itself, and the output you want will actually change depending on the context.

-- 
Michel Fortin
[email-masked]
http://michelf.ca

---

From: "John Levine" <[email-masked]>
Date: 22 Sep 2014 21:45:21 -0000

>is needed to process what the author put there.

This is sounding a lot like RFC 4263, for text/troff, where you can't
usefully format the troff without knowing what options and pre- or
postprocessors to use.  So it has a process option to give people
a hint of what flavor of troff it is:

      Content-Type: text/troff ; process="dformat | pic -n | troff -ms"

I'm not saying this is a fabulous approach, but it's one we've used
before.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Mon, 22 Sep 2014 18:09:03 -0700 (PDT)

> is needed to process what the author put there.

Have you spent any time working with various markdown formats? I have, and one
of the characteristics of it that I find most appealing is that it tends to
produce something usable in almost all cases.

It may not be exactly what the author intended, but if your representation
needs are that exacting, you're shopping in the wrong market, and I suggest
heading over to the one of the kiosks (PDF/Tex/Postscript/whatever) across the
street.

> However the writeup needs to be clear/explicit about any lack of clarity

Absolutely.

> /added/ to the imprecision.

My personal preference would be to have a single parameterized type
that covers a range of markdown formats.

> needs to permit easily distinguishing between them.

Speaking as an outside observer who may not be aware of all the specifics,
I think this really depends on what you mean by "conflict". If you 
mean "produces output that's not bit-for-bit compatible", then that's
nothing new.

If you mean "produces different and less than optimal output in some
situations", then yeah, I think that's what's happened, and it's 
unfortunate but not fatal.

But if you mean "produces indecipherable crap in some situations", then
I really need to see some specific examples.

				Ned

---

From: Graham Klyne <[email-masked]>
Date: Tue, 23 Sep 2014 09:57:50 +0100

> heading over to the one of the kiosks (PDF/Tex/Postscript/whatever) across the

+1.

I have used Markdown in a context with exacting output format requirements, and 
those requirements were satisfied not by some "standard" specification, but by 
controlling the toolchain (using PanDoc to feed LaTeX).  But in the document 
production process, Markdown provided a very useful tool for sharing and 
reviewing draft text without getting bogged down in formatting minutiae - IMO 
that's its value.

Sean:  I think that something missing from your introduction is that users may 
turn to Markdown for its "humane" qualities, even though (or even because) it 
requires surrendering a degree of fine control over the output generated.

#g
--

---

From: Sean Leonard <[email-masked]>
Date: Tue, 23 Sep 2014 11:53:53 -0700

+1 to the concern.

We should find out if an informational RFC can create an IANA registry.

+1 also to the point about left-to-right processing of parameters to resolve conflicts.

The CSS resolution mechanism is also a good one. In Cascading Style Sheets, if a style property (or whatever) is unrecognized, the whole thing *and everything after it* is ignored, until a new thing is encountered. (Note that this is from my own memory, and recent CSS standards updates may have changed the behavior.)

Sean

On Sep 16, 2014, at 7:10 AM, Graham Klyne <[email-masked]> wrote:

> Further to my previous review, I just noticed another thing.   The draft indicates it is intended to be informational.  Is it allowed in IETF process for an informational RFC to create an IANA registry?  I don't know, just asking.  (I usually associate such actions with standards track or BCP documents.)
> 
> 1. That I think it's a reasonable idea to define this media type, and that the broad approach is OK, but ...
> 
> 3. I'd prefer to see the 'processor' and 'processor-args' parameters dropped completely, and focus on creating a tabulation of rules to capture information
> (And a thought: rather than requiring the rule definitions to resolve any potential conflicts between themselves, have a simple left-to-right processing of parameters such that where there is a conflict, parameters appearing later in the list override earlier ones.  That would provide a well-defined way to say something like "github flavoured markdown, except that newlines are not rendered as line breaks in the output".)
> 
> #g
[SNIP]

---

From: Sean Leonard <[email-masked]>
Date: Tue, 23 Sep 2014 12:05:27 -0700

> Further to my previous review, I just noticed another thing.   The draft indicates it is intended to be informational.  Is it allowed in IETF process for an informational RFC to create an IANA regi…

From my read, the answer is yes.

Sources:
http://www.iana.org/protocols
RFC 5226 Section 4.2. What To Put in Documents That Create A Registry:
   It is the working group and/or document author's job to formulate an
   appropriate policy and specify it in the appropriate document.  In
   almost all cases, having an explicit "IANA Considerations" section is
   appropriate.

Since a working group is not required, I don’t think Standards Track or BCP is required, either.

In any event, a number of registries are created and updated with Informational RFCs. For example, see Autonomous System (AS) Numbers:
RFC 7249 (Informational)
RFC 7020 (Informational)
RFC 6996 (BCP)
RFC 6793 (Standards Track)
RFC 5398 (Informational)
RFC 1930 (BCP)

-Sean

---

From: "Fletcher T. Penney" <[email-masked]>
Date: Tue, 23 Sep 2014 15:22:16 -0400

> is needed to process what the author put there.

I expect that vast majority of use cases will fall into 1 of 2 camps:

1) The content is basic Markdown.  Any flavor that didn't go off the reservation will process it correctly, or close enough. (e.g. `This is **really** important.`)

2) The content is really something else (e.g. MultiMarkdown).  In this case, the user will either have the desired software as part of their installed software or not.  But giving them a list of requirements is unlikely to change what they do("Ok.  You need to find this particular flavor of Markdown.  Yes, I know you'll never use it again, but  you need to install it now.  Don't worry, I'll wait the 15 minutes while you figure that out.  And then you need to configure it this particular way…."  At this point the user moves on to another web site and finds the content in a .doc file, since that is now easier to use.)

I think this discussion has forked into two separate discussions, though we all think we're talking about the same thing.

1) How do we indicate that a piece of content is not plain text, but instead Markdown, or something really close to Markdown?

vs

2) How do we describe the exact series of steps that one person took in order to convert plain text into a piece of HTML (or other formatted information), so that another person can exactly replicate that process without having to just read the directions.

I would argue that the first is really important for the idea of a media type.  The second is only important in the context of a given piece of software (e.g. a CMS, or whatever), and in that case whatever is decided here is unlikely to be of much use.

I mean, if it's that crucial that the output matches, just use the raw HTML (as I believe Michel suggest), or include directions for those who care.  I suppose, ultimately the decision is not really up to me, but I strongly feel that this project is heading off in a direction that will result in a dead-end, or being incredibly useful for the five people on the internet who care.

If it's kept simple, e.g. "text/markdown" and **MAYBE** the ability to specify a flavor (but even that is iffy and politically charged) then I think it will gain a great deal of traction.  I just worry that getting into all this hair-splitting is likely to have more negative effects than positive effects.

FTP

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 23 Sep 2014 12:58:52 -0700

On this one procedural point:

> Since a working group is not required, I don’t think Standards Track or

It is possible for a non-WG document to get on the Standards Track, namely
by being sponsored by an Area Director.

Regardless, my understanding is also that an Informational document can
create or update a registry, so we're fine here.

I can't remember: Did we already discuss whether this draft should be
Standards Track?

-MSK

---

From: Sean Leonard <[email-masked]>
Date: Tue, 23 Sep 2014 13:29:45 -0700

Thanks for the review. Here is some feedback (now with reference to draft-02).

On Sep 16, 2014, at 3:31 AM, Graham Klyne <[email-masked]> wrote:

> This seems a bit bloated.  I don't think anything relevant is lost by deleting from "Compare with [RFC6838] Section 4.2.1." to the end of the paragraph.

I admit it’s a bit academic/pedantic.

At the same time, I occasionally wonder to myself, “you know, there are all these sad, lonely, unused control characters…they’re just taking up space in the table. Why can’t we use them to tag things or signal other out-of-band information? It would solve a lot of the escaping problems, which lead to security holes.”

Then I remember: it’s because you can’t type them. They’re not printable. You can’t represent them on paper as they are. And people are lazy. Give me a <tag> over an ESC-sequence any day, they say. It is the way it is.

> and leave just "Many users have turned to Markdown for document processing”

I might remove that language prior to publication. However, I think that the language captures the Zeitgeist.

People don’t want to deal with HTML directly anymore because it is getting too complicated, and because you have to deal with “tag balancing”. Who likes tag balancing by hand??? Tag balancing forces you into the think of it, not the feel of it.

As for word processors, see <http://blogs.law.harvard.edu/pamphlet/> and <http://blogs.plos.org/mfenner/2012/12/13/a-call-for-scholarly-markdown/>.

> I have my doubts about creating a registry of processors.  Could the required information for interoperability not be captured by capturing the processor capabilities as rules?
> 
> For comparison, consider the example of HTTP feature negotiation (type, language, encoding, etc.) vs UA string testing and/or user-agent sniffing, which is frequently regarded as a poor way to do content matching.  This specification appears to be blessing an approach analogous to UA string testing.
> 

First of all, draft-02 is much clearer about how the registry looks (registries look) like. This should give us something concrete at which to lob shells.

It’s an active question whether we need flavors (rules) and processors, just one, just the other, or neither.

Ignoring the flavors (rules) for the rest of this response:

The issue is that in the Markdown world, people don’t write specs at first glance. They write implementations. If someone doesn’t like how MultiMarkdown or Markdown.pl do it, they’ll just create their own thing and do it “better”. (And in so doing, they’ll probably throw in a couple of kitchen sink items…) Unlike undertaking the writing of an entire modern web browser, any professional software engineer can basically hack out a Markdown processor in a few hours. For that matter, you can get most of the way on repeated applications of regular expressions.

Just because someone creates a spec (e.g., CommonMark) doesn’t mean it will get adopted. And people will sometimes deliberately disagree with the spec and do their own thing. CommonMark itself is guilty of this: there are a couple of places where they disagreed with Gruber’s writeup, and did something else. Referring to a concrete implementation eliminates ambiguity.

The other thing is that the vast majority of Markdown processors are free and portable. PHP Markdown depends on PHP; Markdown.pl depends on Perl; Python-Markdown needs Python. But PHP runs on all operating systems; likewise with Perl; likewise with Python. And then the C implementations, like stmd and MultiMarkdown, are more-or-less ISO or POSIX compliant.

> 
> 
> The description of "processor-args" seems odd to me - it seems to tie the media type string to a particular form of implementation (posix commands).  Would it not be more flexible to use some kind of attribute/value list (e.g. similar to media type parameters themselves), and let the application turn them into command line options or environment variables or whatever is needed?  (As you plan to allow references to web resources here, maybe use something like encoded JSON, which can be a common representation for direct or indirect values?)
> 
> I think such an approach could also sidestep some of the unresolved security concerns in the draft (e.g. [[TODO: discuss the implications of processor-args, and safeguards.]], etc.)

In draft-01, I thought that referring to the POSIX standard would make things easier and well-specified, since “Everyone Understands POSIX®”.

Not actually the case! POSIX made things more complicated, not less. Too many options. So I got rid of POSIX in draft-02.

The central problem with an attribute/value list, or generally “some other syntax”, is literally that it’s some other syntax, and therefore it imposes additional documentation burdens on implementers. I wanted to stick with a command-line metaphor (i.e., an ordered, unnamed argument list) because “most” processors are command-line tools.

I suppose that statement is not actually true…for example, PHP Markdown Lib 1.4.1 is a library, and as a library <https://michelf.ca/projects/php-markdown/configuration/>, it expects arguments to be named (because they are essentially named member variables). Admittedly I did not seriously consider named arguments in draft-02. On the other hand, it is pretty easy to map named arguments to an unnamed list. For example, for PHP Markdown, the following Argument Syntax (see draft-02 Section 5.2) could be used:
 --empty_element_suffix {suffix}
 --tab_width {width}
 --no_markup
 --no_entities
 --predef_urls_and_titles {urls_and_titles}
   where: urls_and_titles is a single argument; URIs are separated by whitespace; if there is a title, the title is separated from the URI by the | character and terminated with the | character (which is not a valid URI character and therefore is safe to use as a delimiter)

That is just an example. Of course, if Michel Fortin, the implementer of PHP Markdown, does not want to permit this flexibility, it is his choice to register “PHPMarkdown” (or whatever identifier) and simply say “no arguments”. Or only some arguments could be registered. For example, empty_element_suffix and tab_width are not directly relevant to HTML content—users aren’t going to see the difference when they open the webpage in a web browser. But, they will see a difference with no_entities; they will also get a qualitatively different experience if predefined URLs are not passed.

Basically we could introduce a name/value syntax into the processor parameter, such as @NAME=VALUE or some such. But ultimately that introduces more complexity (what are valid NAME characters? How do we escape the VALUE characters? What if the option really is a boolean, like no_markup—aren’t we being pedantic by insisting on VALUE all the time?), and I thought that we want to reduce complexity. (Of course, we could eliminate complexity by removing processor arguments from the processor parameter entirely. But that also reduces precision, which reduces interoperability.)

In addressing these topics, draft-02 strikes a balance between processor implementers and media type parsers. If the implementer (who is registering the processor) wants interoperability and wants the swiss army knives, the implementer can register them. If they don’t want interop, they don’t have to register them. Maybe the implementer put the options in the code, but in retrospect, thinks “well this option is stupid, nobody actually uses this/nobody is actually going to use this”. Or they put the option in for valid debugging reasons, but debugging reasons are not interchange reasons. In this sense, you could say that it gives the implementer a second bite at the apple.

In retrospect, perhaps I should have written a stronger statement about whether to honor processor arguments: “A receiver that receives unregistered arguments SHOULD NOT assume what those arguments mean, and SHOULD NOT pass them to the processor”.

Security considerations are now fully fleshed out in draft-02.

> Contains the text: "When it is desirable to reflect the author's intent in the output, stick with the flavor identified in the flavor parameter."  What is this "flavor" parameter?  I'm not seeing it.
> 
> [later: looks like left over from a previous incarnation - maybe worth a global search for changed names?]

Yup, that missed draft-01. It’s aligned in draft-02.

> 
> 
> # 4.  IANA Considerations
> 
> Is it really necessary to have "expert review" for these registries?  That requires a volunteer and may impose some additional overhead on IANA.  Would "First come first served" not work here?

I spoke with Murray Kucherawy separately about the standard of review off-list (thanks for the feedback). Draft-02 is clearer in lowering the bar. In particular, processor registrations are First-Come, First-Served. Flavor registrations are sort of First-Come, First-Served+ (the + meaning “sanity checking”).

> What are the requirements for updating a registry entry?  I'd suggest including an "escape" clause that allows IESG or an IETF-stream RFC to update any entry. (I'd trust the community to not do this capriciously).

Sorry that I missed this suggestion in draft-02. It’s a good suggestion, and I’ll try to make a note of it for the next draft. Ok, just made the note. :)

> 
> Rather than reserve some names for future use, why not just pre-register them (even if the descriptions are vague for now, allowing that they can be updated later)?

Good point—it should be done.

In draft-02, I removed a lot of the reserved names.

Cheers,

Sean

---

From: Sean Leonard <[email-masked]>
Date: Tue, 23 Sep 2014 13:48:03 -0700

On Sep 23, 2014, at 12:58 PM, Murray S. Kucherawy <[email-masked]> wrote:

> I can't remember: Did we already discuss whether this draft should be Standards Track?

I don’t know if the whole group as arrived at consensus on Standards Track vs. Informational, but I think Ned Freed’s response from July 10 is a good summary:

mid:[email-masked]
http://mailarchive.ietf.org/arch/msg/apps-discuss/VMnuU25ZlBM1IN9_Kk0MYYmn1Cw

To distill: the engineering aesthetic of Markdown differs from the IETF aesthetic. If it were Standards Track, maybe all sorts of heavy lifting would be required, including formally defining the syntax. “But we're not being asked to do that. We're being asked to publish a registration of a name and set of parameters for something that already exists. Nothing more.”

Sean

---

From: Steve Atkins <[email-masked]>
Date: Tue, 23 Sep 2014 14:36:36 -0700

> 1) The content is basic Markdown.  Any flavor that didn't go off the reservation will process it correctly, or close enough. (e.g. `This is **really** important.`)

That's not a particularly well-defined subset, unfortunately. Even something as "basic" as this ...

- basic markdown
- rendering details

1. markdown-ish
2. rendering perfectly

... will render quite differently using different markdown renderers (some will do the "right thing", others will render a single unordered list).

Is that level of uncertainty about the semantic meaning of simple content acceptable? Maybe it is, but it'd lead to text/markdown interoperating about as well as text/html in email did a decade ago (and adding flavours won't help much with that, unless senders can assume that recipients have multiple renderers integrated into their browsers or MUAs).

> 2) The content is really something else (e.g. MultiMarkdown).  In this case, the user will either have the desired software as part of their installed software or not.  But giving them a list of re…

RFC 4263 ("text/troff") might be interesting reading there. Markdown is roff for a new generation and seems to have all the same issues.

Content-Type: text/troff ; process="use pic -n then troff -ms"

Cheers,
  Steve

---

From: Larry Masinter <[email-masked]>
Date: Wed, 24 Sep 2014 00:36:21 +0000

If it's at all possible, put version/processing instructions/profiles as in-band comment / processing instructions, rather than MIME-type parameters, which inevitably are lost

So, if I believe http://stackoverflow.com/questions/4823468/store-comments-in-markdown-syntax
For example you could indicate the flavor and desired processor with that information in a Markdown comment.

[//]: # (  {Markdown-favor: "Original", processor: "Markdown.pl-1.0.2b8 --html4tags"} )

MIME should discourage parameters, since OS's don't seem to get them even after a few decades.

Larry
--
http://larry.masinter.net