Subject: Validing parser required?

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jul 2004 11:02:33 -0400

In the interest of being "cleanly and thoroughly specified", the AtomPub 
Working Group should decide whether consumers should be required to use 
validating XML parsers in order to consistently and correctly handle the 
Atom feed and protocol.

First, here is a relevant portion of the XML specification[1]:

> For maximum reliability in interoperating between different XML
> processors, applications which use non-validating processors should
> not rely on any behaviors not required of such processors.
> Applications which require facilities such as the use of default
> attributes or internal entities which are declared in external
> entities should use validating XML processors.

Now for historical benefit, the Netscape RSS 0.91 spec, revision 3 
contains the following text[2]:

> Files must be 100% valid XML. We're trying to move towards a more
> standard format, and to this end we have included several tags from
> the popular <scriptingNews> format. We have also ensured that this
> version is 100% valid XML. We did this by requiring that a DOCTYPE
> tag be included, and validating each RSS document against that DTD.
> This means that it is not enough for an RSS document to be
> "well-formed". It must also be "valid" with respect to its DTD.

The requirement for a DTD was dropped in both the UserLand 0.91 spec[3] 
(and has not reappeared in subsequent revisions derived from this spec), 
nor was is it present in the RSS 1.0 spec[4].

  - - -

As I said, I would like to see Atom "cleanly and thoroughly specified". 
  People should not be required to use a validating parser simply 
because the spec is silent on whether this is allowed or not, and 
therefore must be allowed.

One way to handle this is the way SOAP[5] does :

> A SOAP message MUST NOT contain a Document Type Declaration.

That seems pretty clear, and certainly is very effective.

However, if someone is interested in crafting more surgical language 
which only addresses the issue of validation and does not prevent, for 
example, inline DTDs, I will gladly defer to them.  Otherwise, I will 
write up a Pace to include something like the sentence above into the 
Atom specifications and such a Pace will eventually get scheduled and 
disposed of.

- Sam Ruby

[1]<http://www.w3.org/TR/2000/REC-xml-20001006#safe-behavior>
[2]<http://my.netscape.com/publish/formats/rss-spec-0.91.html>
[3]<http://backend.userland.com/rss091>
[4]<http://web.resource.org/rss/1.0/spec>
[5]<http://www.w3.org/TR/2000/NOTE-SOAP-20000508/#_Toc478383492>

---

From: James Aylett <[email-masked]>
Date: Fri, 2 Jul 2004 17:02:22 +0100

On Fri, Jul 02, 2004 at 11:02:33AM -0400, Sam Ruby wrote:

> I would like to see Atom "cleanly and thoroughly specified". 

I have two thoughts on this:

 (1) a DocType declaration could be of use in /creating/ an Atom
     document, because it allows a producer to validate immediately on
     production. However if we're stating that an Atom document
     must be able to be understood entirely by a non-validating
     parser, there is no other use of a DocType declaration (at least,
     that I can see) - except for named entity declarations, which
     are only really of use if you're writing this stuff by hand.

 (2) the language to express that will be difficult to make clear. The
     best I've come up with is: 

       An Atom document MUST be expressed such that its meaning is
       identical whether a validating or non-validating XML parser is
       used.

     And I'm not terribly happy with that phrasing in all sorts of
     ways.
 
And of course we can suggest using a schema, not a DTD, for validity
checks. So I'd probably be happy with outlawing DocType declarations
in Atom.

James

-- 
/--------------------------------------------------------------------------\
  James Aylett                                                  xapian.org
  [email-masked]                               uncertaintydivision.org

---

From: Michael Champion <[email-masked]>
Date: Fri, 2 Jul 2004 12:06:52 -0400

On Jul 2, 2004, at 11:02 AM, Sam Ruby wrote:

>
> As I said, I would like to see Atom "cleanly and thoroughly 
> specified".  People should not be required to use a validating parser 
> simply because the spec is silent on whether this is allowed or not, 
> and therefore must be allowed.
>

SOAP 1.2 has much more explicit language about the non-requirement for 
runtime validation that you may wish to borrow. 
http://www.w3.org/TR/soap12-part1/#reltoxml

>> A SOAP message MUST NOT contain a Document Type Declaration.

At least in SOAP 1.2, this is not to prevent a requirement for runtime 
validation but to align SOAP on a subset of XML that does not contain 
entity declarations or references to any other then the built-in XML 
entities.  That is for performance and security reasons that may or may 
not apply to Atom.  This is the mother of permathreads in the SOAP 
world -- while I will strongly defend the removal of entities from 
SOAP, I am not prepared to do that for Atom.  On the other hand, you 
may wish to consider the "billion laughs" XML denial of service attack 
http://www.securityfocus.com/archive/1/303509/2002-12-13/2002-12-19/0 
and determine how / whether to address it in Atom.  Forbidding DTDs is 
at least a simple way of doing this, but again the cost for Atom is 
probably prohibitive.  (Anticipating much stronger statement of this 
from Elliotte Rusty Harold  ... <grin> )

---

From: Robert Sayre <[email-masked]>
Date: Fri, 02 Jul 2004 12:24:28 -0400

>       identical whether a validating or non-validating XML parser is

How about the following?

Cribbing a bit from the section 5.2 of the XML spec:

"Atom documents can be processed by non-validating XML processors. An 
Atom document MUST NOT rely on any behaviors not required of such 
processors."

Robert Sayre

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jul 2004 18:28:46 +0200

Sam Ruby wrote:

 > ...

I think this discussion is pointless unless there'll be a normative 
schema (in the generic sense) that *can* be used to validate Atom. As 
far as I understand namespace usage and extension rules already rule out 
DTDs and XML Schema.

Are we prepared to require something else like RNG? I doubt so.

On the other hand (as others pointed out), allowing document types 
introduces several kinds of Denial-of-Service attacks, and generally 
makes little sense, unless the recipient *always* validates against it's 
own copy of the DTD instead of the one reference in the request.

WebDAV faces a similar problem, and the current consensus seems to be:

- recipients MUST NOT attempt to validate the message, and
- recipients MAY reject certain DTD subsets in requests (such as when 
external entities are referenced, or the recipient detects situations 
like the one called the "billion laughs attack"

Regards, Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 02 Jul 2004 17:44:58 +0100

Sam Ruby wrote:

 > [...]

Inline DTDs aside, a DTD for Atom itself doesn't sound feasible 
given the current spec wording on ordering [1].

cheers
Bill

[1] http://www.intertwingly.net/wiki/pie/PaceElementOrder

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jul 2004 12:47:34 -0400

Julian Reschke wrote:

> Sam Ruby wrote:
> 
>>
>> In the interest of being "cleanly and thoroughly specified", the 
>> AtomPub Working Group should decide whether consumers should be 
>> required to use validating XML parsers in order to consistently and 
>> correctly handle the Atom feed and protocol.
> 
>  > ...
> 
> schema (in the generic sense) that *can* be used to validate Atom.

I don't understand what you are trying to say.

What I would like to see is some normative text that I can use to 
justify adding a check in the feedvalidator[1].  My preference is that 
such normative text promotes interoperability by discouraging or 
outright prohibiting features which pre-req a validating parser.

Does this make sense?

- Sam Ruby

P.S.  If it is relevant, the current incarnation of the feedvalidator is 
written in Python.  It is open source, and can be found at [2].

[1] http://feedvalidator.org/
[2] http://sourceforge.net/projects/feedvalidator

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jul 2004 19:02:49 +0200

> I don't understand what you are trying to say.

Talking about using a validating parser implies that their exists a 
normative DTD/XML Schema/RNG Schema/Whatever against which Atom *can* be 
validated. Futhermore, in the strict sense defined by the XML REC, that 
would need to be a DTD (the XML REC itself does not define any other 
validation method).

> outright prohibiting features which pre-req a validating parser.

Yes (sort of :-).

I think we need to distinguish between (at least) two things:

1) Validation of XML in the sense of using a DTD/Schema whatever to add 
specific constraints on the allowable content and

2) the ability to use internal DTD subsets to define external inclusions 
or entities.

I think 1) is only possible if there is a normative Schema (in whatever 
schema language) that actually *can* be used to validate Atom.

For 2), I think the spec should allow recipients to reject/ignore 
certains parts of an internal DTD subset, thereby requiring producers 
not to use those.

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jul 2004 13:05:29 -0400

Bill de hÓra wrote:

>> The requirement for a DTD was dropped in both the UserLand 0.91 
>> spec[3] (and has not reappeared in subsequent revisions derived from 
>> this spec), nor was is it present in the RSS 1.0 spec[4].
> 
> Inline DTDs aside, a DTD for Atom itself doesn't sound feasible given 
> the current spec wording on ordering [1].

There's a difference between "a full and general purpose DTD isn't 
feasible" and "usage of DTDs for specific purposes are not allowed".

An example of an attempt to use a DTD for the purpose of importing a set 
of predefined character references is described here:

http://philringnalda.com/blog/2004/07/ah_sweet_irony.php

I either want to identify this as something that is clearly legal (and 
therefore something a conformance test should be written for), or 
discouraged/illegal (and therefore something that the feedvalidator 
should detect).

- Sam Ruby

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Fri, 02 Jul 2004 19:04:42 +0200

> validating XML parsers in order to consistently and correctly handle the  

I think that Atom SHOULD NOT use a DOCTYPE or DTD to validate. I'm not  
sure it should be enforced validated at all, but if it should, the way to  
do it is with XSD, RNG or some other more updated definition language.

So, I'd say a clear and bastant «no» to DOCTYPE's and DTD's, but a «dunno»  
to other validation mechanisms. I think a normative XSD would be nice, but  
how we would knit it into the specification is something I have no clue  
about.

-- 
Asbjørn Ulsberg         -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: Walter Underwood <[email-masked]>
Date: Fri, 02 Jul 2004 10:14:36 -0700

>        identical whether a validating or non-validating XML parser is

That is pretty good. Here is a simpler version. We should be specific
about what this means, though. This is a profile of XML, right?

  The meaning of an Atom document MUST be the same whether processed
  with a validating or non-validating XML parser.

wunder
--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: Sam Ruby <[email-masked]>
Date: Fri, 02 Jul 2004 13:17:02 -0400

> certains parts of an internal DTD subset, thereby requiring producers 

In order to help focus on specifics, take a look at the following feed:

http://www.oreillynet.com/meerkat/?_fl=rss10&t=ALL&c=47

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Jul 2004 19:22:41 +0200

> discouraged/illegal (and therefore something that the feedvalidator 

 From an pure XML Rec point of view, using an internal DTD subset 
clearly is legal. However, you can't rely on the parser to actually 
resolve externals, so producing content that becomes illformed when not 
definitively is an incredibly stupid thing to do.

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Fri, 02 Jul 2004 11:05:42 -0700

> Atom document MUST NOT rely on any behaviors not required of such 

+1.

And on the SOAP approach (forbidding <!DOCTYPE>); that explicitly flies 
in the face of RFC3470 and if we want to try that, we're going to have 
to come up with persuasive language as to why we're doing something 
that an IETF Best Current Practice says you SHOULD NOT do.  I think 
Robert Sayre's language hits the sweet spot.  Walter's retake is OK too 
if people prefer that.  -Tim

---

From: Walter Underwood <[email-masked]>
Date: Fri, 02 Jul 2004 11:19:14 -0700

> http://www.oreillynet.com/meerkat/?_fl=rss10&t=ALL&c=47

Amusing. They specifically allow the latin 1 entities, but the
feed is pure ASCII, with an ASCII apostrophe for "O'Reilly" and
" -- " for the em dash. Those aren't in Latin 1, but they could use
something outside of ASCII.

Here is a Meerkat feed which uses a numeric ref for ã, which is
in Latin 1.

  <http://www.oreillynet.com/meerkat/?_fl=rss10&t=ALL&c=5800>

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Walter Underwood <[email-masked]>
Date: Fri, 02 Jul 2004 11:35:26 -0700

I prefer this wording, but with MAY instead of "can". --wunder

--On Friday, July 02, 2004 11:05:42 AM -0700 Tim Bray <[email-masked]> 
wrote:

> in the face of RFC3470 and if we want to try that, we're going to have to
> come up with persuasive language as to why we're doing something that an
> IETF Best Current Practice says you SHOULD NOT do.  I think Robert
> Sayre's language hits the sweet spot.  Walter's retake is OK too if
> people prefer that.  -Tim
>

--
Walter Underwood
Principal Architect
Verity Ultraseek

---

From: Mark Pilgrim <[email-masked]>
Date: Fri, 2 Jul 2004 14:34:52 -0400

On Fri, 02 Jul 2004 11:19:14 -0700, Walter Underwood <[email-masked]> wrote:
> feed is pure ASCII

$ curl -sI "http://www.oreillynet.com/meerkat/?_fl=rss10&t=ALL&c=47" |
grep "Content-Type"
Content-Type: text/xml

-- 
Cheers,
-Mark

---

From: Danny Ayers <[email-masked]>
Date: Sat, 03 Jul 2004 11:35:16 +0200

>  with a validating or non-validating XML parser.

I don't see how this could work. Say an element's missing. Part of the 
meaning obtained using a validating parser would be the flag 'invalid'. 
The non-validating parser presumably wouldn't be providing this 
information, the value of the element being null or empty. Or to put it 
another way, the meaning of the document doesn't change, only the 
interpretation ;-)

If I read the spec (5.1) correctly then internal DTDs have to be 
respected whatever, so I'm not sure whether it makes sense to forbid 
importation of external entity definitions (this seems to me to be a 
slightly different issue than that of validation per se).

If a imported set of entities breaks the well-formedness, the feed's 
broken, period. If it doesn't, what's the problem?

Generally I've a feeling the approach where validation is kind-of 
discouraged may be a step too far. There may be apps for which an extra 
level of validation (in the general sense) is required, and checking 
against an e.g. XML Schema could offer that.

Returning to Sam's original query, he says:
[[
As I said, I would like to see Atom "cleanly and thoroughly specified". 
 People should not be required to use a validating parser simply because 
the spec is silent on whether this is allowed or not, and therefore must 
be allowed.
]]

Big +1 to the first sentence, but I think it unlikely anyone would 
interpret the spec being silent on the issue as implying that a 
validating parser is required. But ok, just in case, why not then simply 
add the sentence: "Validation is not a requirement of an Atom processor.".

Regarding the Feed Validator, I don't think there's any extra action needed.

Cheers,
Danny.

-- 

Raw
http://dannyayers.com

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Sat, 03 Jul 2004 21:10:49 +0200

On Sat, 03 Jul 2004 11:35:16 +0200, Danny Ayers <[email-masked]>  
wrote:

> "Validation is not a requirement of an Atom processor.".

But should we, or shuld we not have normative XSD's, RNG's, WSDL's et al?  
If we are, I think we should encourage validation, but I agree that  
requiring it is not feasible.

---

From: Kevin Marks <[email-masked]>
Date: Wed, 7 Jul 2004 17:11:07 -0700

> > Atom document MUST NOT rely on any behaviors not required of such

Can you apply De Moivres theorem to that  so it gets fewer negatives in it?