Subject: Re: Last Call: 'The Atom Syndication Format' to Proposed Standard

From: Robert Sayre <[email-masked]>
Date: Tue, 3 May 2005 11:31:09 -0400

On 4/29/05, Martin Duerst <[email-masked]> wrote:
> At 03:33 05/04/29, Alexey Melnikov wrote:
>  >
>  >Ok, maybe it is just me, but what does it mean to "collapse white-space"?
> Does this mean to replace FWS (in RFC 2822 sense) with a single space?
> 
> Making this more precise is definitely desirable. But there is also
> an i18n issue: This works fine for languages that use spaces between
> words. It doesn't work for languages that don't have spaces between
> words (Chinese, Japanese, Thai,...). If Text elements are only used
> for short things such as names or titles, that's not a big issue,
> the text in question can just be put on a single line. However,
> when the texts in question are long, it's a serious issue, and
> should be fixed.

I believe the intent of this text was to match HTML's text treatment,
so that implementations can avoid preprocessing whitespace.

http://www.w3.org/TR/html4/struct/text.html#h-9.1

Suggestions for less vague text is welcome, but I want to make sure
the text remains comprehensible to non-experts.

Robert Sayre

---

From: Thomas Broyer <[email-masked]>
Date: Tue, 03 May 2005 19:27:13 +0200

Martin Duerst wrote:
>  > >   If the value is "text", the content of the Text construct MUST NOT
>  > >   contain child elements.  Such text is intended to be presented to
>  > >   humans in a readable fashion.  Thus, Atom Processors MAY collapse
>  > >   white-space (including line-breaks),
>  >
>  >Ok, maybe it is just me, but what does it mean to "collapse 
>  >white-space"? Does this mean to replace FWS (in RFC 2822 sense) with a 
>  >single space?
> 

My understanding of type="text" is that this is "just text" without any 
"formatting". Hence, it is not meant to be "preformatted text" such as 
text/plain or inside an (X)HTML "pre".

This means type="text" content is a single paragraph of text. If you 
need paragraphs, lists or any other "structural formatting", you have to 
use type="html" or type="xhtml" with the appropriate content.

I was about to writing a Pace about white-space handling in type="text" 
(either using xml:space or an attribute that would have mimic'd the 
"white-space" CSS property) when I understood/recalled that Text 
Constructs have accessibility in mind (hence their limitation to textual 
contents) and preformatted text is not accessible enough.

-- 
Thomas Broyer

---

From: Graham <[email-masked]>
Date: Tue, 3 May 2005 20:32:04 +0100

On 28 Apr 2005, at 7:33 pm, Alexey Melnikov wrote:

> Ok, maybe it is just me, but what does it mean to "collapse white- 
> space"? Does this mean to replace FWS (in RFC 2822 sense) with a  
> single space?

Since the statement is a MAY, I don't think any exact meaning is  
necessary. It's simply a hint to publishers that whitespace may not  
be preserved.

> when the texts in question are long, it's a serious issue, and

A consumer may do anything that can reasonably be described as  
"collapsing whitespace", but are not required to. How does this cause  
problems in Asian languages?

Graham

---

From: Henri Sivonen <[email-masked]>
Date: Wed, 4 May 2005 09:20:48 +0300

> when the texts in question are long, it's a serious issue, and

You seem to be assuming that the length of a "line" is restricted in 
XML source. Why? As far as I can tell, it should be permissible to 
produce Atom documents that contain no LF or CR characters.

Can't languages without spaces use long source "lines" and apply soft 
wrapping in a source view if necessary? Why is this a wire format 
problem?

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Henri Sivonen <[email-masked]>
Date: Wed, 4 May 2005 13:48:38 +0300

On May 4, 2005, at 12:05, Brian E Carpenter wrote:

> Henri Sivonen wrote:
> Are you suggesting that a canonical format without CRLF should be 
> mandatory?

No, not mandatory, although I expect to produce such feeds myself (even 
in English or Finnish). I am suggesting that pretty-printing line 
breaks should not be introduced in places where normalizing them to a 
space would be inappropriate. So if you have a long string of Chinese, 
Japanese or Thai and you don't want spaces in that string, just don't 
put white space there.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: "A. Pagaltzis" <[email-masked]>
Date: Thu, 5 May 2005 16:09:09 +0200

* Thomas Broyer <[email-masked]> [2005-05-03 19:35]:
> This means type="text" content is a single paragraph of text.
> If you need paragraphs, lists or any other "structural
> formatting", you have to use type="html" or type="xhtml" with
> the appropriate content.

Or type="text/plain", I’d assume?

Regards,
-- 
Aristotle

---

From: Martin Duerst <[email-masked]>
Date: Wed, 04 May 2005 10:39:45 +0900

>My understanding of type="text" is that this is "just text" without any 
"formatting".

That's my understanding, too.

 >Hence, it is not meant to be "preformatted text" such as text/plain or 
inside an (X)HTML "pre".

Yes. But that's exactly where the spacing problems with Chinese/Japanese/Thai
are. There are no such problems for preformatted text, because the line breaking
in the source (as sent) is the same as the line breaking when displayed.
For free-flowing text, however, the line breaks in the source and those in
the display are not (necessarily) the same, and so linebreaks have to be
changed to spaces for Western languages, but to nothing for Chinese/Japanese
(and most possibly to a zero-width non-breaking space for Thai), and the spec
has to say something about this.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 04 May 2005 10:41:49 +0900

>problems in Asian languages?

If the consumer does the right thing, it won't cause problems.
But the chance that the consumer does the right thing without
being told what this is (or, without being told that this is
different for different languages or scripts) is rather low.
So we better improve the spec to help consumers do a better
job.

Regards,    Martin.

---

From: Henri Sivonen <[email-masked]>
Date: Sat, 7 May 2005 11:29:07 +0300

> has to say something about this.

Why would you put line breaks in the CJK source, then? Isn't the 
"problem" solved with the least heuristics by the producer not putting 
breaks there?

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Walter Underwood <[email-masked]>
Date: Sat, 07 May 2005 20:30:15 -0700

--On May 7, 2005 11:29:07 AM +0300 Henri Sivonen <[email-masked]> wrote:
>
> Why would you put line breaks in the CJK source, then? Isn't the "problem"
> solved with the least heuristics by the producer not putting breaks there?

It would be even better if they would just speak English. :-)

White space is not particularly meaningful in some of these languages,
so we cannot expect them to suddenly pay attention to that just so
they can use Atom. There will be plenty of content from other formats
with this linguistically meaningless white space.

If we get this wrong, Atom-delivered content will look broken in
some languages, and a bunch of extra-spec practice will build up about
how to fix it. Much better to get it right in 1.0.

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Henri Sivonen <[email-masked]>
Date: Sun, 8 May 2005 10:26:22 +0300

On May 8, 2005, at 06:30, Walter Underwood wrote:

> they can use Atom.

Why not? We expect them not no insert other random characters there. 
What do the same producers do with XHTML? Opera 7.53 and Safari 1.3 
render a space between the second and third Kanji in
http://hsivonen.iki.fi/test/cjk-whitespace.xhtml

> There will be plenty of content from other formats

Why not just get rid of it in the producer end like you have to get rid 
of form feeds?

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Martin Duerst <[email-masked]>
Date: Sun, 08 May 2005 15:38:45 +0900

People in China, Japan, and so on (Korean uses spaces, so it's not CJK)
tend to use similar tools to those in the western world. Tools for
editing XML, e.g., usually don't make it easy to edit very long lines
because they assume that such long lines can be broken. So it's not
as easy as it looks for the producer.

Regards,    Martin.

---

From: Henri Sivonen <[email-masked]>
Date: Mon, 9 May 2005 10:57:24 +0300

On May 8, 2005, at 09:38, Martin Duerst wrote:

>> "problem" solved with the least heuristics by the producer not 
>> putting breaks there?
>

If the XML editing tools are broken, they should be fixed instead of 
requiring consumers to perform context-sensitive DWIM. Besides, Atom 
feeds are typically produced programmatically instead of editing 
manually, so I think the point about editors is moot. The content in 
Atom feeds may be manually edited, but putting stray line feeds in 
XHTML is not safe, so producers need to take care of not including 
stray line feeds anyway.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Thomas Broyer <[email-masked]>
Date: Mon, 09 May 2005 15:24:38 +0200

> Or type="text/plain", I’d assume?

If you're talking about atom:content, not for Text Constructs.

-- 
Thomas Broyer

---

From: Sam Hartman <[email-masked]>
Date: Mon, 09 May 2005 09:33:42 -0400

>>>>> "Martin" == Martin Duerst <[email-masked]> writes:

    Martin> At 17:29 05/05/07, Henri Sivonen wrote:
    >>> For free-flowing text, however, the line breaks in the source
    >>> and those in the display are not (necessarily) the same, and
    >>> so linebreaks have to be changed to spaces for Western
    >>> languages, but to nothing for Chinese/Japanese (and most
    >>> possibly to a zero-width non-breaking space for Thai), and the
    >>> spec has to say something about this.
    Martin> "problem" solved with the least heuristics by the producer
    Martin> not putting breaks there?

    Martin> People in China, Japan, and so on (Korean uses spaces, so
    Martin> it's not CJK) tend to use similar tools to those in the
    Martin> western world. Tools for editing XML, e.g., usually don't
    Martin> make it easy to edit very long lines because they assume
    Martin> that such long lines can be broken. So it's not as easy as
    Martin> it looks for the producer.

My personal opinion as someone who is very shortly going to have to
evaluate the atom specification is that you've identified an issue
(space and line breaking) for some languages that should be
considered.  Your proposed solution seems highly undesirable in that
it requires us to understand the language of the text being displayed.
In the past we've had all sorts of problems doing that.  Your proposed
solution also seems quite complicated.

It may well be that the solutions to this problem are worse than the
problem itself.  However I think it is important to specifically
understand that is the case rather than failing to solve the problem
because we failed to understand it.

At least based on the discussion the IESG has been copied on, it
doesn't sound like the working group has fully considered this issue.
The responses have more of the character of those found from people
trying to brush aside an issue than of people who have carefully
considered something and concluded there is nothing to be done.

Moreover, thisn issue cannot be unique to atom: it must effect many
XML based protocols both within the IETF and within other standards
organizations.

It may be that the right people haven'tspoken up or that the right
part of the discussion hasn't been copied to the IESG.  I'd sort of
expect the apps ADs and atompub chairs to be familiar with these
issues but so far have not seen them chime in.

Anyway as someone evaluating atompub's output it would be very useful
if the working group responded to this last call comment.  IN my mind
a response would start with a researched description of the issue:
either confirm that Chinese and Japanese and Thai tools work as
described or explain how they actually work.  Then describe what other
standards have done about this problem.  Finally describe what atompub
has done about the problem and why.  

I'm not asking for a lot of text; probably something about as long as
this message.

Such a response would make evaluating this issue much easier when the
document comes before the IESG.

---

From: Robert Sayre <[email-masked]>
Date: Mon, 9 May 2005 17:04:51 -0400

> XML based protocols both within the IETF and within other standards

Martin,

I agree with Sam on both points. Can you give us an example of an XML
format that successfully deals with your issue?

Does XHTML differ from Atom?

Robert Sayre

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 9 May 2005 15:49:38 -0700

>solution also seems quite complicated.

Fully agree. Please note the text in the spec we are working from:

    If the value is "text", the content of the Text construct MUST NOT
    contain child elements.  Such text is intended to be presented to
    humans in a readable fashion.  Thus, Atom Processors MAY collapse
    white-space (including line-breaks), and display the text using
    typographic techniques such as justification and proportional fonts.

FWIW, this appears twice, identically, in the spec.

Martin Dürst brought up CJK (well, actually CJT), saying that they 
don't use inter-word spacing. That's fine, but it is irrelevant to 
the text in the draft. If some text comes through with no spaces, 
there is no white space to collapse. His argument that some XML 
editors make long lines of text difficult to edit is clearly *way* 
out of scope for Atom, or any other XML-using protocol for that 
matter.

>because we failed to understand it.

The "case" is that text that is supposed to be read by humans comes 
in many forms, with different line lengths, and so on. The paragraph 
from the spec says that Atom processors may alter these so that they 
can be presented better for the reader. Of course, they may also 
alter it to make it less readable, as many mail user agents do 
(<sigh>). Regardless, this says that the Atom processor is free to 
present things in text constructs in any fashion it deems suitable. 
This is particularly important for making Atom content accessible; 
for example, the Atom processor can use this rule to present text 
content by reading it aloud, by putting it on a screen greatly 
magnified one character at a time, and so on.

>considered something and concluded there is nothing to be done.

Sorry, but that's unfair. Alexy asked "Ok, maybe it is just me, but 
what does it mean to "collapse white-space"? Does this mean to 
replace FWS (in RFC 2822 sense) with a single space?" Martin's 
response was orthogonal: "Making this more precise is definitely 
desirable. But there is also an i18n issue: This works fine for 
languages that use spaces between words." The rest of the thread 
wandered into the weeds because it was hard to figure out what was 
being discussed.

>XML based protocols both within the IETF and within other standards

Any protocol that has XML that includes human-readable text has this 
issue. Well, the processors of that XML does; the protocols 
themselves do not.

>I'm not asking for a lot of text; probably something about as long as

I believe that it can be a lot shorter: given the rationale above, 
it's not a problem for Atompub or any other XML-using protocol. For 
that matter, it's not really and XML problem at all: it affects text 
formats like HTML and RFC 2822 as well.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: fantasai <[email-masked]>
Date: Mon, 09 May 2005 19:32:46 -0400

> http://hsivonen.iki.fi/test/cjk-whitespace.xhtml

See also Ishida's tests:

http://www.w3.org/International/tests/results/white-space-ideograph

Special handling of white-space in CJK context is accounted for in the
CSS2.1 spec (and will be described in more detail in CSS3 Text).

> Why not just get rid of it in the producer end like you have to get rid 

Because form feeds are normally not used in source code files whereas
line breaks and indendation often are?

~fantasai

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 10 May 2005 08:57:47 -0400

> formats like HTML and RFC 2822 as well.

I have to agree with Paul.  I don't believe that the issue of white space in
the syndicated content is really an Atompub issue.  It might be an issue for
the content creator.  It might be an issue for the reader.  As long as the
pipe between the two passes the content as submitted, though, the pipe has
done its job.

-Scott-

---

From: Graham <[email-masked]>
Date: Tue, 10 May 2005 14:31:53 +0100

> with this linguistically meaningless white space.

So the idea is that whitespace should not appear at all in certain  
texts, and you'd like it to be stripped out at the consumer? There  
are only three possible ways for this to happen:

1) The consumer removes all whitespace, even in western texts
2) The consumer recognizes these languages and removes the whitespace  
automatically
3) The consumer is told what to do by an attribute

(1) is obviously not plausible, but included for completeness. (2) is  
impractical. (3) is plausible, but may or may not end up being  
implemented in all consumers, making it kind of useless.

I don't see how there's a better solution than texts that shouldn't  
be shown with whitespace not containing whitespace in the first  
place, which is what we have.

Graham

---

From: Walter Underwood <[email-masked]>
Date: Tue, 10 May 2005 08:16:11 -0700

> pipe between the two passes the content as submitted, though, the pipe has

If publishers and subscribers have obstacles to using Atom, that sounds
like a problem to me.

"Everyone has this problem" is not a good reason to ignore it. Someone
has to be the first to solve it, might as well be us. It is not acceptable
to build formats for the "English Wide Web". That doesn't exist any more.

wunder

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 10 May 2005 11:39:15 -0400

> -----Original Message-----
> From: Walter Underwood [mailto:[email-masked]] 
> Sent: Tuesday, May 10, 2005 11:16 AM
> To: Scott Hollenbeck; 'Paul Hoffman'; [email-masked]; 'Atom WG'
> Subject: RE: Last Call: 'The Atom Syndication Format' to 
> Proposed Standard
> 
> 
> 

Why?  If the "problem" is in the content creation and consumption processes,
why must the conduit provide the solution?  I agree that Atompub shouldn't
make the problem worse, but I don't believe that it should attempt to solve
endpoint problems.

> to build formats for the "English Wide Web". That doesn't 

Sorry, but I disagree with the "might as well be us" part.  Atompub is
chartered to define a syndication format and protocol, not to solve the
web's i18n problems.

-Scott-

---

From: Henri Sivonen <[email-masked]>
Date: Tue, 10 May 2005 18:46:10 +0300

> to build formats for the "English Wide Web". That doesn't exist any 

I believe the problem should be addressed in the producer end. Not in 
the wire format nor at the consumer end.

No one is preventing non-English text in Atom--just saying that if you 
don't want stuff to appear in the feed do not put the stuff there and 
expect the consumer to take it out. It's the usual "It hurts when I do 
this. -- Don't do it then." situation.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 10 May 2005 09:10:07 -0700

>If publishers and subscribers have obstacles to using Atom, that sounds

It is a problem, of course.

>"Everyone has this problem" is not a good reason to ignore it.

No one is ignoring it. This thread started because the format draft 
pointed out at least one aspect of the problem, which is more than 
most other RFCs do.

>  Someone
>has to be the first to solve it, might as well be us.

May I suggest that there are groups with more experience in the area 
than ours that would be more appropriate? In specific, since this 
problem affects all internationalized text, the Unicode Consortium 
has a much higher chance of "solving" the problem than an IETF 
Working Group who is focused on a syndication format.

If you have a proposed solution to the problem (you didn't include 
one in your message to the WG), the Unicode Consortium is quite open 
to outside input on this type of thing.

>It is not acceptable

That is both grossly insulting to those of us have spent a great deal 
of time trying to make the Internet internationalization-friendly, 
and is also grossly technically inaccurate, unless you consider every 
written language other than Chinese, Japanese Kanji, Burmese, Khmer, 
Thai, Tagalog, Lao, and Tibetan to be "English". (The folks who speak 
all the other languages might find you calling them "English" to be 
insulting too, of course.)

--Paul Hoffman, Director

---

From: Sam Hartman <[email-masked]>
Date: Tue, 10 May 2005 14:14:32 -0400

>>>>> "Scott" == Scott Hollenbeck <[email-masked]> writes:

    >> >I'm not asking for a lot of text; probably something about as
    >> long as >this message.
    >> 
    >> I believe that it can be a lot shorter: given the rationale
    >> above, it's not a problem for Atompub or any other XML-using
    >> protocol. For that matter, it's not really and XML problem at
    >> all: it affects text formats like HTML and RFC 2822 as well.

    Scott> I have to agree with Paul.  I don't believe that the issue
    Scott> of white space in the syndicated content is really an
    Scott> Atompub issue.  It might be an issue for the content
    Scott> creator.  It might be an issue for the reader.  As long as
    Scott> the pipe between the two passes the content as submitted,
    Scott> though, the pipe has done its job.

Except that we try to build deployable protocols.  If there aren't
content creation tools that can do the right thing then it becomes a
deployment issue for atompub.

A perfectly reasonable response would be that you've thought about and
understood the problem and there are sufficient tools available that
can work with your proposed pipe that you don't need to care about the
issue.

--Sam

---

From: "Scott Hollenbeck" <[email-masked]>
Date: Tue, 10 May 2005 14:26:30 -0400

> can work with your proposed pipe that you don't need to care about the

Paul described text that's in the document to describe what MAY be done.  I
would argue that the existing text is evidence of the thought that has gone
into understanding the issue and the alleged problem.

-Scott-

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 10 May 2005 11:29:15 -0700

>deployment issue for atompub.

True. Fortunately, there have been plenty of text editing tools that 
work with the "no spaces between words" languages for at least 20 
years in the case of Chinese and Japanese Kanji (probably 15 years 
for the other languages).

>can work with your proposed pipe that you don't need to care about the

I'll make that response. :-)

--Paul Hoffman, Director

---

From: Martin Duerst <[email-masked]>
Date: Wed, 11 May 2005 10:11:36 +0900

Hello Sam, others,

 >solution also seems quite complicated.

It's the solution that is being adopted for HTML and CSS, based on a lot
of experience. See Fantasai's post. This also points to a way to solve
this, similar to how this is being addressed by other XML-based specs
starting with XHMTL: Assume that getting space collapsing/removal right
isn't the job of the XML-based format, but the job of the rendering
engine and styling. In that sense, Paul and Scott are right that other
XML formats themselves don't deal with that.

So I think there would be two possible solutions:
- Leave the text as is, with the implication that this includes possible
   removal of all spacing around line ends in some cases. I.e. answering
   Alex's question to the negative, even if on the list only.
- Being more explicit about what is going on. This may include as little
   as saying that the details are up to rendering/styling mechanisms
   such as CSS or XSL-FO, but could go further if deemed necessary.
   In my eyes, this would be desirable, at least to give implementers
   (of viewers, not of "Atom processors") some hints.

A solution that, explicitly or implicitly, says that languages having
word spaces can decently format and indent their XML source, but
languages that don't have word spaces have to use very long lines
in the source for paragraphs, as suggested by some of the commenters
on this list, isn't appropriate at all.

 >because we failed to understand it.

The solutions are not worse than the problem itself. But Paul is
right that it may not be a good idea for Atom to deal with them,
because this is an issue on another layer, which shouldn't be
dealt with separately by each format spec.

Regards,    Martin.