Subject: PaceIRI status

From: Tim Bray <[email-masked]>
Date: Mon, 24 Jan 2005 16:17:41 -0800

If there were no further discussion: It's hard to see how to avoid 
adopting this now that IRIs are standards-track RFC.  -Tim

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Jan 2005 09:16:59 +0100

> adopting this now that IRIs are standards-track RFC.  -Tim

I saw some concerns (with which I agree) that requiring the presence of 
an IDN library is problematic. As far as I can tell, it's likely to be 
ignored by developers of clients that run on somwehat constrained devices.

Also; I sypmathize with supporting IRI, but that shouldn't mean it needs 
to replace any usage of URIs (for instance, I also think that the 
introduction into XMLNS 1.1 was a mistake).

So: -0.

Best regards, Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Martin Duerst <[email-masked]>
Date: Tue, 25 Jan 2005 17:42:44 +0900

>Also; I sypmathize with supporting IRI, but that shouldn't mean it needs 
to replace any usage of URIs

Every URI is an IRI by definition. So all URIs that are in use can be
used with Atom without any problems even if the spec says we use IRIs.
Replacement is definitely not the right term.

 >(for instance, I also think that the introduction into XMLNS 1.1 was a mistake).

Well, I might agree that it was a mistake. Observable behavior for
most implementations (in particular XSLT implementations, where you
most easily can test actual namespace behavior) allows IRIs in
namespaces anyway, so this could just have been a clarification
to the XMLNS 1.0 spec, rather than bumping up the number.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Jan 2005 14:29:04 +0100

Martin Duerst wrote:
>  >Also; I sypmathize with supporting IRI, but that shouldn't mean it 
> needs to replace any usage of URIs
> 

Let's disagree here. If the spec currently says "URI" and is supposed to 
say "IRI", we *are* replacing URI by IRI (even if every URI is an IRI).

> to the XMLNS 1.0 spec, rather than bumping up the number.

The big difference here is that XMLNS uses IRIs/URIs as identifiers and 
only for that. However, what is an XSLT that transforms Atom content to 
HTML supposed to do when it encounters a IRI which isn't a legal URI? 
For instance, it can't put it into an HTML href attribute without 
producing invalid HTML.

Best regards, Julian

---

From: Anne van Kesteren <[email-masked]>
Date: Tue, 25 Jan 2005 14:39:07 +0100

> For instance, it can't put it into an HTML href attribute without 

I guess that is a HTML errata issue rather then an Atom issue. Atom can 
not make sure it is compatible with every specification out there.

Perhaps we could add an informative about it although I doubt if it 
would be useful. Furthermore, browsers do not really support IRIs either 
so authors are not likely to use them in HTML context anyway.

-- 
  Anne van Kesteren
  <http://annevankesteren.nl/>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 25 Jan 2005 14:45:16 +0100

>HTML supposed to do when it encounters a IRI which isn't a legal URI? 

http://www.w3.org/TR/xslt#section-HTML-Output-Method

  The html output method should escape non-ASCII characters in URI
  attribute values using the method recommended in Section B.2.1 of
  the HTML 4.0 Recommendation.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Jan 2005 15:11:12 +0100

>   the HTML 4.0 Recommendation.

How does that help with IDNs?

Best regards, Julian

---

From: Bill de hÓra <[email-masked]>
Date: Tue, 25 Jan 2005 14:37:05 +0000

> Replacement is definitely not the right term.

Martin

Supersetting would be the right term perhaps; that has its own issues.

I'm 0 on IRIs.

cheers
Bill

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 25 Jan 2005 15:47:41 +0100

>How does that help with IDNs?

After applying the algorithm, how is the situation different from
the current draft-ietf-atompub-format-04.txt draft which refers to
RFC2396bis?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Martin Duerst <[email-masked]>
Date: Tue, 25 Jan 2005 23:47:02 +0900

only for that. However, what is an XSLT that transforms Atom content to 
HTML supposed to do when it encounters a IRI which isn't a legal URI? For 
instance, it can't put it into an HTML href attribute without producing 
invalid HTML.

First, the Atom spec just says that HTML or XHTML goes into certain
elements, it leaves it to other specs to say what HTML or XHTML is.
So Atom doesn't deal with the question of whether IRIs are allowed
there or not, and I hope PaceIRI is worded correctly in that respect.

Second, HTML href attributes are defined as CDATA, so in terms of
validity, any Unicode character goes anyway.

Third, the HTML 4 recommendation, dating back to 1997
(http://www.w3.org/TR/REC-html40-971218/appendix/notes.html#h-B.2)
contains language that in today's terms amounts to saying
"browsers should treat URIs with non-ASCII characters as IRIs,
even though strictly speaking, they're illegal". Many browsers
to some extent already do, and when the IRI RFC gets out, I guess
some more will.

Forth, when IRIs work, people, especially in regions that don't
use the Latin script, will just expect them to work, and tools
will just have to support them, so things will just develop
naturally.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Jan 2005 15:53:43 +0100

> the current draft-ietf-atompub-format-04.txt draft which refers to

If an Atom document contains a URI, it by definition does not contain 
non-ASCII characters, such as no conversion is needed. Basically, the 
XSLT would consume a URI and be able to copy it verbatim to the output.

Am I missing something?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 25 Jan 2005 15:59:37 +0100

Martin Duerst wrote:
>  >The big difference here is that XMLNS uses IRIs/URIs as identifiers 
> and only for that. However, what is an XSLT that transforms Atom content 
> to HTML supposed to do when it encounters a IRI which isn't a legal URI? 

<http://www.w3.org/TR/html401/struct/links.html#adef-href>

As far as I can tell, HTML 4.01 normatively refers to RFC2396, which 
doesn't allow non-ASCII characters. The *DTD* may allow any kind of 
string here, but the spec text doesn't.

> to some extent already do, and when the IRI RFC gets out, I guess

So are there plans to add an erratum to HTML 4.01?

And what's the situation for XHTML?

 > ...

Best regards, Julian

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Jan 2005 00:13:08 +0900

>>  >The big difference here is that XMLNS uses IRIs/URIs as identifiers 
and only for that. However, what is an XSLT that transforms Atom content to 
HTML supposed to do when it encounters a IRI which isn't a legal URI? For 
instance, it can't put it into an HTML href attribute without producing 
invalid HTML.
 >As far as I can tell, HTML 4.01 normatively refers to RFC2396, which 
doesn't allow non-ASCII characters. The *DTD* may allow any kind of string 
here, but the spec text doesn't.

I know. You were using 'invalid', not 'illegal'.
The former is a technical term that is very strongly related to DTDs.

 >So are there plans to add an erratum to HTML 4.01?

If there was a working errata process for HTML 4.01, I guess
such an erratum would already have happened.

 >And what's the situation for XHTML?

For XHTML 1.0/1.1, that's based on HTML 4..., so see above.
For XHTML 2.0, that will just refer to the IRI RFC, I guess.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Jan 2005 00:03:56 +0900

>How does that help with IDNs?

There are two ways to look at this:

a) It doesn't, because B.2.1 was written before anybody knew much about IDNs.

b) It does, because rfc2396bis actually allows percent-encoding in
    the host name part, and defines what should happen with it.

In practice, it will work on most browsers and other software that
support IDNs, and it will not work on browsers,... that don't support IDNs.

However, there are some browsers (Mozilla and Safari) that support
IDNs but at least in some versions can't handle it if they are
percent-escaped.

Regards,    Martin.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 25 Jan 2005 08:52:35 -0800

At 9:16 AM +0100 1/25/05, Julian Reschke wrote:
>I saw some concerns (with which I agree) that requiring the presence 
>of an IDN library is problematic. As far as I can tell, it's likely 
>to be ignored by developers of clients that run on somwehat 
>constrained devices.

I would like to hear more from developers whether or not they think 
this is a problem. (I'm asking this wearing my 
co-chair-looking-for-consensus hat, not my 
author-of-IDN-and-stringprep hat.)

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Jan 2005 08:23:00 +0900

>If an Atom document contains a URI, it by definition does not contain 
non-ASCII characters, such as no conversion is needed. Basically, the XSLT 
would consume a URI and be able to copy it verbatim to the output.
 >Am I missing something?

Bjoern was making a vaild, but fine, point: Because we decided to
refer to RFC2396bis, rather than to RFC2396, we already have bought
into the fact that RFC2396bis allows percent-encoded domain names,
and thus essentially requires IDN support. (apart from the basic
fact that no resolver is required to resolve all URIs or IRIs)

If you want to add another fine point to this, you could say that
while RFC2396bis allows percent-encoded domain names, the scheme
definitions, e.g. for http:,..., haven't yet been updated to allow
that.

I personally think that fine points are interesting, but that
implementers and users won't bother too much with them if it's
perfectly clear how things are supposed to work.

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Jan 2005 09:45:42 +0900

Just a correction to what Julian wrote: Stringprep may be ignored
or partially ignored, because it requires lots of data, but I don't
think the Unicode->punycode conversion will be ignored.

 >I would like to hear more from developers whether or not they think this 
is a problem. (I'm asking this wearing my co-chair-looking-for-consensus 
hat, not my author-of-IDN-and-stringprep hat.)

Although I don't have any experience with developing blogging tools,
I have some experience in integrating IDN support into a browser, and
I can report on some other browsers. I hope this breaks the ice for
others to chip in.

1) I have integrated IDN support into libwww, and used it in Amaya
    (that version isn't yet public, but it should be soon).
    Integrating idnkit into libwww was easy (except for the autoconf/
    automake,... files, which I'm leaving to somebody else).
    idnkit of course includes full stringprep.

2) The maker of a widely used browser for Mobile phones has just
    recently announced that they support for internationalized
    domain names at http://www.access.co.jp/english/press/050120.html
    My guess is that they take some shortcuts on stringprep, but
    that's really just a guess. If you really need to know, I may
    be able to ask somebody from that company.

3) Browsers start to integrate blog support. These browsers are at
    the leading edge of development, and usually support IDNs already.
    The examples I know are Opera and Safari. IDNs will just work
    on those browsers, and people will come to expect them to work
    in other contexts.
    I think at least at some stage, Opera didn't implement strinprep
    (or not completely), but I haven't tested lately.

Regards,     Martin.

---

From: David Czarnecki <[email-masked]>
Date: Tue, 25 Jan 2005 21:48:00 -0500

> I can report on some other browsers. I hope this breaks the ice for

From an SDK perspective, there are a number of libraries available to handle
IDN support. As a Java person developing blogging tools, I've looked at the
following. Hope this helps.

Name: IDN SDK
Vendor: Verisign
URL: 
http://www.verisign.com/products-services/naming-and-directory-services/nami
ng-services/internationalized-domain-names/idn-registrars/page_001408.html
License: Apache/BSD

Name: Java-IDNA (now part of GNU IDN Library)
Vendor: GNU
URL: http://www.gnu.org/software/libidn/
License: LGPL

International Components for Unicode for Java (there are also C/C++ bindings
which also have IDN support)
Vendor: IBM
URL: http://oss.software.ibm.com/icu4j/
License: IBM Open Source License (
http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html)

-David

---

From: Martin Duerst <[email-masked]>
Date: Wed, 26 Jan 2005 11:38:57 +0900

Some good news:

The IRI spec is now published as RFC 3987 (Proposed Standard,
http://www.ietf.org/rfc/rfc3987.txt).

The update of the URI spec, known as RFC2396bis, is now published
as STD 66, RFC 3986.

Even less reason for not adopting them. Editors, please update
your references. I'll update PaceIRI in a day or two.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 26 Jan 2005 08:54:51 +0100

> fact that no resolver is required to resolve all URIs or IRIs)

OK, thanks for making me aware of this subtle change.

Regards, Julian

---

From: Danny Ayers <[email-masked]>
Date: Wed, 26 Jan 2005 20:39:31 +0100

+1 on PaceIRI

I'm a little hesitant on this because I'm not familiar with the
issues, but it's something we'll probably all have to broach sometime
soon. Martin seems to know what he's talking about ;-)

-- 

http://dannyayers.com

---

From: Robert Sayre <[email-masked]>
Date: Fri, 28 Jan 2005 19:15:47 -0500

> your references. I'll update PaceIRI in a day or two.

IRIs are a step forward and important to include in the spec, but they 
also worry me. In RFC3987, I read the following:

"The approach of defining a new protocol element was chosen instead of
 extending or changing the definition of URIs. This was done in order
 to allow a clear distinction and to avoid incompatibilities with
 existing software."

Do you expect Atom implementors will be using incompatible existing 
software? I think this question should face roughly the same scrutiny 
that PUT/DELETE did.

I'm also worried that the term "IRI" will cause confusion. After all, 
the catch phrase is not "Cool IRIs Don't Change." What can we do 
minimize confusion?

Robert Sayre

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 29 Jan 2005 16:00:59 -0800

Not to advocate one position or another, but RFC 3987 doesn't obsolete 
RFC 3986; we have a choice.

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 29 Jan 2005 16:25:42 -0800

As I understand it, if I implement an Atom processor that doesn't 
support IRIs, It's not conformant, and will have problems with some 
valid feeds, when dereferencing IRIs.

So, if we require support, it's potentially setting a higher bar for 
implementations. It wouldn't be a big deal to require IRIs if support 
were available in most languages that people are likely to use Atom in; 
e.g., Perl, Python, C#, Java, ECMAScript, VB, Ruby, etc.; preferably in 
their standard libraries, but at least in a single download.

Martin, do you know when can we expect to see such things appear? I've 
seen a few suggestions for Java go by, but not much else.

Also, has anyone identified which uses of URIs in Atom are likely to 
pose problems when converted to IRIs? Some URIs in Atom are used as 
identifiers, and not dereferenced. In those cases, my (possibly naïve) 
inclination would be that as long as they're compared 
character-by-character, making them IRIs wouldn't be an issue, and we 
could use IRIs for at least some things in Atom (e.g., id, but not 
link) no matter what the status of library implementations.

Cheers,

---

From: Martin Duerst <[email-masked]>
Date: Sun, 30 Jan 2005 10:27:30 +0900

Hello Robert,

Thanks for your questions, and for studying IRIs so carefully.

 > to allow a clear distinction and to avoid incompatibilities with

Yes. That was put in to clearly say that you can't just expect IRIs
to work wherever URIs work without doing anything. Because Atom is
a new protocol, we are in a very good position to do the right thing,
which turns out to be very little.

 >Do you expect Atom implementors will be using incompatible existing 
software? I think this question should face roughly the same scrutiny that 
PUT/DELETE did.

Yes. If you look at the pace (http://www.intertwingly.net/wiki/pie/PaceIRI,
just updated to take into account the RFC publications, most of this
is described under "Impact". Converting an IRI to UTF-8 and doing
%-encoding is a lot easier to program if there is no IRI library
available than to program your own PUT or DELETE requests, in
particular because the former is a completely internal operation,
whereas the later is a network operation. Integrating conversion
to punycode in case there is no IDN support available is a bit
more work, but if you don't do normalization (stringprep,...),
the code footprint should be extremely small. Except for encoding
issues (UTF-8 or UTF-16 vs. UTF-32), you can basically just
copy the code from http://www.rfc-editor.org/rfc/rfc3492.txt.

 >I'm also worried that the term "IRI" will cause confusion. After all, the 
catch phrase is not "Cool IRIs Don't Change." What can we do minimize confusion?

First, every URI is an IRI, so there is absolutely no need for
any existing URI to change. Second, there is the confusion with
terms. I agree that the term IRI shouldn't be pushed too much,
I expect users e.g. in Japan to just use language like
"software x-y-z now accepts URLs/Web addresses with Japanese characters".

But in the spec, we have to make clear that IRIs are allowed, in a way
that implementers actually see it. So that's why I propose to replace
the term URI with the term IRI throughout the spec, but not to change
the attribute names. But I'm open to discussion about this; if you have
some other ideas, please send them to the list.

Regards,    Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 31 Jan 2005 15:49:44 -0800

This is a pretty persuasive argument, IMO. There is always a natural 
profile of what is supported by most implementations; that shouldn't 
necessarily constrain the specs.

Thanks,

On Jan 31, 2005, at 3:46 PM, Martin Duerst wrote:

> Yes and no. Does the Atom spec require that an Atom processor support
> http? Maybe it does. Does it require that any other uri scheme/protocol
> is supported? I think it doesn't. Does it require that all http uris
> always resolve? No, it just can't, because of network reality.
> So I guess there is some slack for some implementations, although
> of course we wouldn't like them to use that slack.

---

From: Martin Duerst <[email-masked]>
Date: Tue, 01 Feb 2005 08:46:53 +0900

Hello Mark,

I have just responded in quite some detail to Robert, which
should cover quite some of your questions.

At 09:25 05/01/30, Mark Nottingham wrote:
 >
 >As I understand it, if I implement an Atom processor that doesn't support 
IRIs, It's not conformant, and will have problems with some valid feeds, 
when dereferencing IRIs.

I also think that testing is quite important. If somebody could
point to some basic Atom tests, I'll try to set up some equivalent
tests that use IRIs. For browsers, even a single test that I put
up some years ago lead to a lot of improvement.

 >So, if we require support, it's potentially setting a higher bar for 
implementations. It wouldn't be a big deal to require IRIs if support were 
available in most languages that people are likely to use Atom in; e.g., 
Perl, Python, C#, Java, ECMAScript, VB, Ruby, etc.; preferably in their 
standard libraries, but at least in a single download.

I have up to now concentrated on pushing browser support, so I haven't
spent that much time looking at library/language support, but I think
it's very important, and any information (such as just even a list of
languages like above) is highly appreciated. Also, if you can point
to any specific libraries, we can look at these libraries and look
at how to integrate IRIs. As I have written in my mail to Robert,
in terms of actual code, it's not a really a big deal. Also, as
far as I understand, most of the above languages allow integration
of C code, so it's mostly a question of integration.

In addition to that, the Unicode support in these languages is
varying, and in some cases developping. For example for Perl,
there is quite some differences e.g. between 5.0x, 5.6x, and 5.8x.
Of course, to correctly implement Atom, appropriate Unicode support
is needed. In many ways, Unicode support is 90% of IRI support.

Also, I'm quite sure that using IRIs in Atom will be a great
motivation for various library maintainers.

I have already contacted the perl Unicode list yesterday (after
some naive attempts of mine to use some Perl modules failed)
and already got some initial responses. I'll update this list
once I know more.

 >Also, has anyone identified which uses of URIs in Atom are likely to pose 
problems when converted to IRIs? Some URIs in Atom are used as identifiers, 
and not dereferenced. In those cases, my (possibly naive) inclination would 
be that as long as they're compared character-by-character, making them 
IRIs wouldn't be an issue, and we could use IRIs for at least some things 
in Atom (e.g., id, but not link) no matter what the status of library 
implementations.

This is not a naive assumption. In fact, although XML Namespaces 1.0
says that namespaces identifiers are URIs (namespaces 1.1 extends this
to IRIs), the vast majority of XSLT implementations work without any
problems on input that uses IRIs in namespace URIs.

But the fact that we get IRI support for IDs for free, whereas IRI
dereferencing will require some work, should not mean we only do
the former and not the later. Although IRI dereferencing requires
some work, that work isn't very much work, even if not relying on
libraries. Also, I wouldn't expect users to understand why they
could use IRIs in some places and not in others.

Regards,   Martin.