Subject: internationalization of URIs

From: Thomas Narten <[email-masked]>
Date: Mon, 15 Oct 2007 15:39:18 -0400

As some of you may know, as part of testing the readiness of IDNs,
ICANN has inserted a set of internationalized versions of ".test" into
the root zone of the DNS. See
http://www.icann.org/announcements/announcement-15oct07.htm for
details.

One of the questions that this has prompted (again) is what about that
pesky "http:", that still needs to typed in ascii. And what about the
rest of the URL for that matter.

I know this is not a new issue, but could someone summarize the
landscape here on what "needs to be done" to internationalize the rest
of the URI (other than the DNS name)? Is this considered to be
completely an application issue? (I note that URIs are ascii, but
there are escaping mechanisms to handle other characters.)

Is there additional IETF work that needs to be done here? Or does this
all fall under application-specific enablement?

Or is even worse, in that this largely falls outside of applications
and more into what is typically done by OS libraries and on the type
of internationalization support the OS provides indirectly?

Thomas

---

From: Martin Duerst <[email-masked]>
Date: Tue, 16 Oct 2007 09:18:41 +0900

Hello Thomas,

I have some good news for you:

For 'the rest of the URI', please have a look at RFC 3987
Internationalized Resource Identifiers (IRIs)
(http://www.ietf.org/rfc/rfc3987).

Work is currently going on to move this to Draft standard, please see:
http://www.ietf.org/internet-drafts/draft-duerst-iri-bis-01.txt

Please review and send your comments to [email-masked].

As for the "http:", that's indeed a piece that some might say is missing.
Even the IRI spec still

My current take on this is as follows:
1) Only very few URI schemes are actually used widely, mostly only "http:"
2) On most browsers, "http:" can be dropped
3) Things like "http:" look like secret incantations or garbage
   to most end users familiar with the Latin script, too
   (that doesn't make it easier to type for people not familiar
    with the Latin script, though)
4) I think an easy way to help would be to provide a drop-down
   menu at the start of the address/location field e.g. in a browser
   or a Web page editor, with some of the following selections:

   http (Web Page Address)
   ftp (File Exchange)
   mailto (Email Address)
   ...
   [of course the explanatory text being in the user interface language]

5) Some browsers might automatically transscribe character sequences
   in non-Latin scripts corresponding to things such as "http://" or
   "ftp://" to Latin automatically when they appear at the start of
   the browser address/location bar. So e.g. somebody in Greece
   would type something like "φτπ://", and it would automatically
   be converted to "ftp://".

   [neither 4) nor 5) are currently implemented as far as I know,
    but they wouldn't be rocket science, and they would be pretty
    local, low-key solutions, and other, similar ideas, may also
    come up]

6) The next step would be to come up with some matching mechanism
   (the easy part :-) and some political framework to create parallel
   names for the schemes (the tough part :-(. I haven't looked at
   the details, but it might be possible to reuse NAPTR RRs or
   some such for the technical part. (that idea is due to James
   Seng, in personal communication at the Kuala Lumpur ICANN
   meeting in 2004).

I'm sorry this is a bit browser-centric, but that's where I see
most users use most URIs/IRIs.

[Private rant: 1) and 2) together mean that non-ASCII TLDs are way
more important than getting scheme names internationalized, despite
of what some shy ICANN people might say ("we can't possibly move ahead
with non-ASCII TLDs unless we have parallel versions of scheme names")
or what some internationalization zealots might say ("everything
or nothing")]

Hope this helps.

Regards,    Martin.

>of internationalization support the OS provides indirectly?

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Ted Hardie <[email-masked]>
Date: Mon, 15 Oct 2007 22:01:02 -0700

>rest of the URL for that matter.

So, I've read Martin's answer, but I'd like to take a shot at this from
a slightly different angle.    Inside the IETF, we commonly treat IRIs
as a presentation layer for URIs.  There is a URI form for any IRI
(and all URIs are also IRIs), so it is always possible to "stick to" the
URI as the protocol element and as use IRIs as presentation elements.
(The big exception to this is inside XML, where the "anyURI" element
got deployed with a syntax that didn't really match URIs at all;
the result is that those strings (which appear to be IRIs to the casual
observer) are really protocol elements using different rules than
those normally used by URIs.)

When I read Martin's comments about drop-downs, elided scheme
names, and similar tricks, my protocol-geek hat tightened on my head
and gave me a pretty severe headache.  Taking it off for a moment,
though, showed me things are still okay.  As presentation elements,
things like drop-downs, inference of scheme by an initial www, and
similar tricks are more reasonable.

A big question, then, is whether we have all the bits
we need to map between a presentation element and a protocol element,
and whether all of those mechanisms need to be standardized.
The answer to the first is almost certainly no.  There are some contexts
where the UI aspects of a decent presentation element are just beyond
the IETF's expertise.  Taking even a simple protocol element like
the scheme portion of an HTTP URI and determining how best to represent
that in, say, modern Mongolian as used by the Oirat is no easy task.   The monk
who developed it didn't have URIs in mind when Clear Script was being
developed.  Should we recommend they use the Latin letters in consequence?  or
the Cyrillic alphabet (as many  other Mongolian speakers do)?  Is either
really the right choice?   Especially, is it the right choice for the IETF to take on?

If it is not clear, I think the answer to the question of whether all presentation
elements need to be standardized is "not in the IETF, anyway".  I think the IETF
does need to make sure that presentation elements can use the UCS in useful
and reasonable ways.  We have worked on that, and there continues to be work on
that, largely through the efforts of dedicated individuals at this point, rather than working groups. 

We also have agreed, as a community, to take on work on some work
that does not rely on a presentation layer separation from the protocol.
We have agreed to work on email addresses, as one example,
and that working group decided not to use a pure presentation layer
approach:

This working group will address one basic approach to email
internationalization. That approach is based on the use of an SMTP
extension to enable both the use of UTF-8 in envelope address local-
parts and optionally in domain-parts and the use of UTF-8 in mail
headers -- both in address contexts and wherever encoded-words are
permitted today. Its initial target will be a set of experimental
RFCs that specify the details of this approach and provide the basis
for generating and testing interoperable implementations. Its work
will include examining whether "downgrading" -- transforming an
internationalized message to one that is compatible with unextended
SMTP clients and servers and unextended MUAs -- is feasible and
appropriate and, if it is, specifying a way to do so. If it is not,
the WG will evaluate whether the effort is worth taking forward.
Other approaches may be considered by the formation of other
working groups.

 (see http://www.ietf.org/html.charters/eai-charter.html for
the full context).   There will be consequences for lots of
other protocol slots if this experiment succeeds, as there
are lots of places for which there is a tacit assumption that
the identifier can "look like" an email identifier (think SIP
AoR s and certs, to take two examples).  But the changes needed
to those slots and the changes needed to the URIs  which refer
to those (or the IRI representation of those URIs) may not be quite
the same. 

I doubt this has helped much, honestly, but hopefully the urge
to correct my mistakes will prompt others to step in and say
something more useful. 
			regards,
				Ted

---

From: Martin Duerst <[email-masked]>
Date: Tue, 23 Oct 2007 16:58:39 +0900

Hello Ted,

Many thanks for your contribution. I'm cc'ing it (with my
comments interspersed) to the [email-masked] mailing
list.

>those normally used by URIs.)

Some additonal comments:
- Atom, as an IETF standard, is an example where the IETF uses
  IRIs as protocol elements. Atom is also XML-based, so this doesn't
  contradict the above, but I just wanted to mention this for people
  who might think, from the above, the the IETF doesn't use IRIs.
- RDF, whether written in XML or not, also uses IRIs.
  Because of the way RDF compares resource identifiers, converting
  from IRIs to URIs isn't a good idea in the case of RDF.
- There is in principle nothing that would prevent the IETF from
  using IRIs as protocol elements in a new protocol. In my opinion,
  this would actually be the right thing. The conversion to URIs
  as protocol elements is there first and foremost for existing
  protocols that are based on URIs.

>similar tricks are more reasonable.

Detail: the scheme isn't inferenced by an initial www. A very quick
test on one single browser showed that a leading 'ftp' label inferences
ftp://, but there is no need for an initial www to infer http://.

In general, yes, all these tricks are very much presentation issues.
Overall, we can think about this in three layers:

Final presentation: May include tricks as above,...

IRI: sometimes protocol, sometimes presentation

URI: protocol

>The answer to the first is almost certainly no.

I fully agree.

>There are some contexts

Yes. I expect this way of designing protocols to become more frequent.
For new protocols, for me, it would be a non-brainer. For existing
protocols, the decision is of course much more difficult, and
will once go one way, and once the other way.

>to correct my mistakes will prompt others to step in and say

I think this was very useful background, thanks a lot.

Regards,    Martin.

---

From: Ted Hardie <[email-masked]>
Date: Wed, 24 Oct 2007 10:55:43 -0700

Hi Martin,
	Some comments below.  I've kept your distribution,
but I'd suggest we pick a place to continue discussion; suggest
public-iri for followups.

>  protocols that are based on URIs.

I agree that there is nothing to prevent the IETF from using IRIs
as protocol elements.  For the large majority of IETF protocols,
though, this isn't the optimal choice.  For most protocol slots,
the aim isn't to present the element to a human, but to enable
protocol processing to proceed in a deterministic way.    This
means that most protocol slots don't benefit in any meaningful
way from the greatly increased  set of unreserved characters.
IRIs do introduce increased complexity in protocol processing
(I believe your "Ladder of comparison" section describes some
of the most critical ways), and when this is not needed because
of human interaction it is simpler to avoid it. 

Any protocol that wishes to use IRIs is also strongly advised
to take into account what restrictions of the UCS it wishes to
make; as you point out in IRI-bis, section 6.1, a scheme-specific
definition or processor may need to make a number of rules that exclude
look-alikes in both characters and delims.  Those will add increased
complexity as well, especially if they vary from scheme to scheme.

>ftp://, but there is no need for an initial www to infer http://.

I actually didn't mean in browser behavior; sorry for being unclear.  I mean
that humans see "www" now and assume that they should plug it into a web browser;
"infer a scheme" was probably not the ideal way of describing that, since most
users don't think of the scheme in that process (though many browsers do
prepend the scheme in final display).

>will once go one way, and once the other way.

For new protocol slots that will be exposed to humans, I agree that
this will be more frequent.  The tricky bit is always figuring out
what will be.

Thanks for your comments,
			regards,
				Ted