Subject: Wiki setup issues: IRI

From: Martin Duerst <[email-masked]>
Date: Fri, 09 Jul 2004 09:40:27 +0900

When working on http://intertwingly.net/wiki/pie/PaceIRI
(not at all complete yet), I put my name (the real one, with umlaut,
not the one I have to use in my outdated Japanese mailer) in square
brackets at the start of the Abstract like I had seen it on other Paces
such as http://intertwingly.net/wiki/pie/PaceUriOrItsSuccessor.

When I then clicked on it, I got to
http://intertwingly.net/wiki/pie/MartinD_fcrst, which shows
that with a few, probably very simple, changes to the wiki
setup/code, this could be made compatible with IRIs.

Two changes are necessary:
1) Change the wiki to use UTF-8 for its page encoding rather than
    iso-8859-1. For a new wiki, this should just be a setup issue
    (otherwise, choose another wiki software, or maybe another ISP).
    For the current wiki, pages with non-ASCII characters have to be
    converted to UTF-8 at the same time the setup is changed.
    Doing that is easy if you have access to the actual files.
    In line with http://intertwingly.net/stories/2004/04/14/i18n.html,
    I would suggest that we do that anyway, and I'm ready to help.
    This would then change the above URI from
    http://intertwingly.net/wiki/pie/MartinD_fcrst to
    http://intertwingly.net/wiki/pie/MartinD_c3_bcrst.

2) Change the escape character from '_' to '%'. This would change
    the above URI from
    http://intertwingly.net/wiki/pie/MartinD_c3_bcrst to
    http://intertwingly.net/wiki/pie/MartinD%c3%bcrst.
    While at it, please also change escaping from lowercase to
    upper case, in accordance with 2396bis. This would give
    http://intertwingly.net/wiki/pie/MartinD%C3%BCrst.

The two changes are largely independent, and can be done in any order.
If everything goes well, the file names are then actually readable,
on an Unix/Linux system assuming you have selected an UTF-8 locale (most
new Linux systems are shipped that way, as far as I understand).

As I said, I'd be very glad to help get this fixed.
I don't really care too much about my own name, I could
use 'Duerst' as a fallback, but it'd really be better
to fix this on this occasion.

Regards,    Martin.

---

From: Sam Ruby <[email-masked]>
Date: Fri, 09 Jul 2004 13:12:25 -0400

>    http://intertwingly.net/wiki/pie/MartinD_c3_bcrst.

In moin_config.py, there is the following:

   charset = 'iso-8859-1'

While this looks hopeful, it seems to me that such a change would not 
only affect the character set used within the pages, but would also 
change the NAME of the page.  Simply put, MartinD_fcrst and 
MartinD_c3_bcrst are separate pages, and changing the encoding used in 
the page would change which one of these pages were the target of the link.

Examples of pages it would affect:

http://www.intertwingly.net/wiki/pie/Fran_e7oisGranger?action=fullsearch&value=Fran%E7oisGranger&literal=1&case=1&context=40

Thinking about it a bit, my preference is to NOT directly fix the source 
files.  It seems to me that the number of issues should be small - I am 
quite willing to write programs to generate reports of potentially 
problematic pages, but unless the changes required are massive AND 
readily and safely automatable, then I would prefer that the change be 
made by hand.

>    http://intertwingly.net/wiki/pie/MartinD%C3%BCrst.

This also changes the name of the page.  In particular, some of the help 
pages delivered with Moin have URI escaped slash characters in their 
name (example: http://www.intertwingly.net/wiki/pie/HelpOnInstalling).

However, it looks like the code change itself should be simple as the 
mapping is done in exactly one place.

> new Linux systems are shipped that way, as far as I understand).

It looks to me that the way to proceed is to pick some relatively quiet 
time (perhaps this weekend), make both changes at once, and then fix 
what breaks.

> to fix this on this occasion.

- Sam Ruby

---

From: Martin Duerst <[email-masked]>
Date: Sat, 10 Jul 2004 10:08:17 +0900

Hello Sam,

Many thanks for looking into this. There is a slight additional catch
that I realized a few hours after writing my mail. See below.

At 13:12 04/07/09 -0400, Sam Ruby wrote:

>While this looks hopeful, it seems to me that such a change would not only 
>affect the character set used within the pages, but would also change the 
>NAME of the page.  Simply put, MartinD_fcrst and MartinD_c3_bcrst are 
>separate pages, and changing the encoding used in the page would change 
>which one of these pages were the target of the link.

Yes. I didn't mention this because I didn't actually create my page,
but as you point out, we already have such cases.

>http://www.intertwingly.net/wiki/pie/Fran_e7oisGranger?action=fullsearch&va 
>lue=Fran%E7oisGranger&literal=1&case=1&context=40
>
>problematic pages, but unless the changes required are massive AND readily 
>and safely automatable, then I would prefer that the change be made by hand.

Would be fine by me, but I have to admit that I have no clue about
how much non-ASCII content there is in the various pages. But I can
provide scripts for checking.

>pages delivered with Moin have URI escaped slash characters in their name 
>(example: http://www.intertwingly.net/wiki/pie/HelpOnInstalling).

Oh well. Leaving the slash escaped as _2f is probably best.
Prohibiting the creation of such pages, or actually creating
subdirectories and using a real slash, would have been better.

>mapping is done in exactly one place.

Here's the catch. The _hh escape and the %HH escape work on a
slightly different level. Here's a table comparing the two:

escape     File system      URI            href          <a> content
_hh        _hh              _hh            _hh           raw text
%HH        raw text (1)     %HH            %HH (2)       raw text

Column legend:
escape:       what escaping convention is used
File system:  how do the file names appear in the file system
URI:          what gets sent to the server in a GET request
href:         what's in the href attribute of a link
<a> content:  what's in the content of a link

(1) is the catch: While with _hh escaping, the file name in the file system
     is _hh, with %HH escaping, the file name has to use raw bytes. This is
     both desired (because then you can look at the files with a
     finder/explorer or with dir/ls and see the real names) as well as
     necessary (an URI sent with %HH escaping will be converted to raw bytes
     by the server).
     So this means that you have to be slightly more careful with the change;
     just changing the escaping convention is unfortunately not enough.
     I still hope it will be easy enoungh.

(2) Instead of %HH, it is also possible to use raw text. This would then
     be an IRI. While full IRI functionality isn't implemented in all browsers
     yet, I haven't heard about any browser that wouldn't let this work
     as long as the actual page is encoded in UTF-8.

>time (perhaps this weekend), make both changes at once, and then fix what 
>breaks.

That would be great. Please tell me if you need further help.

Regards,    Martin.