Subject: Re: [idn] IDN spoofing

From: "Doug Ewell" <[email-masked]>
Date: Mon, 21 Feb 2005 11:14:39 -0800

Hans Aberg <haberg at math dot su dot se> wrote:

>> That isn't the point.  It doesn't matter if the mapping takes place
>> at the character-encoding level or at some other level.  The problem
>> of determining which pairs of characters are confusable and should be
>> folded together, and which pairs aren't and shouldn't, still remains
>> to be solved.
>
> If one does the way, no characters or character sequences are actually
> folded together, but one defines a semantics which tells which
> character sequences should have the same semantics.

Well, whatever.  You know what I mean: "which pairs of characters should
be treated the same for IDN purposes."

>> And for something like IDNs, once you have decided on a mapping, you
>> can never, ever change it.  Otherwise you will have a domain name
>> available for registration by customer A today, but a similar one not
>> available to customer B six months later (or vice versa, A can't get
>> it but B can).  Either way, you have a lawsuit.
>
> Sure you can change it: One can make the equivalence classes smaller,
> whenever one wants. For example, upper and lower case letters are now
> considered equivalent. One takes that equivalence away, names must be
> used as registered with respect to casing. If one introduces new
> equivalence classes, then one must run through all registered names,
> and require that names that are merged onto each other are changed.
> But if one is sufficiently conservative when defining these
> equivalences, that should not happen too often. For example, if now
> Latin "o" and Greek "omicron" are viewed inequivalent but are declared
> equivalent, then, if more than one user has used the name "foo" in
> various combinations of the letter, then all but one must change
> registration name.

You make it sound so easy.  Try explaining to a company that has
advertised its Web address all over television, radio, newspapers,
magazines, billboards, and business cards that they now have to change
their address because someone changed the IDN rules.

-Doug Ewell
 Fullerton, California
 http://users.adelphia.net/~dewell/

---

From: Erik van der Poel <[email-masked]>
Date: Mon, 21 Feb 2005 12:03:40 -0800

Hans Aberg wrote:
> whenever one wants.

As a mathematician, one might be inclined to think that way. But here, 
we're not talking about theoretical mathematics. We're talking about 
network engineering. A totally different way of thinking.

You can't just change the mapping whenever you want because there are 
many (client and server) installations out there that can't be changed 
overnight (what is known in network parlance as a "flag day").

For example, even if a registry were to change their mapping, go through 
their entire database, and delete the names that are determined to be 
duplicates (however one might accomplish that), there will be people 
with the old version of the app, which uses the old mapping, and will 
not be able to find the name (since it has been deleted).

Now, this might be a good thing if the name is an evil spoof, but what 
about innocent registrations? What if two separate parties have an 
equally legitimate claim on a particular name? This happens a lot in the 
ASCII DNS, and basically, whoever got there first (or is willing to pay 
a lot of money) wins.

One way to continue to support these innocent duplicates is to use a 
different prefix (i.e. something other than xn--) in the new mapping, 
and keep the old names (with the old prefix) in the database (instead of 
deleting them). This way, the old clients continue to find the old 
innocent names.

But what about the new clients? Now they will suddenly end up on a 
different Web site when the user clicks on a link. I suppose the user 
will just have to update their client, or the domain name owner will 
have to register a different name and update all the Web pages to point 
to the different name (assuming that they even have control over *all* 
of the Web pages that might contain a link to their site).

And so on. Do you get it now? You can't just change the mapping 
"whenever" you want. If you do this at all, you do it as few times as 
possible.

Now, you may point out that we are just getting started with IDN and 
that not very many names have been registered (and I may even agree with 
you), but it would still take a while to come up with a better mapping 
(and reach consensus on it -- shudder), and in the meantime, more names 
would be registered.

And this still would not negate my main point, which is that you can't 
do this "whenever" you want.

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Mon, 21 Feb 2005 12:25:29 -0800

Erik van der Poel wrote:
> will just have to update their client

Sorry, I don't know how that last part got in there. Brain fart?

Erik

---

From: Erik van der Poel <[email-masked]>
Date: Mon, 21 Feb 2005 21:18:07 -0800

OK, I think this latest flurry of emails is starting to form a picture 
of What We Must Do (re: nameprep, IDN spoofing and the registries):

Basically, the work to install filters at the registries, and the work 
to write the next version of the nameprep spec can proceed in parallel, 
pretty much independently.

As George points out, the registries are going to have to start 
filtering IDN lookalikes, otherwise they will eventually face lawsuits 
from the "big boys" (as George so delightfully puts it). The ccTLDs will 
have a relatively easy task, while the gTLDs like .com will have the 
difficult task of deciding which subset of Unicode to allow. They will 
also have to go through their database, looking for lookalikes, and 
deleting them or transferring them to new owners, probably paying their 
previous owners back. The registrars might have to be involved in the 
money transaction too. What a mess. I don't envy the gTLDs. Maybe the 
Unicode Consortium could help them out by providing homograph tables.

One possible approach for the gTLDs is to halt IDN registration now. 
Then they can work on their filters, starting with a small subset of 
Unicode. After reopening IDN registration, they can grow the subset if 
there is enough demand for characters outside the initial subset.

If the gTLDs are going to do some serious subsetting, then they will 
probably also need to provide software to the registrars that will map 
users' characters into the subset. E.g. converting a user's local 
charset to the subset of Unicode. Then again, this might be an area 
where registrars could compete with each other, to provide the most 
friendly software to the end-user (registrant).

On the other side, we have the nameprep spec, and the work required to 
rev it. As John Klensin points out in another email, nameprep will 
eventually have to be updated to include new Unicode characters. 
Nameprep specifies Unicode 3.2, but Unicode itself is already at 4.0.1, 
and may be even further along by the time we finish discussing and 
drafting nameprep bis (new version). Call it nameprep2.

Now, one item that is clearly on nameprep2's table is the new version of 
Unicode. Another item that could be considered is the banning of slash 
'/' homographs and others. This type of spoofing was recently discussed 
on the IDN list. Certain Unicode blocks, like the math characters, might 
also be banned instead of mapped as they are now. And I'm sure we would 
discuss mapping or banning the homographs, such as Cyrillic small 'a'. A 
lot of this is likely to be controversial, and some people might suggest 
that we leave the subsetting to the registries, since they have to do it 
anyway. So, instead of shrinking the character set, nameprep2 might just 
grow it (for the new version of Unicode). I don't know. We'll see.

I'm not sure whether we would need a new ACE prefix if we are only 
adding characters (and not removing any). I'm too tired right now to 
think about it.

Erik

George W Gerrity wrote:
> The two references below summarise much that has been said about the 
> difficulty of dealing with the internationalisation of Domain Names. Let 
> us agree once and for all:
> 
> 1. The completely general problem is mathematically */and/* 
> computationally intractable, even if we use fuzzy mapping;
> 
> 2. The problem is a typical engineering challenge to find a workable 
> solution — future-proofed as much as possible — which is minimally complex;
> 
> 3. If the engineers (us?) don't solve it, the lawyers will have a 
> heyday, the courts will find expensive solutions, the cost of running 
> the web will blow out, and all of us will have mud all over our faces.
> 
> 4. Now is the time — when there are only a very few registered names 
> with possible clashes — to do it before we */have/* to go through the 
> painful process of unregistering names and upgrading TLD machine codes.
> 
> So let's sketch out an approach, using <.com.ru> as an example.
> 
> a) The <.com.ru> registrar only accepts latin characters for that domain 
> name, or only accepts Cyrillic characters, */no mix/*, and maps the two 
> as equivalent. Case-equivalence mapping */may/* also be allowed, at a 
> cost of more complexity. Let the registrar decide that, and let's be 
> sure that as far as possible, the issuing authority licencing the TLD to 
> the registrar ensures legal protection for these */arbitrary/*, but 
> fixed decisions.
> 
> b) the first filter selects name tags whose codes (including diacritics, 
> etc) are either not all in the Cyrillic block or the Latin block(s) for 
> special attention.
> 
> My guess is that at this point, only a few percent will require special 
> attention.
> 
> c) At this point, the <.com.ru> registrar will need to exercise some 
> common sense. For instance, it seems unreasonable that this domain 
> should accept codes outside the Latin and Cyrillic code blocks, and if 
> they do, then mixes should be strongly discouraged. Certainly, the use 
> of, say, Hebrew vowel pointing with Latin Codes, while perhaps 
> acceptable in Israel TLD, should be unacceptable in the Russia TLD. In 
> fact, as a general rule, mixes of diacritics from one code block with 
> code points from another, should never be allowed.
> 
> Further rules can limit legal sequences of the allowed mixes. For 
> instance, in alphabetic scripts such as Latin and Cyrillic, isolated 
> code points from one script found in another make no sense unless 
> spoofing is intended. Earlier, I suggested that a code-point string of a 
> single script found mixed with strings of other scripts, should be of 
> minimum length 2. One can also limit the number of separate substrings 
> of an alternate script found interspersed with a dominant (national?) 
> script.
> 
> These sort of common-sense rules can be easily implemented and the 
> computational overhead is minimal. Of course, owners of ridiculous trade 
> marks (such as <U+004B U+0049 U+039B>, “KIΛ”, for the brand name of the 
> automobile “KIA”) will disagree, but realism has to intrude somewhere 
> into the free market economy.
> 
> The problems for universal TLDs (<.com>, <.net>) are far more complex, 
> because they are required to accept all language scripts. At the TLD 
> itself, one can allow a limited, but finite number of character strings 
> to be equivalent, including the rule that script mixtures are 
> inadmissable, but maybe case folding will be allowed.
> 
> Once again, however, application of some judicious sieve filters and 
> rules about how mixed scripts may be composed, can simplify the handling 
> of the name tags. There are also sieve rules that can immediately throw 
> out most inadmissable combinations, such as the string length rule 
> mentioned above. Those strings remaining can be tossed to a human, who 
> will be required to be an expert in orthography (nice new line of 
> business for many on the Unicode list?).
> 
> Now, it doesn't make sense for these rules to be part of a standard on 
> how to extend Domain names to use scripts other than Latin: they are 
> much better handled as (algorithmic where possible) regulations 
> specified by the authority for a given TLD, or set of TLDs, in the case 
> of the universal TLDs.
> 
> By using this approach, and starting off with a set of rules that 
> disallow most forms of script mixes, then where appeals to common sense 
> and the wishes of a reasonable number of potential clients suggest a 
> loosening of the rules, this can be done with little disruption to the 
> existing state of affairs.
> 
> George
> ------
> 
> On 22 Feb 2005, at 08:40, Doug Ewell wrote:
> 
>         The suggestion I made, was to use a function to detect
>         confusables by
>         declaring them equivalent, but retaining the full Unicode character
>         set for representing the IDN's. If this is used at the registration
>         level only, the only thing that happens when somebody enters a
>         confusable, is that it is rejected. There is a problem only when an
>         authority admits parallel, confusable names to be registered.
> 
> 
>     Granted. The problem, as I have said so often, is determining what the
>     set of "confusables" is. Don't just say a/а and o/ο, either; that's
>     only the tip of the iceberg.
> 
> 
> On 22 Feb 2005, at 07:03, Erik van der Poel wrote:
> 
>     Hans Aberg wrote:
> 
>     As a mathematician, one might be inclined to think that way. But
>     here, we're not talking about theoretical mathematics. We're talking
>     about network engineering. A totally different way of thinking.
> 
>     You can't just change the mapping whenever you want because there
>     are many (client and server) installations out there that can't be
>     changed overnight (what is known in network parlance as a "flag day").
> 
>     For example, even if a registry were to change their mapping, go
>     through their entire database, and delete the names that are
>     determined to be duplicates (however one might accomplish that),
>     there will be people with the old version of the app, which uses the
>     old mapping, and will not be able to find the name (since it has
>     been deleted).
> 
>     Now, this might be a good thing if the name is an evil spoof, but
>     what about innocent registrations? What if two separate parties have
>     an equally legitimate claim on a particular name? This happens a lot
>     in the ASCII DNS, and basically, whoever got there first (or is
>     willing to pay a lot of money) wins.
> 
>     different prefix (i.e. something other than xn--) in the new
>     mapping, and keep the old names (with the old prefix) in the
>     database (instead of deleting them). This way, the old clients
>     continue to find the old innocent names.
> 
>     different Web site when the user clicks on a link. I suppose the
>     user will just have to update their client, or the domain name owner
>     will have to register a different name and update all the Web pages
>     to point to the different name (assuming that they even have control
>     over *all* of the Web pages that might contain a link to their site).
> 
>     "whenever" you want. If you do this at all, you do it as few times
>     as possible.
> 
>     that not very many names have been registered (and I may even agree
>     with you), but it would still take a while to come up with a better
>     mapping (and reach consensus on it -- shudder), and in the meantime,
>     more names would be registered.
> 
>     And this still would not negate my main point, which is that you
>     can't do this "whenever" you want.
> 
>     Erik

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 22 Feb 2005 06:47:49 +0000

Erik van der Poel <[email-masked]> wrote:

> adding characters (and not removing any).

Nameprep was designed so that if Unicode evolves in a backward
compatible way (new characters added, old normalization mappings
unchanged), then Nameprep can evolve along with it in a backward
compatible way, with no need to change the ACE prefix.

(Actually Unicode normalization did change in an incompatible way
between 3.2 and 4.0 when five erroneous compatibility mappings were
fixed, but the NormalizationCorrections.txt file allows applications to
choose whether to keep the errors for backward compatibility.  A new
Nameprep would have to specify which choice to make.)

AMC

---

From: William Tan <[email-masked]>
Date: Wed, 23 Feb 2005 00:06:33 +1100

Hi,

> from the "big boys" (as George so delightfully puts it). 

Actually, I don't see that happening. It is generally accepted that 
neither gTLD registries or registrars check the names with human eyes 
before registering them. You can't sue if they've not been negligent. 
While I hope that gTLDs such as Verisign start filtering, I'm afraid we 
can't wait that long. That is why an application level solution is 
necessary.

> The ccTLDs will have a relatively easy task, while the gTLDs like .com 
> will have the difficult task of deciding which subset of Unicode to 
> allow. 

Actually, most ccTLDs who are currently not offering a bundling solution 
specifically decided not to do bundling. They were fully aware that by 
not bundling, they are allowing the possibility of spoofing with the 
introduction of IDNs. And they have valid reasons for not doing so. This 
homograph attack advisory may well trigger some of them to change 
policies (though highly unlikely IMO), and hopefully get registries that 
are considering implementing IDN to employ some form of bundling 
solution, if applicable.

> They will also have to go through their database, looking for 
> lookalikes, and deleting them or transferring them to new owners, 
> probably paying their previous owners back.

If you're running a registry, you'd understand that deregistering or 
transferring a name against the registrant's will is no where as easy as 
it sounds. You might see lawsuits here.

> there is enough demand for characters outside the initial subset.
Sounds like a logical solution to me, that's the easiest way to stop 
spoofed names from being registered. Again, it may or may not happen 
depending on the registries in question.

> friendly software to the end-user (registrant).
It would really be nice if filtering is done earlier (i.e. at the 
registrar level instead of registry level) but that's not currently the 
case for Verisign's registrars, I suspect because it requires 
significantly more work for both the registry and registrars, not to 
mention that when the rules change, the registry needs to push the 
changes to the registrars.

Right now, if you went to an ICANN accredited registrar and try to 
register an IDN, you'd need to input your unicode string, and select a 
language tag from a list, then click submit. The registrar does 
ToASCII(your_domain_name) and send the results along with the language 
tag to the registry after you have submitted your credit card details. 
Now, if the language tag you selected has a table attached, the registry 
will check the submitted domain against the table. At this point, if the 
test fails, an error will be returned to the registrar, then only do you 
get the message. And if you're lucky, at this point your credit card 
would be refunded.

That might just be a particular registrar's implementation, but I did 
experience it. I don't know if Edmon's IDN-over-EPP is going to help 
with the situation, since I don't know what's going on in the SRS.

wil.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 22 Feb 2005 14:26:56 +0100

On Mon, Feb 21, 2005 at 09:18:07PM -0800,
 Erik van der Poel <[email-masked]> wrote 
 a message of 218 lines which said:

> filtering IDN lookalikes, otherwise they will eventually face
> lawsuits from the "big boys" (as George so delightfully puts it).

Quite the opposite: according to our lawyer, if the process is
completely automatic (no human eyes involved), you can disclaim any
responsability. But if you do screen, you accept a liability if the
screening fails (and it will fail, trying to catch homographs is an
hopeless task).

I seriously doubt that european registries, which all moved from a
"screen every domain to check if it is legal" model to a "accept
anything" model in the '90s will go back...

---

From: Erik van der Poel <[email-masked]>
Date: Tue, 22 Feb 2005 09:41:37 -0800

> anything" model in the '90s will go back...

Chuckle. That's funny. Here I am telling a mathematician to think more 
like a network engineer, then I turn around and say something about law 
even though I'm not a lawyer!

Seriously, I did not say that human eyes would do the filtering (though 
of course humans would have to come up with the policies and code to do 
the filtering).

So, if a registry can claim that it can disclaim responsibility for 
spoofing *because* it is using an automatic registration process, then 
wouldn't it be possible for someone (or a class action) to claim that 
their automatic process isn't good enough? I mean, we all know where the 
obvious homographs are, and any engineer can tell you that it is easy to 
write a program to generate all the spoofs from those, or to filter them.

This may be a gray area that I believe Peter may have been referring to 
when he said that in some countries it might be possible to force the 
registry to change.

As it turns out, mozilla.org has also discussed the idea that Mozilla 
may not want to try to solve the IDN spoofing problem, since it cannot 
accept the legal responsibility for doing so.

Is the Unicode Consortium now also going to say "Sorry, we cannot 
provide homograph tables because we cannot be held responsible for any 
spoofing that may occur."?

Is everyone just going to pass the buck? How sad.

By the way, not all European registries "accept anything". Some of them 
are checking a character inclusion table to see if the domain name is 
allowed. What do you say to this?

Erik

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 22 Feb 2005 16:58:37 +0100

>anything" model in the '90s will go back...

Full agreement. Now, only for the reasons explained below, I am ready to 
test and propose several ccTLDs a filtering experimentation.

1. I filter the registered names (against foul names, blocked names) at 
registration level, before accepting payment.
2. the filtering will therefore be on xn--entries strings.

This being accepted:

1. could someone point a "C" source code to carry what has to be carried to 
filter out the dangerous names? Please help: I have no resource on this.
2. could someone list all the Unicode codes to blacklist that way?
3. could someone point a Perl code to use to enter a IDN and to get it 
properly punycoded, which could use such a list.

My rationale is that I only want to protect my own operations from 
confusion. I will only extend the description of the non-authorized 
characters in the terms and conditions. If this works properly I will 
describe the solution and experience in a for information Draft and a 
request to the IANA to list a Unicode black list.

jfc

jfc

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Tue, 22 Feb 2005 22:59:22 +0100

At 21:30 22/02/2005, Erik van der Poel wrote:
>JFC (Jefsey) Morfin wrote:
>It will take a while to create a relatively complete table of homographs, 
>but here are a couple of starting points:
>
>https://bugzilla.mozilla.org/attachment.cgi?id=174139
>https://bugzilla.mozilla.org/show_bug.cgi?id=279099#c192
>
>Also, I've been thinking of writing a program that would look at the 
>"cmap" of every font on a Windows box and check to see which pairs of 
>Unicodes have the same glyph index (which leads to identical display).

This would help.
But a ccTLD managing IDNs in computer environment and wanting to avoid any 
mistake, manages names in most of the case under the ACE format. In ASCII. 
I am not sure about existing dispute cases, but we consider that two IDNs 
are different if they have different in ACE format?
Anyway, I answer you below.

>I don't know about Perl, but I believe Python has IDN.

Thank you, but as I said, I have no resource on this. So what would be 
great wold be that this list would actually help preparing a Draft - may be 
someone of more technical skill and competence would be interested in 
leading it? So we can start working on something real. I listed my pratical 
needs. I suppose others would have others to add.

Stephane is key person in supporting many ccTLDs in real life. I am sure he 
will be of great help. So would Gervase's with the ability to test in 
Firefox environment.

I have reported the problem and my request on the ccTLD list. I asked about 
the additional requirements they might have. I will inform this list of any 
additional demands they may have IRT a practical solution for them. I also 
documented that my concern was not about the phishing issue but about the 
ccTLD owns operations. This leaves the legal aside and may be more 
motivating since their own Registry could be the first victim of a 
confusion (in Whois display, for example).

jfc

---

From: William Tan <[email-masked]>
Date: Wed, 23 Feb 2005 10:27:49 +1100

Jefsey,

Which ccTLDs do you intend to propose this experiment? If you don't mind 
revealing...
Have these ccTLDs rolled out IDN implementation? If so, do they 
currently employ a character table of sorts?\

> 1. could someone point a "C" source code to carry what has to be 
> carried to filter out the dangerous names? Please help: I have no 
> resource on this.

I'm not sure what exactly you need. If you have the tables it should be 
fairly trivial to write some functions in C to filter names. Make sure 
your inputs are properly normalized and just compare the 
to-be-registered name against your blacklist, reject the name on any match.

> 2. could someone list all the Unicode codes to blacklist that way?

If character tables are used, they effectively serve as a whitelist. If 
you wish to disallow any character from being registered, just make sure 
that it's not on the whitelist.

> properly punycoded, which could use such a list.

Try  Net::LibIDN (http://spaceship.berlios.de/index.php?page=idn.php) 
which is a Perl binding for GNU Libidn. It seems to have some support 
for character inclusion list, not sure about blacklists though.

Regards,
wil.