Subject: [EAI] SMTPUTF8 and 8BITMIME

From: Franck Martin <[email-masked]>
Date: Wed, 15 Apr 2015 09:24:25 +0000

it seems that a few MTA can be thrown at an email with a EAI RFC5321.mailfrom and pass the message fine to other MTA or local delivery, as long as the MTA supports 8BITMIME and the RFC5321.rcptto is a standard email address. 

replying to such email is a challenge, but it does not seem any MTA or MUA (with 8BITMIME) will complain because the RFC5321.from is an EAI...

What’s your experience?

---

From: Mark Martinec <[email-masked]>
Date: Wed, 15 Apr 2015 16:42:38 +0200

2015-04-15 11:24, Franck Martin wrote:
> it seems that a few MTA can be thrown at an email with a EAI
> RFC5321.mailfrom and pass the message fine to other MTA or local
> delivery, as long as the MTA supports 8BITMIME and the RFC5321.rcptto
> is a standard email address.
> 
> replying to such email is a challenge, but it does not seem any MTA or
> MUA (with 8BITMIME) will complain because the RFC5321.from is an
> EAI...
> 
> What’s your experience?

Tried it with Postfix (with SMTPUTF8 disabled):

it gladly accepts UTF-8 in localpart of a RFC5321.mailfrom (or in the
RFC5321.rcptto) address, as well as UTF-8 in a mail header section - but
rejects UTF-8 in domain part of such address (501 5.1.7 Bad sender
address syntax).

... unfortunately the cyrus LMTP server then rejects the 
RFC5321.mailfrom
(501 5.5.4 Syntax error in parameters) - although it does survive UTF-8
in a mail header section.

   Mark

---

From: [email-masked]
Date: Wed, 15 Apr 2015 07:50:19 -0700 (PDT)

> it seems that a few MTA can be thrown at an email with a EAI RFC5321.mailfrom
> and pass the message fine to other MTA or local delivery, as long as the MTA
> supports 8BITMIME and the RFC5321.rcptto is a standard email address.

> replying to such email is a challenge, but it does not seem any MTA or MUA
> (with 8BITMIME) will complain because the RFC5321.from is an EAI...

> What’s your experience?

Depending on the version and configuration, the Oracle MTA will either accept
such a MAIL FROM address but sanitize the 8bit, or reject it outright.

RCPT TOs containing 8bit are always rejected.

				Ned

---

From: Oleksandr Tsaruk <[email-masked]>
Date: Tue, 21 Apr 2015 16:07:01 +0300

Is it possible to reconsider (in a very long run) EAI WG general approach to:

“This working group's previous experimental efforts investigated the use of UTF-32 as a general approach to email internationalization.”
In such case email/domaine internationalization problem could be solved in original scripts? 

Charter for Working Group
"The email address has two parts, local part and domain part. Email address internationalization must deal with both. This working group's previous experimental efforts investigated the use of UTF-8 as a general approach to email internationalization."
https://datatracker.ietf.org/wg/eai/charter/

Oleksandr

Sincerely yours, 
Oleksandr Tsaruk, Ph.D.

-- реклама -----------------------------------------------------------
Не дорогой качественный хостинг. Приятные сюрпризы для каждого!
http://freehost.com.ua/unix/

---

From: John C Klensin <[email-masked]>
Date: Tue, 21 Apr 2015 10:38:13 -0400

--On Tuesday, April 21, 2015 16:07 +0300 Oleksandr Tsaruk
<[email-masked]> wrote:

> "This working group's previous experimental efforts
> investigated the use of UTF-32 as a general approach to email
> internationalization." In such case email/domaine
> internationalization problem could be solved in original
> scripts? 

In principle, yes.  In practice, given the increasing (and
increasingly universal) use of UTF-8 on the wire, probably not.

The more important question is what you think going to UTF-32
would accomplish.  It is fully isomorphic with UTF-8 -- there is
no information that can be represented one way and not the
other.  It is much less compact than UTF-8 for "western"
alphabetic scripts (including Cyrillic), less compact for any
BMP code point, and never worse (in terms of more bytes per code
point.  UTF-32 does not get involved with the "surrogate" mess,
but neither does UTF-8.  Neither helps at all with the various
normalization or comparison problems.   The only advantage I can
think of at the moment is that UTF-32 permits getting a count of
the number of code points present by counting octets and
dividing by four while UTF-8 (and UTF-16) require some
calculations.  However, one rarely cares about number of code
points as compared to, e.g., number of "print positions" or
"characters" and, given combining sequences and non-spacing
characters and marks, getting from a code point count to print
position information cannot be done without considerable
knowledge of the code points involved (and, for some scripts,
rendering procedures).

So, can you explain what you think a move to UTF-32, even if it
were possible, would accomplish?

    john

---

From: Oleksandr Tsaruk <[email-masked]>
Date: Wed, 22 Apr 2015 15:54:43 +0300

Dear John

Suggestion on UTF32 was about capacity of the table which can handle all known and future developed symbols and hieroglyphs, for case of our days UTF-16 with about 1 million cells could keep all basic symbols of known languages in the Earth. 

Best 
Oleksandr

21.04.2015 17:38, John C Klensin <[email-masked]>

Sincerely yours, 
Oleksandr Tsaruk, Ph.D.

---

From: John C Klensin <[email-masked]>
Date: Wed, 22 Apr 2015 09:21:49 -0400

--On Wednesday, April 22, 2015 15:54 +0300 Oleksandr Tsaruk
<[email-masked]> wrote:

> Dear John
> 
> Suggestion on UTF32 was about capacity of the table which can
> handle all known and future developed symbols and hieroglyphs,
> for case of our days UTF-16 with about 1 million cells could
> keep all basic symbols of known languages in the Earth. 

Oleksandr,

As I suspected, there is probably a misunderstanding somewhere.
UTF-32 and UTF-16 are just different encodings for Unicode and
can represent exactly the same information.   UTF-16 represents
code points above U+FFFF by the use of so-called "surrogates".
I, personally, dislike surrogates and consider them a kludge,
but their availability as part of the UTF-16 system means that
UTF-16 can represent anything that UTF-32 can... and so can
UTF-8.

Now you may be thinking about something else.  The Unicode code
space ranges only from 0 to 0x10FFFF.   A 32bit code space would
be 0 to 0xFFFFFFFF.  If one believed that the Unicode code space
were too small and that a full 32 bit space were needed, that
would be a different matter entirely.  The Unicode folks are
convinced that more than the current space will never be needed
to represent all symbols of interest but, at one stage, they
believed that a 16bit code space would be enough.  I haven't
studied the mechanisms in some years, but, if a larger code
space were needed, extensions would be needed to both the UTF-8
and UTF-16 encoding models to accommodate it while anything that
used a 32 bit space directly would presumably be fairly
transparent, at least until one got into various Unicode tables
and algorithms that assume the smaller code space.   But, again,
that has little to do with the difference between UTF-16 and
UTF-32 except as a side effect.

 best,
    john

>> > investigated the use of UTF-32 as a general approach to
>> > email internationalization." In such case email/domaine
>> would accomplish. It is fully isomorphic with UTF-8 -- there
>> is no information that can be represented one way and not the
>> BMP code point, and never worse (in terms of more bytes per
>> code point. UTF-32 does not get involved with the "surrogate"
>> mess, but neither does UTF-8. Neither helps at all with the
>> various normalization or comparison problems. The only
>> advantage I can think of at the moment is that UTF-32 permits
>> getting a count of the number of code points present by
>> counting octets and dividing by four while UTF-8 (and UTF-16)
>> require some calculations. However, one rarely cares about
>> number of code points as compared to, e.g., number of "print
>> positions" or "characters" and, given combining sequences and
>> non-spacing characters and marks, getting from a code point
>> count to print position information cannot be done without
>> considerable knowledge of the code points involved (and, for
>> some scripts, rendering procedures).
>> 
>> So, can you explain what you think a move to UTF-32, even if
>> it were possible, would accomplish?
>> 
>> john
> 
> 
> Sincerely yours, 
> Oleksandr Tsaruk, Ph.D.
> 
> 
> -- реклама
> -----------------------------------------------------------

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 22 Apr 2015 09:32:08 -0400

On Wed, Apr 22, 2015 at 09:21:49AM -0400, John C Klensin wrote:
> be 0 to 0xFFFFFFFF.

Importantly, if that's the problem one is trying to tackle, there
isn't anything the IETF can do about it.  It's fully a Unicode issue.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Franck Martin <[email-masked]>
Date: Wed, 22 Apr 2015 16:10:12 +0000

Toute connaissance est une réponse à une question.

> On Apr 22, 2015, at 06:32, Andrew Sullivan <[email-masked]> wrote:
> 
Yes, Unicode has a few problems to tackle that is not in the realm of IETF

https://modelviewculture.com/pieces/i-can-text-you-a-pile-of-poo-but-i-cant-write-my-name

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 24 Apr 2015 19:15:18 +0900

> UTF-32 except as a side effect.

More specifically, at one point, Unicode thought that a 16-bit space 
might be enough (for that time being, at least), while on the other hand 
ISO/IEC JTC1/SC2/WG2, the ones responsible for ISO 10646, thought that 
an architecture with a full 31 bits would be better (the 32nd bit was 
always reserved because nobody wanted to repeat the 8-bit "signed char" 
vs. "unsigned char" mess). UCS-4 and UTF-8 were both designed to 
encompass this 31-bit code space. UTF-8 needed up to 6 bytes for a 
character, in a very straightforward way, according to its original 
design (There is still code out there with traces from this period.). 
UCS-2 was of course limited to 16 bits.

With time passing, it became clearer on both sides that 16 bits wasn't 
enough but 31 bits was overkill. The introduction of UTF-16 created an 
upper limit of 0x10FFFF in one of the Unicode encoding forms. Therefore 
both Unicode and SC2/WG2 agreed on this overall limit. UTF-32 was 
introduced as a version of UCS-4 with an explicit upper codepoint limit 
of 0x10FFFF, and the definition of UTF-8 was changed to only go up to 
four bytes.

In the case of an extraterrestrial invasion by a culture with millions 
of characters or a sudden excessive emoji binge, the limits for UCS-32 
and UTF-8 could be changed again, and some further kludge could be 
introduced in UTF-16. But the chance that we'll get there is very low, 
at least for the moment.

Regards,   Martin.

---

From: Oleksandr Tsaruk <[email-masked]>
Date: Fri, 24 Apr 2015 15:06:31 +0300

Dear colleagues

I am very glad that I successfully have drawn your attention, I an new at IETF, and participated only Dallas meeting but have got some experience at IDN board and at ICANN GAC.
We saw that Punycode solving basic IDN problems but have no perspectives. 

That is why I posted this provocative question on UTF32 and glad to see development of idea in ISO/IEC JTC1/SC2/WG2.

The basic propose is to find solution for IDN domains and email addresses be present in original scripts.  

Best regards,
Oleksandr Tsaruk

24.04.2015 13:15, "\"Martin J. Drst\"" <[email-masked]>

Sincerely yours, 
Oleksandr Tsaruk, Ph.D.

-- реклама -----------------------------------------------------------
Поторопись зарегистрировать самый короткий почтовый адрес @i.ua
http://mail.i.ua/reg - и получи 1Gb для хранения писем

---

From: John C Klensin <[email-masked]>
Date: Fri, 24 Apr 2015 14:34:40 -0400

--On Friday, April 24, 2015 19:15 +0900 "\"Martin J. Dürst\""
<[email-masked]> wrote:

>...
> More specifically, at one point, Unicode thought that a 16-bit
> space might be enough (for that time being, at least), while
> on the other hand ISO/IEC JTC1/SC2/WG2, the ones responsible
> for ISO 10646, thought that an architecture with a full 31
> bits would be better (the 32nd bit was always reserved because
> nobody wanted to repeat the 8-bit "signed char" vs. "unsigned
> char" mess). UCS-4 and UTF-8 were both designed to encompass
> this 31-bit code space. UTF-8 needed up to 6 bytes for a
> character, in a very straightforward way, according to its
> original design (There is still code out there with traces
> from this period.). UCS-2 was of course limited to 16 bits.
>...

Martin,

Yes.  And thanks for the additional details.

I think the bottom line, at least barring some very unlikely
events (such as the extraterrestrial invasion you mention or a
change to conventions about what gets encoded such as the
breakdown in Han Unification that the article Franck pointed to
seems to request), is that we are not even close to exhausting
that 0x10FFFF limit and hence, whatever the issues are, not
having the full 31 bit (as you correctly point out) range
available is not in the critical path.

Now, there is a lot to dislike about Unicode.   Depending on
one's starting perspective about coding system design, one can
come up with different preferences and priorities.  It is
reasonably clear, at least in retrospect, that some scripts and
writing systems have been better treated than others.
Adjustments made for better compatibility with earlier standards
for particular language-based repertoires (some of which
involved  compromises to fit in seven or eight bit code spaces),
led to later inconsistencies or other difficulties.  However,
there are no other realistic candidates for a "universal"
character encoding system and, while a hypothetical system based
on a different set of rules, priorities, and tradeoffs  might
easily demonstrate a different collection of difficulties, there
is no evidence at all that it would be objectively better...
just different.

So here we are, with a coding system that will never be perfect,
possibly because we'd really like a consistent and predictable
set of rules that apply across the entire code space but that
neither the diversity of human (to say nothing of
extraterrestrial) languages and writing systems nor the politics
and conflicts over priorities that always intrude on such
standards lend themselves to that level of consistency and
predictability.  Building on top of Unicode to fit well with
particular applications can be hard, but we seem to have little
choice other than to keep trying (see, e.g., IDNA and PRECIS as
well as the recent LUCID discussions and the associated drafts).

Whatever the path forward, it is clear to me that there is no
plausible path for email addresses that adopts a fundamentally
different character coding system or underlying repertoire than
what the rest of the IETF (and mostly the rest of the world) are
doing.

   john