Subject: Characters used in a tail-entry

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 21 Sep 2005 23:45:51 +0200

Just because Richard gave me access to his logfiles, I felt like I had to 
try...

these are the characters used in the tail-entry over roughly 100.000 of the 
articles he had some data on.

I think it safe to say that the characters in the string "not-for-mail" are 
overrepresented! - but it seems that * is a more popular character than the 
humble "q"...... and even the dread ":" makes a few appearances....

  50
# 3
$ 16
% 79
* 114
+ 49
- 1530391
. 310129
0 83675
1 89062
2 129645
3 39805
4 7962
5 14998
6 19651
7 1836
8 10336
9 11612
: 8
@ 67
A 28605
B 11613
C 7959
D 1780
E 3547
F 106046
G 101465
H 3814
I 6260
J 109
K 112732
L 552
M 106187
N 105156
O 1163
P 74809
Q 298
R 766
S 112420
T 214607
U 3595
V 61
W 1676
X 27671
Y 4697
Z 429
_ 3586
a 828374
b 158969
c 59143
d 18529
e 77279
f 796241
g 168753
h 143763
i 794495
j 1570
k 19999
l 932620
m 827806
n 858626
o 1578315
p 185160
q 94
r 812120
s 85812
t 843265
u 23566
v 4044
w 30353
x 153475
y 7880
z 2031
{ 6
| 66
} 6
€ 77
¥ 77
¯ 77
° 77
³ 77
ž 154
Œ 77
À 77
Æ 77
æ 77
ê 77

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 22 Sep 2005 17:19:30 +0200

Harald Tveit Alvestrand wrote:

> it seems that * is a more popular character than the humble
> "q"...... and even the dread ":" makes a few appearances....

The 154 =B8 are also odd.
http://www.cs.tut.fi/~jkorpela/latin9.html says it's u+017E.

There's a bunch of other Latin-9 charaters occuring 77 times,
maybe it's always the same tail entry with two =B8 (154=2*77).

Your results indicate that allowing "." (dot) is a good move.
And "+" is apparently irrelevant (abusing 154 as the limit):

#   3		{   6		}   6
:   8		$  16		+  49
   50		V  61		|  66
@  67		%  79		q  94
J 109		* 114
                      Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 22 Sep 2005 11:51:47 GMT

>humble "q"...... and even the dread ":" makes a few appearances....

I think the main thing is to exclude any character which some server might
think was a delimiter (for then you could not be sure to detect the
boundaries of the <tail-entry>). Hence my suggestion for:

tail-entry = 1*( alphanum / "-" / "_" / "." )

which gives you the union of barewords, domains and IP addresses (except
for ':'). Whatever we put, somebody will try to use something outside of
it (maybe even in supposed barewords). I don't think we should particularly
care, and it is unlikely any actual harm would arise; but I would prefer
not to encourage some of the odder characters by actually including them
in the syntax. Obviously, the few Non-ASCII characters in your sample
ought not to be there.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5