Subject: Re: [imapext] IMAP message size limits

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 24 Jul 2012 07:12:22 +0000

------ Original Message ------
From: "Lyndon Nerenberg" <[email-masked]>
>On 2012-07-23, at 18:21 PM, Adrien W. de Croy wrote:
>
>
>>
>>What's the alternative you suggest?
>>
>
>
>As you alluded, moving beyond 32 bits means a new protocol beyond IMAP4.
>
>But first I ask, where are these 4 GB email messages you keep sending around?
>
I did ask myself that.

then I decided I wouldn't try to answer that for everyone else.

The recording studio below us commonly deals with files much bigger 
than that.

Why should email be the poor cousin of other protocols and be unable to 
deal with files of that size.

I'm pretty sure a POP3 server can serve an email over 4GB.  I checked 
RFC1939 and didn't see any limits in there.

So it's a bit sad that IMAP has this limit, when actually it could be 
argued that the size of numbers is an implementation issue rather than 
a protocol issue, and implementors could be guided to think about the 
future (e.g. support bigger numbers).

Adrien
>Is sending them in and out of email mailboxes the most efficient way of sending this data around?  Before looking at ways to copy terabytes of data back and forth, might a way of exchanging pointers to the data not be more efficient use of resources?
>
>--lyndon
>
>
>

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 24 Jul 2012 10:00:33 -0400

Hi Adrien,

--On July 24, 2012 7:12:22 AM +0000 Adrien de Croy <[email-masked]> wrote:

> The recording studio below us commonly deals with files much bigger than
> that.
>
> argued that the size of numbers is an implementation issue rather than a
> protocol issue, and implementors could be guided to think about the

Are people really sending files that big via email as opposed to using a 
file sharing service like e.g. DropBox? File sharing service is going to be 
a lot more efficient at upload/download than email. Perhaps it is a shame 
that no one really supports message/external-body because that could have 
really helped the process of sending attachments "by reference" instead of 
"by value".

-- 
Cyrus Daboo

---

From: Andrew Laurence <[email-masked]>
Date: Tue, 24 Jul 2012 10:47:04 -0700

On Jul 24, 2012, at 7:00 AM, Cyrus Daboo wrote:

> <snip>

> Are people really sending files that big via email as opposed to using a file sharing service like e.g. DropBox? File sharing service is going to be a lot more efficient at upload/download than email. 

Many years ago, a couple of wiseacres here emailed a CD-ROM disk image to each other, in order to see if it would arrive.  It eventually did, but took days to wind its way through the various queues.  (I suppose this was before a maximum message size limit was imposed on the central systems.)  Now that people routinely handle ISOs and video files, I think it's entirely likely for someone to try sending a DVD-size file via email.  The mental association of "I am sending information to you" does not carry a gate point of size vs delivery mechanism.

> Perhaps it is a shame that no one really supports message/external-body because that could have really helped the process of sending attachments "by reference" instead of "by value".

Over in vendorland, slipstream products such as Symantec Enterprise Vault (an acquisition, used to be called something else) decouple the binary payloads from message content parts. The former are stored outside the mailstore, with just references stored in the mailstore.  These solutions are pricey, but the core email system is left smaller, faster, and more manageable.

-- 
Andrew Laurence                Office of Information Technology 
[email-masked]               University of California, Irvine

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 24 Jul 2012 21:55:20 +0200

On 07/24/2012 07:47 PM, Andrew Laurence wrote:
> Many years ago, a couple of wiseacres here emailed a CD-ROM disk image to each other, in order to see if it would arrive.  It eventually did, but took days to wind its way through the various queues.

I did that in May, 1992. Laid waste to the NZ university network's 
satellite link from Friday to the following Monday. A month or two later 
the nation of South Africa upgraded its internet link from 9600 bps to 
64kbps, only to encounter me on the first evening.

Is there something to learn from this? Really big mail happens, and it's 
not good for anything?

Arnt

---

From: Andrew Laurence <[email-masked]>
Date: Tue, 24 Jul 2012 13:45:46 -0700

On Jul 24, 2012, at 12:55 PM, Arnt Gulbrandsen wrote:

> Is there something to learn from this? Really big mail happens, and it's not good for anything?

I don't know about that.  These trail breakers cleared the path for crazy uncles to email conspiracy video clips to everyone they've ever met.  :-)

Nearly twenty years into an IT career, my takeaway is that the layperson will never sufficiently associate the notion of storage quantity with a file they want to send to another person.  We will never win the battle of educating users to not use email for sending files.  While I find it annoying, I don't dispute that sending files via email happens, and (for the user) is useful and efficient.

So long as messaging is email as we know it includes the ability for users to bundle fat payloads with the transport of message content, we will battle this problem.  As disks and pipes get bigger it becomes less of a problem, but there is always a point at which transport of a given payload is destructive to the transport systems.

Getting back to the original question, I think a size limit inherent to the protocol should be corrected.  There will always be a point at which what was once absurdly large becomes commonplace.  Leave it to the operators to enforce limits that make sense for their site.

(This site imposes a 30-million-byte limit.)

-Andrew

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 24 Jul 2012 21:55:13 +0000

We don't have such limits with other protocols such as FTP or HTTP (or 
POP3 or SMTP).  At least not baked into the protocol itself.

One of the benefits of a text-based protocol, esp when communicating 
numbers, is the protocol can transmit a number of any size.  Just send 
more digits.  BNF doesn't change.

What size numeric types are used by agents to transmit numbers is then 
an implementation decision.

We can and should give guidelines to make people think about the scale 
of things, but putting a hard limit into the protocol I think has 
issues.

If developers know that they can always see a bigger number, then they 
can code / plan accordingly.  E.g. use 64 bits, and keep an eye on 
things.  Or whatever.

When people have to store UIDs and sizes, then the extra 64 bits per 
message can add up, so I understand people wanted to keep this down.  
But disks are a lot bigger now and 32 bits is looking smaller and 
smaller with the passage of time.  We only recently added support for 
64bit sizes to WinGate.  It took about 1/2hr.  We changed some atoi to 
atou64, and some %u to %I64u and changed types.

So for mainstream email clients, I don't see much of a problem moving 
to handling 64 bits for sizes etc (maybe some cache migration issues).  

For other IMAP clients, it's only going to be a problem talking to a 
newer server.  Most servers allow setting a max message size limit 
administratively.

I agree, "by-reference" could be very interesting.

Eudora used to strip attachements out of their MBXs and store them 
independently.  IME it actually created other problems.

Adrien

------ Original Message ------
From: "Cyrus Daboo" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>;"Lyndon Nerenberg" 
<[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 25/07/2012 2:00:33 a.m.
Subject: Re: [imapext] IMAP message size limits
>>than 
>Are people really sending files that big via email as opposed to using 
>a file sharing service like e.g. DropBox? File sharing service is 
>going to be a lot more efficient at upload/download than email. 
>Perhaps it is a shame that no one really supports 
>message/external-body because that could have really helped the 
>process of sending attachments "by reference" instead of "by value". 
>
>-- Cyrus Daboo 
>