IMAP debate

[Derek Martin]

On Mon, Oct 20, 2003 at 08:26:39PM -0400, bscott at ntisys.com wrote:
> On Sat, 18 Oct 2003, at 10:52am, plussier at mindspring.com wrote:
> > Based on the stupidity of the UW IMAP design, I'd never use it for more
> > than a couple people in a non-critical environment.  It completely falls
> > over with large numbers of users who have large folders with many messages
> > with many large attachements.
> 
>   To be fair, this has more to do with the design of mbox/mbx then the
> design of UW-IMAP.

To some degree.  But UW-IMAP does a particularly bad job of handling
it.

> The lack of a persistent index makes mbox
> pretty much the worst format imaginable.  

There's no reason why your client can't provide such a thing, even if
your mail system as a whole does not...

> Even with mbx, purge/expunge is absolutely horrible.

This is only needs to be true if you are deleting messages from near
the beginning of a large spool file, and this is not a very common
case with typical e-mail usage.

In general, e-mail near the beginning of an active folder is generally
mail you want to keep there for whatever reason.  Mail you want to
delete is most commonly recent mail that you don't care about, which
will be at the end of the file.  If you re-write the folder in place,
expunging a few messages from the end of the file will not take long.
This should be the common case, logically.

Especially arge folders tend to be archive folders.  Deleting messages
from these is fairly uncommon, and still deleting them from near the
beginning is even less common.  So, only in rare cases should an
expunge be particularly slow, if the algorithms used to perform it are
not brain-dead.  UW-IMAP's algorithms are brain-dead.  But they are at
least brain-dead for a reason: compatibility.  They don't need to
worry about whose mmap() works and whose doesn't, etc.  Their
algorithms /lose/, but they always /work/.


> On Sun, 19 Oct 2003, at 12:27pm, p.lussier at comcast.net wrote:
> > Well, it's that, and the fact that to delete/expunge messages, it first
> > copies the entire mailbox to /tmp, unlinks the file, then copies the
> > mailbox back to var/spool/mail one message at a time, skipping the
> > messages flagged for deletion.
> 
>   The thing is, that one minor implementation detail can be easily changed,
> but mbx would still suck mud, and mbox would still suck radiologically
> contaminated mud.

a) This is NOT a minor implementation detail, and b) see above.  The
most common cases of e-mail usage can be made a lot more reasonable.
Mbox clearly has some disadvantages, but its creators were not idiots.
Maildir also has disadvantages.

 - Because parsing directories on most filesystems is an O(n)
   complexity problem, large numbers of messages make opening maildir
   slower than mbox.  This will also make unlink() slower, which
   somewhat narrows the gap in performance between delete/expunge
   operations in maildir and well-implemented mbox.  This is less true
   on filesystems specifically designed to handle this case.  Most are
   not.
 
 - Because of file open() overhead and greatly increased probability
   of disk seeking between reading messages, opening maildir folders
   is slower than mbox.  File system design probably does not affect
   this.  Maintaining a proper index will (but likewise for mbox).
 
 - it becomes easy to run out of inodes and other file-related
   structures if you did not create the filesystem with Maildir in
   mind.

 - other (mostly minor) oness that I'm not thinking of right now.

>   Of course, there is also the question of designer intent.  The UW-IMAP
> people obviously favor mbx and dislike mh/Maildir.  I regard that as a poor
> decision, but hey, it's their code.  

I think if you look at it carefully, comparing WELL-DESIGNED
implementations of both, the performance trade-offs between mbox and
maildir are not as great as many would have you believe.  Cyrus's own
figures suggest that there are areas where mbox is better than
maildir, and they used UW's crufty implementation to compare, which
IMO is not necessarily a very good comparison.  My contention is that
for operations which matter most, mbox is actually faster or
sufficiently fast enough, in the common cases, that there is no clear
performance winner between the two formats, given efficient
implementations of both.  I have been working on an implementation of
both to demonstrate this, but I'm not terribly committed to the
project...  So it's still quite a long ways off.  ;-)

Cyrus may maintain an index, speeding up many of its operations, but
there's no reason you can't also do that in your mbox implementation.
Obviously you'd need to patch Sendmail/procmail to do it, or just
write your own MDA.  Which is what the Cyrus people did.  You may want
to call the resulting mail folders mbx, which is fine; to me it's the
same difference.  The index is seperate from the format.

The only clear benefit Maildir has over mbox, IMO, is lock contention.

-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.
Replying to it will result in undeliverable mail.
Sorry for the inconvenience.  Thank the spammers.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20031021/e75e9777/attachment.bin