IMAP debate

[Derek Martin]

On Tue, Oct 21, 2003 at 07:02:26PM -0400, bscott at ntisys.com wrote:
>   My understanding is that mbx is more-or-less mbox with a persistent index
> added on.
> 
>   I might point out that, for an IMAP server at least, client-side
> persistent indexes are just about useless.

Of the sort we're talking about, obviously, since the client isn't
dealing with the on-disk folder format.  But the server can use them
to speed response time up, and the client can use caching techniques
to improve performance on the local end.  In the case of the IMAP
client, as you point out yourself in a message to Paul, the on-disk
file format is irrelevant.  I'm not really sure why you mentioned
it here...

> >> Even with mbx, purge/expunge is absolutely horrible.
> > 
> > If you re-write the folder in place, expunging a few messages from the end
> > of the file will not take long. This should be the common case, logically.
> 
>   The problem with re-writing the folder in place is that an interruption at
> the wrong time corrupts the mailbox, leading to lost data.  That isn't
> something I would want to use.

Mutt has been using this technique for years, and I have never
experienced any data corruption because of it, or for that matter
heard anyone else make a similar complaint.  

Assuming the program does proper signal handling, the only reason for
such a thing to happen is some sort of hardware failure.  If the
program does not do proper signal handling, the fault is in the
program, not the format.  No format is immune to bad programming
practices...

If such a failure happens, you're probably going to have to restore
from backups anyway.  Practically speaking, I think that's not a case
that's worth being worried about.  Furthermore, the rewrite-copy
method is not immune to this problem.  If the same sort of
interruption were to occur while the copy is being re-written to its
proper place, you'll still end up with a corrupted mail spool.

Maildir is safer, but it is not immune to this problem either.  The
same kind of failures can cause file system corruption.  If the blocks
affected are directory blocks, your folder is shot.  Maybe fsck can
fix it, but maybe it can't.

Data corruption happens.  Make good archives, test them periodically,
and don't worry about it.


> > Especially arge folders tend to be archive folders.
> 
>   Heh.  For a great many people, it appears an "inbox" is considered an
> "archive folder".  :-)

True enough, but the same principle holds.  Earlier messages tend to
be saved.  Later messages tend to be deleted.  There will always be
exceptions, but they will be relatively infrequent.

> >> The thing is, that one minor implementation detail can be easily changed
> > 
> > a) This is NOT a minor implementation detail
> 
>   Um, how hard would it be to change the location of the temporary file from
> /tmp to the direction the mailbox is in?  Granted, I haven't looked at the
> code, but I have a hard time envisioning an implementation where this would
> not be a quick fix.

The minor change you are describing would certainly be an improvement,
but it has a long way to go.

But, I said it isn't a minor implementation detail precisely because
how you implement this aspect of the program has a profound effect on
the performance of the server.  Therefore, it is a major
implementation detail.

> > Mbox clearly has some disadvantages, but its creators were not idiots.
> 
>   I'm not calling the creators of mbox idiots.  Nor do I intend to imply the
> problems with mbox are due to designer brain damage.  I probably should 
> have stated that explicitly, though.

I really didn't mean to imply that you were saying that.  Nor, for
that matter, am I suggesting that mbox is actually better than
Maildir, or that it doesn't have any limitations.  I only meant that
mbox really isn't as bad as a lot of people make it out to be.  

I feel confident that if you were to experience using mbox with a
better IMAP server than UW-IMAP, which used some of the ideas I've
been describing, you never would have found mbox's performance
sufficiently bad to warrant considering another format.  You would
still experience file size limitations, which you would still have to
educate your users about.  You would still have to worry about
locking, and NFS shares of your mail spool, if you were dumb enough to
offer them.  But I guarantee the performance would be a lot better,
without having to make /tmp a 10 GB partition...

> Yes, all those are well-known, and frequently brought up by the UW folks.  
> You'll note that I was not talking about Maildir or mh, but Cyrus.  The
> persistent index which Cyrus maintains eliminates most of them.  

If I ever get off my duff and write it, my IMAP server will have the
same for mbox.  This is not a performance gain inherent to the format
(which is still maildir).  

> The rest are, as you note, easily eliminated by spec'ing the
> filesystem to handle Cyrus in the first place.  Which seems like a
> no-brainer to me.

Yeah, but it's a consideration for someone who is moving from UW-IMAP
to something else.  Which IIRC (I may not) is where this thread came
from.  If your filesystem is not already set up for it, you'll need to
make a new FS.  For some people, that may be a barrier to switching.

> > My contention is that for operations which matter most, mbox is
> > actually faster or sufficiently fast enough, in the common cases,
> > that there is no clear performance winner between the two formats,
> > given efficient implementations of both.
> 
>   All I really know is this: In a production environment, with an IMAP
> server, given the way most companies use email these days, UW-IMAP+mbox is
> orders of magnitude worse then Cyrus.  

And there, I can not argue with you.  But the fault is a bad program,
not a bad data format.

-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.
Replying to it will result in undeliverable mail.
Sorry for the inconvenience.  Thank the spammers.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20031022/074a0f3d/attachment.bin