big mailing lists / slashdotable hosting

[Seth Cohn]

On 3/28/07, Bill McGonigle <bill at bfccomputing.com> wrote:
> I'm wondering if anybody has experience running big mailing lists.
> Sure, I run a bunch of mailman lists, but I'm looking at what it
> would take to run a mailing list with a quarter million subscribers.
> (all opt-in, non-spam, properly run).

Big, but not quite that big.  Tens of thousands, yes.

> 1) software - I hear mailman can scale that big with a mysql
> backend.

Bleh.  With that many subscribers, it's not a many to many list, it's
a one to many list, and mailman is the wrong tool.  There is but one
good answer (IMHO): phplist.

> 2) facility - in-house (bigger Internet connection) vs. co-locating
> vs. outsourcing.

inhouse is totally possible, coloing as well though you'll want a
dedicated box not shared for that huge a list.  (For 10Kish, shared
works fine though)

> think, but quite expensive per bit.  in-house has advantages as I can
> do things like rate-limiting to reduce the load, but:

phplist will do that, in a variety of ways.

> 3) blacklists - I expect with that many subscribers enough will have
> forgotten they opted in that blacklists will start listing whatever
> server sends mails.

With that many, yes, it's possible. With one 10K+ list, the initial
seeding of that list from some preexisiting related lists (despite
optout messages and the first few messages carefully disclaimered with
a unsub link right up front) caused some *multiple times* wrongful
spamfilter/blocking from one of the worst trigger-happy antispam
services , but since then it's been fine...

> good reputation (I have no idea what that means).  Is it a full time
> job to babysit such a list and get off of blacklists?

Depends on the level of opt-in, your subscriber base, the temperment
of your users, content of your emails, etc, etc.

> 4) image hosting - the mails will have image links in them because
> apparently lots of people like to see pictures (I always look at the
> text alternative myself...).

While you could do it that way, another option is to embedded the
images, so they are sent as part of the message.  phplist does this.
There are perks to both methods, and disadvantages, but you should
consider it.

> I expect to see 100GB or so of image
> traffic, peaking up to 50Mbps.
> Can I load this up on a shared hoster
> offering 3TB per month of transfer for $12 or will they kill anything
> that approaches those limits?   I'd like to find a hoster that brags,
> "we can handle a Slashdotting", because this would be large, bursty,
> very intermittent traffic.  Akamai-type solutions look to be too high-
> end for this task.

If your figures are accurate, no good shared hoster will want that
business, for just $12 a month.  Most of the braggers either charge
more, or are falsely bragging.

> 5) everything else I didn't think to ask

AOL and other ISPs are moving toward blacklisting bulk mailing except
if you pay them for access.  There are non-profit discounts/free/etc,
for example, but it's some work to get listed. With that sort of
volume, I suspect it would be worth while look into if you qualify,
and if not, what is the effect of not paying on your delivery rates.

> I'd appreciate feedback on any of those items.  If anybody has
> actually done this soup-to-nuts before, I'm interested in hiring that
> person for a short consulting gig.

I'll contact you offlist so we can discuss it further.

Seth