FeedDigest users will have noted that at least once a month I post a "I've changed a bunch of stuff, now FeedDigest is faster!" sort of post to the FeedDigest blog. This is because FeedDigest has grown so quickly that the technology just hasn't scaled perfectly. Servicing, say, 15000 users is radically different to servicing 1500. Through FeedDigest's (and RSS Digest's) history it's typically run at 'almost capacity', and my upgrades and tweaks have helped it grow for another month or so before requiring yet more fixes.

A couple of weeks ago I adjusted FeedDigest to be able to run off of static file backups so that digests could be served even with the database turned off. This meant I'd be able to take the database down and do all sorts of optimization work and not many people would notice any change in service. Digests wouldn't update, of course, but it'd let me do some crucial work. And.. merely a few days before jetting off to the other side of the world, I did.

After quite some stress, sweat, and panic, I've upgraded FeedDigest to MySQL 5. I also made significant tweaks to the post queueing system and.. wow.. performance has gone through the roof. Load has dropped 75%! MySQL 5 is certainly faster, although I've found a couple of weird bugs, mostly related to the Perl client libraries though. Crawling has also gotten faster since the database is a lot faster to write to. Rather than having 'free user' feeds queueing up and taking ages to update, they're going straight through almost as quickly as the paid user feeds.

I'm still going to get a separate server set for paid users so I can do database replication and keep paid users happy, but I'm really impressed with these upgrades. MySQL 5 gets a major thumbs up from me.