UPDATE: This advice is a bit outdated and only applied back when I was hosting Seaside on Windows servers (which I advise against), see Scaling Seaside Redux Enter the Penguin for more up to date advice.

I've been busy with non Seaside projects lately, but one of the things I have squeezed in was a bit of configuration to make Seaside scale better. I was having some performance problems when more than a few sessions were running concurrently and after discussing the issue on the Seaside developers list, Avi popped in from DabbleDB and told us the way to scale was load balance many VMs with a few sessions running on each. So I had to read up a bit on the issue and learn what to do.

It turns out one of the best ways to learn about scaling Seaside is to read about scaling Ruby on Rails. The architecture for scaling both is pretty much the same. Ruby developers use a web server called Mongrel, a light weight single threaded server that works well with Rails but isn't a heavy duty web server like Apache. This is much the same position Seaside is in with Comanche, though not single threaded, the Squeak VM can't take advantage of multiple processors and doesn't do well with too many concurrent connections.

UPDATE: I neglected to mention one major requirement of Seaside, your load balancer must support sticky sessions. Seaside uses sessions heavily and does not support the shared nothing approach where every request can hit a different server. This isn't an issue unique to Seaside, many frameworks use sessions and must deal with this. Other frameworks handle such issues either by sticking a session to a server, or by having a shared session cache that all the servers can access, such as memcached or a sql server. Currently, to the best of my knowledge, no one has externalized Seaside sessions in this manner, so sticky sessions is the only viable approach.

The solution for both is quite simple really, setup a heavier duty web server/load balancer (Apache/LiteHttpd) to serve up static content, and load balance and proxy connections to a farm of light weight application servers (Mongrel/Comanche) running on other ports. I'm actually using an F5 as my front end load balancer, but Apache has all the necessary features including the ability to create pools of virtual servers which it will load balance requests across with its new mod_proxy.

One only need Google scaling Rails to find many examples of detailed setup information and articles for such a setup, so I won't repeat it here. I will mention that neither Mongrel or Comanche are anywhere near as rock solid stable as Apache, so one thing you'll want when having a setup like this to ensure maximum uptime is a process running to poll all of your servers to ensure they aren't hung up for any reason.

UPDATE: I don't want to imply Comanche is unstable, it is very rare that a service needs reset, I only do this because it "can" happen, not because it happens a lot. Seaside is very stable and under normal conditions, doesn't crash.

Here's a little bash script I found somewhere that makes checking a site for a specific response string simple and easy to use from the command line allowing you to easily schedule some scripts to reset any hung processes. Oh, I use cygwin on all my servers so I can have a decent Unix command line on my Windows servers.

checkSite.sh UPDATE: This script eventually freaks out Windows and causes random network errors because lynx somehow eats up network resources. Don't use it, Seaside is quite stable without it.

#!/bin/bash if [ $# -lt 2 ]; then exit fi URL=$1 findText=$2 lynx -dump -error_file=/tmp/x$$ $URL 2>/dev/null | grep "$findText" &>/dev/null if [ $? != 0 ]; then echo WARNING: Specified search text was not found fi stcode=`awk '/STATUS/{print $2}' /tmp/x$$ 2>/dev/null` if [ $? != 0 ]; then echo site is down fi for code in $stcode do case $code in 200) echo OK;; 302) echo redirecting awk -F/ '/URL/{print " ",$3}' /tmp/x$$;; *) echo $code esac done if [ -f /tmp/x$$ ]; then rm /tmp/x$$ fi

Then I use this in another script built specifically to monitor instances of my Seaside app. Though this could be more generic, I haven't bothered yet because I only have one Seaside site in production to worry about.

checkSeaside.sh

echo "$1 $2 $3..." sh checkSite.sh http://$1:$3/seaside/someApp "Some Required Text" | grep WARNING &>/dev/null if [ $? = 0 ]; then echo "restarting $2 on $1" psservice \\\\$1 restart "$2" >/dev/null #NT util to restart services on remove machines echo "Restarting $1 $2" | wsendmail -- -CGI -Ssome.mail.server.com \ -s"App Monitor reset $1 $2" someEmail@someAddress.com -Fmonitor@someAddress.com -P1 fi

Then on each web server, I'm running a pool of 10 instances of Seaside setup as services and ensuring they're up by scheduling a simple batch file with the windows task scheduler.

monitorSomeApp.cmd

@echo off bash checkSeaside.sh serverName "Some Service1" 3001 bash checkSeaside.sh serverName "Some Service2" 3002 bash checkSeaside.sh serverName "Some Service3" 3003 bash checkSeaside.sh serverName "Some Service4" 3004 bash checkSeaside.sh serverName "Some Service5" 3005 bash checkSeaside.sh serverName "Some Service6" 3006 bash checkSeaside.sh serverName "Some Service7" 3007 bash checkSeaside.sh serverName "Some Service8" 3008 bash checkSeaside.sh serverName "Some Service9" 3009 bash checkSeaside.sh serverName "Some Service10" 3010

Yea, I'm mixing and matching bash and dos scripts, so sue me! Anyway, the setup works great, I'm running 30 instances of Squeak across 3 servers and these scrips ensure they're always up and responding, and reset them and email me if they go down for any reason. Response time is now much better and I can fully take advantage of the multiple processors on the web boxes.

My process isn't nearly as fancy as Avi's (he's dynamically bringing images up and down based on the host header), but balancing the connections across a bunch of fixed sized pools works well. I started with 10 processes per box, just for the hell of it, but I'll increase or decrease the size of the pool as load dictates to eventually find the sweet spot for the pool size. For now, 10 per box works, I've got plenty of spare ram.

Of course, doing a setup like this means you'll need to automate your deployment process for new code as well. So far this mean keeping a master image I upgrade with new code and test, then a script to take down each process, copy the image file over the old one, and bring the process back up. Seems Avi's doing the same thing, works pretty well so far.