In my previous post, I went over the changes I made to Workling to add threading and allow it to run for long periods. Now, we need to deploy and monitor everything. In the Linux word, there are several options, but monit seems to be the most popular. However, I wanted to give god.rb a shot. God.rb is basically a clone of monit written in Ruby. What it’s written in doesn’t really matter to me, but setting up my config files in Ruby was interesting. That’s one more set of script commands I don’t have know.
Installing god.rb is as simple as:
sudo gem install god
The god.rb web site has some decent documentation to help you understand how to build a config file. I tend to build my config files for as much as I can using erb templates rather than manually updating the config files for each environment. I build my apache config file this way as well.
1. Download my god config to /lib/templates/god.conf. This is an example of an actual god.conf file. You will want to update it for paths and to setup the mongrels and the listeners to not run as root — not a good idea to run those as root. My apache and mysql scripts already run as a different user.
2. Now, add the following to your config/deploy.rb file:
namespace :god do
desc <<-DESC
Generate the GOD configuration. We will create the appropriate GOD
configuration stanza for the application and copy it to:
/shared/path/god.conf
Once it's there, somebody with the required permission to manage the
GOD configuration should somehow incorporate that file and restart
the GOD server. Each time the runtime configuration (ie number of
mongrels running) changes, the configuration will have to be manually
updated to match.
DESC
task :config, :roles => :app do
rails_env = fetch(:rails_env, 'production')
dispatcher_starting_port = fetch(:dispatcher_starting_port, 8000)
dispatcher_instances = fetch(:dispatcher_instances, 3)
dispatcher_ending_port = dispatcher_starting_port + dispatcher_instances - 1
dispatcher_ports = dispatcher_starting_port..dispatcher_ending_port
god_conf_path = "#{shared_path}/god.conf"
god_conf_template = File.join(File.dirname(__FILE__), '..', 'lib', 'templates', 'god.conf.erb')
begin
god_conf = ERB.new(File.read(god_conf_template)).result(binding)
put god_conf, god_conf_path, :mode => 0644
rescue Exception => e
abort "An error occurred in the GOD config generation: #{e}"
end
end
end
3. Execute cap god:config with the desired environment, and a new god.conf file will be copied to the shared directory on your server. I’m assuming you have capistrano 2.x here.
4. On your server, execute: sudo god -c /shared/path/god.conf
That’s it! Now, all your processes will be monitored by god.rb and restarted whenever there is a problem. Hold on though, we’re not completely done. You want god.rb to start up when your server boots, right? We also have to update our deploy process to use god.rb to start/stop our services now. Try to use you existing spin or spawn tasks, and god.rb will fight with you.
First, let’s get god.rb setup as a service. I found this script in the god.rb sources, and I tweaked it a bit for my system.
#!/bin/bash
# god startup script for god (like monit, only Ruby)
# Author: Dave Dupre
# Comments to support chkconfig on CentOS
# chkconfig: - 85 15
# description: god - monitor all my processes
CONF_DIR=/shared/path
LOG_DIR=/var/log
PID_DIR=/var/run/god
BIN_DIR=/usr/local/bin
RETVAL=0
# Go no further if config directory is missing.
[ -d "$CONF_DIR" ] || exit 0
case "$1" in
start)
$BIN_DIR/god -P $PID_DIR/god.pid -l $LOG_DIR/god.log -c $CONF_DIR/god.conf
RETVAL=$?
;;
stop)
$BIN_DIR/god terminate
RETVAL=$?
;;
restart)
$BIN_DIR/god terminate
$BIN_DIR/god -P $PID_DIR/god.pid -l $LOG_DIR/god.log -c $CONF_DIR/god.conf
RETVAL=$?
;;
status)
$BIN_DIR/god status
RETVAL=$?
;;
*)
echo "Usage: god {start|stop|restartstatus}"
exit 1
;;
esac
exit $RETVAL
Put the above contents into /etc/init.d/god and make it executable (sudo chmod +x /etc/init.d/god). Lastly, tell your system there is a new service in town. I use CentOS 5, so I run:
sudo chkconfig /etc/init.d/god on
Now, god.rb will start up whenever your server boots, and you can start/stop/restart it using standard service calls.
The last thing we need to do is update our deployment process to use god.rb to stop/start our processes. Add this to your deploy.rb file:
namespace :deploy do
[ :stop, :start, :restart ].each do |t|
desc "#{t.to_s.capitalize} mongrels using god"
task t, :roles => :app do
sudo "god #{t.to_s} listeners"
sudo "god #{t.to_s} mongrels"
end
end
end
namespace :starling do
[ :stop, :start, :restart ].each do |t|
desc "#{t.to_s.capitalize} starling using god"
task t, :roles => :app do
sudo "god #{t.to_s} starlings"
end
end
end
namespace :workling do
[ :stop, :start, :restart ].each do |t|
desc "#{t.to_s.capitalize} workling using god"
task t, :roles => :app do
sudo "god #{t.to_s} listeners"
end
end
end
OK! Now, we’re done. You deploy as you normally would, and I have full control over Starling and the Workling listener. Notice that there is no mongrel cluster either. God.rb started up all the instances of mongrel I needed, and it will monitor everything so there is no need for mongrel cluster anymore.
That’s it for now. You have a system that will process all your background tasks and stay running. The only thing I didn’t setup here is notifications from god.rb when there is a problem. The god.rb config settings have lots of schemes for email notifications. Take a peak at the docs and make sure god can talk to you.
Leave a Reply