Showing posts from July, 2010

Bootstrapping EC2 instances with Chef

This is the third installment of my Chef post series (read the first and the second). This time I'll show how to use the Ubuntu EC2 instance bootstrap mechanism in conjunction with Chef and have the instance configure itself at launch time. I had a similar post last year, in which I was accomplishing a similar thing with puppet.

Why Chef this time, you ask? Although I am a Python guy, I prefer learning a smattering of Ruby rather than a proprietary DSL for configuration management. Also, when I upgraded my EC2 instances to the latest Ubuntu Lucid AMIs, puppet stopped working, so I was almost forced to look into Chef -- and I've liked what I've seen so far. I don't want to bad-mouth puppet though, I recommend you look into both if you need a good configuration management/deployment tool.

Here is a high-level view of the bootstrapping procedure I'm using:

1) You create Chef roles and tie them to cookbooks and recipes that you want executed on machines which will be asso…

Tracking and visualizing mail logs with MongoDB and gviz_api

To me, nothing beats a nice dashboard for keeping track of how your infrastructure and your application are doing. At Evite, sending mail is a core part of our business. One thing we need to ensure is that our mail servers are busily humming away, sending mail out to our users. To this end, I built a quick outgoing email tracking tool using MongoDB and pymongo, and I also put together a dashboard visualization of that data using the Google Visualization API via the gviz_api Python module.

Tracking outgoing email from the mail logs with pymongo

Mail logs are sent to a centralized syslog. I have a simple Python script that tails the common mail log file every 5 minutes, counts the lines that conform to a specific regular expression (looking for a specific msgid pattern), then inserts that count into a MongoDB database. Here's the snippet of code that does that:

import datetime
from pymongo import Connection

conn = Connection(host="")
db = conn.logs
maillogs = db…

Working with Chef cookbooks and roles

Welcome to the second installment of my Chef saga (you can read the first one here). This time I will walk you through creating your own cookbook, modifying an existing cookbook, creating a role and adding a client machine to that role. As usual, I got much help from the good people on the #chef IRC channel, especially the omnipresent @kallistec. All these tasks are also documented in one form or another on the Chef wiki, but I found it hard to put them all together, hence this blog post.

Downloading the Opscode Chef cookbooks

Step 0 for this task is to actually clone the Opscode repository. I created a directory called /srv/chef/repos on my Chef server box and ran this command inside it:

# git clone git://

This will create /srv/chef/repos/chef-repo with a bunch of sub-directories, one of them being cookbooks. I deleted the cookbooks directory and cloned the Opscode cookbooks in its place:

# cd /srv/chef/repos/chef-repo
# git clone git://…

Chef installation and minimal configuration

I started to play with Chef the other day. The instructions on the wiki are a bit confusing, but help on twitter (thanks @jtimberman) and on the #chef IRC channel (thanks @kallistec) has been great. I am at the very minimal stage of having a chef client talking to a chef server. I hasten to write down what I've done so far, both for my own sake and for others who might want to do the same. My OS is Ubuntu 10.04 32-bit on both machines.

First of all: as the chef wiki says, make sure you have FQDNs correctly set up on both client and server, and that they can ping each other at a minimum using the FQDN. I added the FQDN to the local IP address line in /etc/hosts, so that 'hostname -f' returned the FQDN correctly. In what follows, my Chef server machine is called and my Chef client machine is called

Installing the Chef server

Here I went the Ruby Gems route, because the very latest Chef (0.9.4) had not been captured in the Ubuntu packages yet…