Posts

Showing posts from March, 2011

Working in the multi-cloud with libcloud

I just posted my slides on "Working on the multi-cloud with libcloud" to Slideshare. It's a talk I gave at the SoCal Piggies meeting in February 2011.

ABM - "Always Be Monitoring"

What prompted this post was an incident we had in the very early hours of this past Tuesday, when we started to see a lot of packet loss, increased latency and timeouts between some of our servers hosted at a data center on the US East Coast, and some instances we have running in EC2, also in the US East region. The symptoms were increased error rates in some application calls that we were making from one back-end server cluster at the data center into another back-end cluster in EC2. These errors weren't affecting our customers too much, because all failed requests were posted to various queues and reprocessed.

There had also been network maintenance done that night within the data center, so we weren't sure initially if it's our outbound connectivity into EC2 or general inbound connectivity into EC2 that was the culprit. What was strange (and unexpected) too was that several EC2 availability zones seemed to be affected -- mostly us-east-1d, but we were also seeing increas…

What I like and don't like to see in a technical presentation

What I like to see:

Live demo of the technology/tool/process you are describing (or at least a screencast)Lessons learned -- the most interesting ones are the failuresIf you're presenting something you created:compare and contrast it with existing solutions convince me you're not suffering from the NIH syndromeconvince me your creation was born out of necessity, ideally from issues you needed to solve in productionHard data (charts/dashboards)Balance between being too shallow and going too deep when covering your topickeep in mind both the HOW and the WHY of the topicGoing above and beyond the information I can obtain with a simple Google search for that topicPointers to any tools/resources you reference (GitHub pages preferred)
What I don't like to see:

Cute slides with images and only a couple of words (unless you provide generous slide notes in some form)Humor is fine, but not if it's all there isHand-waving / chest-poundingVaporwareNo knowledge of existing solutions w…

Deployment and hosting open space at PyCon

One of the most interesting events for me this year at PyCon was an Open Space session organized by Nate Aune on deployment, hosting and configuration management. The session was very well attended, and it included representatives of a large range of companies. Here are some of them, if memory serves well: Disqus, NASA, Opscode, DjangoZoom, Eucalyptus, ep.io, Gondor, Whiskey Media ... and many more that I wish I could remember (if you were there and want to add anything, please leave a comment here).

Here are some things my tired brain remembers from the discussions we had:

everybody seems to be using virtualenv when deploying their Python applicationseverybody seems to be using Fabric in one way or another to push changes to remote nodesthe participants seemed to be split almost equally between Puppet and Chef for provisioningthe more disciplined of the companies (ep.io for example) use Puppet/Chef both for provisioning and application deployment and configuration (ep.io still uses Fa…

Monitoring is for ops what testing is for dev

Image
Devops. It's the new buzzword. Go to any tech conference these days and you're sure to find an expert panel on the 'what' and 'why' of devops. These panels tend to be light on the 'how', because that's where the rubber meets the road. I tried to give a step-by-step description of how you can become a Ninja Rockstar Internet Samurai devops in my blog post on 'How to whip your infrastructure into shape'.

Here I just want to say that I am struck by the parallels that exist between the activities of developer testing and operations monitoring. It's not a new idea by any means, but it's been growing on me recently.

Test-infected vs. monitoring-infected

Good developers are test-infected. It doesn't matter too much whether they write tests before or after writing their code -- what matters is that they do write those tests as soon as possible, and that they don't consider their code 'done' until it has a comprehensive suite o…