Showing posts from March, 2006

Running the Selenium Twisted server on Linux

A recent issue that surfaced on the selenium-users mailing list was related to running Selenium in "Driven Mode" (soon to be dubbed Remote Control or RC mode) by using the Twisted-based Selenium server on Linux. Normand Savard had followed my initial instructions, but was getting CGI "premature end of script" errors. I tried to follow my own instructions using the latest Selenium code and I got the same error. I managed to get it to work, so I'm posting the steps I went through to solve it.

My setup was a Linux box running Ubuntu Breezy with Python 2.4.2.

1. Download and install Twisted version 1.3 (this is important, because things have been known not to work so well with Twisted 2.x). Old versions of Twisted, including 1.3, can be downloaded from here.

2. Check out the Selenium source code:

svn co selenium

3. Let's assume you checked out the code in ~/proj/selenium. You need to copy the contents of ~/proj/selenium/code/…

Bunch O'Links on agile/testing topics

Titus suggested I post some of the links I keep sending him, so here they are:

Darren Hobbs talks about "Why a 20 minute build is 18 minutes too long". The build in question included acceptance tests. His conclusion is that I/O is the enemy. My take on it is that a continuous build system should run all the unit tests, plus a subset of acceptance tests that are known to run fast. The other acceptance tests, especially the ones that do a lot of I/O, can be run more leisurely, once every X hours.

Michael Hunter, aka The Braidy Tester, posts his impressions on Uncle Bob's SDWest talk on Agility and Architecture. I particularly enjoyed this point: "The first, most important part of making code flexible is to surround it with tests."

Tim Ottinger (from ButUncleBob's crew) blogs on his experience with Testing Hypothetically. It's all about seeing the big picture when writing acceptance tests, and not being bogged down in the implementation-specific details that …

Ajax testing with Selenium using waitForCondition

An often-asked question on the selenium-users mailing list is how to test Ajax-specific functionality with Selenium. The problem with Ajax testing is that the HTML page under test is modified asynchronously, so a plain Selenium assert or verify command might very well fail because the element being tested has not been created yet by the Ajax call. A quick-and-dirty solution is to put a pause command before the assert, but this is error-prone, since the pause might be not sufficient on a slow machine, while being unnecessarily slow on a faster one.

A better solution is to use Dan Fabulich's waitForCondition extension. But first, a word about Selenium extensions.

If you've never installed a Selenium extension, it's actually pretty easy. You should have a file called user-extensions.js.sample in the same directory where you installed the other core Selenium files (such as TestRunner.html and selenium-api.js). You need to rename that file as user-extensions.js, so that it will b…

Elizabeth Hendrickson on "Better testing, worse testing"

I've always enjoyed Elisabeth Hendrickson's articles and blog posts on testing. Her latest post, "Better Testing, Worse Testing", confirms one of the two main conclusions Titus and I reached while working on our PyCon Agile Testing tutorial. Here's an excerpt from the Wrap-up section of our tutorial handout notes:
Wrap up & ConclusionsHolistic testing is the way to go. No one test type does it all...unit testing of basic codeFunctional/acceptance testing of business/user logicregression testing
That's pretty much what Elisabeth says in her post, with war stories that prove her point. I particularly liked the following "lesson learned" -- and I quote: "This leads me to my next general conclusion: an isolated improvement in one aspect of a development process tends to be offset by declines in another, resulting in no overall improvement in the final result. So how do we improve results? By paying attention to the whole process and not just …

PTTT updates

Several people updated the Python Testing Tools Taxonomy Wiki page in the last couple of months. Here are some tools that were added:
pywinauto (added by its author Mark McMahon): "Simple Windows (NT/2K/XP) GUI automation with Python. There are tests included for Localization testing but there is no limitation to this. Most of the code at the moment is for recovering information from Windows windows and performing actions on those controls. The idea is to have high level methods for standard controls rather then rely on Sending keystrokes to the applications."svnmock (added by its author Collin Winter): "enables easier testing of Python programs that make use of Subversion's Python bindings."testosterone (added by its author Chad Whitacre): "an interface for running tests written with the Python standard library's unittest module. It delivers summary and detail reports on TestCases discovered in module-space, via both a command-line and a cursesinterfac…

Nuxeo mentioned in article on Eclipse

I was glad to see Nuxeo mentioned in a article on Eclipse. Nuxeo offers an open-source content management product called CPS, which uses Zope. The reason I know about them is that I've been in touch with several people who work there and who are very interested in Python testing tools and techniques. As a matter of fact, they released a functional/performance testing tool written in Python, called FunkLoad. Plus they're early adopters of Cheesecake :-)

They still teach Waterfall in schools

The last few days I've interviewed some candidates for an entry-level QA position at the company where I work . All of them were fresh graduates of local universities, some from the University of California system, some from the Cal State system. All of them had something in common though: they very seriously explained to me how they took classes in "Software Development Lifecycle" and how they worked on toy projects, first obtaining requirements, then designing, then implementing, then at the very end, if they had time, doing some manual testing. Of course, no word of automated testing, iterations, or other agile concepts. At least they used source control (CVS).

One guy in particular told me kind of proudly that he knows all about the Waterfall methodology. He said they spent a lot of time writing design documents, and since they *only* had one semester for the whole project, they almost didn't get to code at all. I couldn't help laughing at that point, and I to…

Buildbot Technology Narrative

Titus just put the finishing touches on a very fast-paced and entertaining Buildbot Technology Narrative page on the Agile Testing wiki we used for our PyCon tutorial. Highly recommended if you're curious to see how a continuous integration tool such as buildbot can be used to...well, to integrate all the types of automated tests that we used in our tutorial.

Comments/suggestions appreciated. Please leave them here or send them via email to titus at or grig at

"Articles and Tutorials" page updated

I decided to move the "Articles and Tutorials" page to a different server, since it's annoying to publish it here every time I make a change to it. I'll probably host it here again when Blogger decides to implement categories...

Running buildbot on various platforms

I started to run a bunch of buildbot slaves at work, on various platforms. I encountered some issues that I want to document here for future reference. The version of buildbot I used in all scenarios was 0.7.2. The buildbot master is running on a RHEL3 server. Note that I'm not going to talk about the general buildbot setup -- if you need guidance in configuring buildbot, read this post of mine.

Before I discuss platform-specific issues, I want to mention the issue of timeouts. If you want to run a command that takes a long time on the buildbot slave, you need to increase the default timeout (which is 1200 sec. = 20 min.) for the ShellCommand definitions in the buildmaster's master.cfg file -- otherwise, the master will mark that command as failed after the timeout expires. To modify the default timeout, simply add a keyword argument such as timeout=3600 to the ShellCommand (or derived class) instantiation in master.cfg. I have for example this line in the builders section of m…

Mind mapping

I first encountered the concept of mind maps when I read the 1st edition of "Extreme programming explained" by Kent Beck. I've wanted for a long time to delve more into this subject, but it's only now that I got around to it by reading "Mind maps at work" by Tony Buzan, the inventor of mind maps.

To put it briefly, a mind map is a graph-like structure that starts with a central image of the concept you're trying to focus on. You then start drawing branches radiating out from the center, and on each branch you write some quality or feature of the central concept. Then you draw smaller branches, twigs if you want, with more and more details pertaining to each branch. Since a picture is worth a thousand words, here's a mind map I drew while brainstorming about ways to approach automated testing (apologies for the poor quality of my handwriting):

The beauty of this is that I was able to summarize in a single page almost all the concepts that Titus and I p…

Remote Web app testing with Selenium IDE

If you've been using standard -- aka "table-mode" -- Selenium to test your Web application, you know that the static Selenium files, as well as your custom test files and suites, need to be deployed under the root of your Web application, on the same server that hosts the app. This is because Selenium uses JavaScript, and JavaScript has a built-in security limitation to prevent cross-site scripting attacks against Web servers.

The good news is that if you're using Firefox, you're in luck -- you can pretty much bypass this limitation thanks to the wonderful Firefox extension called Selenium IDE (used to be called Selenium Recorder.) I talked about the Selenium Recorder and other useful Firefox extensions in a previous post. What I want to show here is how easy it is to write Selenium tests, even against 3rd party applications, by using Selenium IDE.

I'll assume you already installed Selenium IDE. You need to launch it via the Firefox Tools menu. It will open a n…

Should acceptance tests talk to the database?

There's an ongoing discussion on the FitNesse mailing list about whether acceptance tests should talk to a "real" database or should instead use mock objects to test the business rules. To me, acceptance tests, especially the ones written in Fit/FitNesse, are customer-facing (a term coined I believe by Brian Marick), and hence they need to give customers a warm and fuzzy feeling that they're actually proving something about the system as a whole. The closer the environment is to the customer's production environment, the better in this case. Here's my initial message to the list:

I would say that since you're doing *acceptance* testing, it doesn't make too much sense to mock the database. Mocking makes more sense in a unit testing situation. But for acceptance testing, the way I see it is to mimic the customer environment as much as possible, and that involves setting up a test database which is as similar as possible to the production database. Of cou…