Test dependency detection in Tack

Tack is a Ruby interface gem I’ve been working on. It’s like Rack for testing: it provides a single API for querying and running tests, regardless of what framework you write your tests in (well, as long as you use one of the supported frameworks - currently RSpec, Test::Unit, and Shoulda).

Also, like Rack, it supports middleware. Tack middleware is just any object that implements two methods: ‘run_suite’ and ‘run_test’. Here’s some example middleware that conforms to the spec (although it doesn’t actually do anything useful in this case).

Using the middleware API, I was able to quickly write some features that help with a common problem: test dependency detection.

As everyone knows, test dependencies are a bad thing. If running one test can affect the results of another test, then you can have strange and difficult-to-diagnose test failures if you move, delete or change tests (or even just run them in a different order).

Let’s say that you have a test suite that you suspect contains a test dependency. One simple way to potentially detect it is to run the tests in reverse. Here’s some middleware that does just that:

Well, that was easy. Another strategy would be to run the tests in a random order and see if they still pass or fail as expected. That’s almost as simple:

Or let’s say you’ve got a failing test and you suspect that one test is polluting your Ruby object space somehow (say, monkey-patching a common class in a way that breaks another test). Solution: use #fork to run each test in a separate process (thanks goes to Ryan Tomayko for his awesome Shotgun gem, from which I took the forking code):

Of course, because these features are written as Tack middlewares, these features just work with test suites written in any of the supported test frameworks.