40withegg

I discovered something interesting about Ruby on Rails fixtures. Fixtures, for those who don't know, are how Rails loads test data for unit and functional testing. YAML files are loaded with sample data, which is loaded into your database when each test executes. If you want to load "users" and "addresses" data, you indicate such by stating "fixtures :users, :addresses" in your tests. Pretty handy!

I would expect this data to be cleared from the database at test teardown time, leaving the DB in a pristine state for the next test but this is not the case. Instead, the target fixture's data is deleted, then reloaded at test setup time, while teardown does not clear this data, leaving schmutz -- left over test data.

When it comes to test data, I have always practiced a "leave no trace" philosophy, so my inclination is to disagree with how the Rails folks implemented this: teardown should not leave schmutz.

And before you say anything, transactional fixtures don't fix this. More on that later.

Let's say we're running a test suite consisting of UserTest and UploadTest, each with a bunch of test methods. UserTest loads "fixtures :users, :addresses" while UploadTest loads"fixtures :files". If UserTest executes first, it's last test method will reload users and address, potentially doing a bunch of stuff with those, and then leave it behind. There will be users and addresses polluting the database, potentially interfering with UploadTest, or not, or sometimes, or never or... who knows? Schmutz! Schmutz!

I have heard some arguments for what I will affectionately term the "schmutzy philosophy":

• If you leave schmutz behind, you can look at the database and see the affects your test had upon the data.
• If schmutz messes up future tests, it is actually exposing an untested area of the system.
• In the real world, functionality does not exist in a vacuum, so it's better to test with schmutz around.

And my favorite ...

• Clearing all the test data is hard. Eventually you load tons of it, and your tests create a bunch more, then they mangle it, and you can't keep track of it all, and so we give up.

None of these arguments are strong enough for me to embrace the schmutzy testing philosophy. Unit testing in isolation has more benefits than any random side effects schmutz might reveal. And as test suites grow and change, schmutz's side affects will be constantly changing, and often muddle the true intent of your unit tests -- I've seen it many times before. Are the features of "save user preferences" truly broken, or is there just a bunch of random junk in the database messing up the results? You won't be able to see the true bugs from the side effect problems.

Ok, now for transactional fixtures. From Clarkware:

Transactional fixtures use database transactions to isolate tests. Rather than deleting and re-inserting fixtures for each test method, transactional fixtures are loaded once at the beginning of the test case. The fixture data in the test database is restored to its original state after each test by doing a transaction rollback.

This still leaves schmutz! It just leaves clean schmutz, which is kind of like clean dirt -- not clean. It seems like transactional fixtures could have solved this problem: perform everything within the transaction: the data inserts, the test's data manipulations, everything, and perform a ROLLBACK in teardown, thus rolling back the inserts as well. Transactional fixtures focus on performance, not on clean tests.

What are your opinions? Does testing with a schmutzy database have merits?

3 Comments (from old blog):

At 1/10/2006 12:01 PM, Chris Nolan.ca said…

I have no problem with it. I don't do anything with the test db except run the tests. Each of my tests have the appropriate fixtures loaded so it doesn't matter to me what is left behind.

At 8/23/2006 5:47 AM, James Mead said…

I totally agree with you, Joseph. In fact this is one of the many reasons we are avoiding the use of fixtures altogether. You might find Mocha gives you another way to approach unit testing.

At 8/31/2006 8:20 AM, Casey Helbling said…

Also be aware of using fixtures, acts_as_enumerated, observers and an enumerated type in an association. You can run into problems when the enumeration fixture data isnt loaded yet but the observer loads the model with the association (trying to use the enum). This is terribly annoying and even loading all fixtures in your test case wont fix it. And to Chris's point, I too load individual fixtures in each test but it is sometimes dificult to know if you add a relationship that depends on a new fixture what other fixtures are required. The arguement could be made that the testing isnt being done at the right granularity but that is different story. Also be aware of problems with rails transactions in general and save points.

http://dev.rubyonrails.org/ticket/5457

Transactions in general in rails do NOT behave as you expect!