Make your test suite UNCOMFORTABLY FAST!
Posted by Jason Morrison
Jul 24
Are your test suites slow? Do you continually refactor your test suites so can get your TATFT on, but still have slow, slow tests? Do you isolate your tests from the database where possible to reduce disk I/O but find yourself mocked by the long runtime of CPU-bound suites?
Maybe it’s time for you to install… PARALLEL SPECS! Originally written by Michael Grosser to speed up RSpec suites, which is awesome, parallel_specs now sports support for running Test::Unit suites. (Yes, that means Shoulda too!)
Remember all that excitement from years ago about multi-core processors? “You can burn a CD while you check your email!” “You can email pictures TWICE AS FAST!!” Well, hold onto your butts, because – yep, that’s right – you can use them to run your tests faster! See for yourself!
This is what your two cores looks like when they’re BOTH RUNNING TESTS AT THE SAME TIME!
Now how about some numbers!
That’s a 30% speedup – REAL SPEEDUP! 30% to 40% was typical for our projects. Think your tests are largely I/O bound due to extreme database interaction and that using two cores won’t speed it up? THINK AGAIN! Our tests were running so fast that the database was like “SLOWWWW DOWWWWWN!”
Installation
Do you use RSpec? Do you use Test::Unit? Then you can get in on the action! FAST TESTING ACTION! Fast tests need a fast installation. Get parallel_specs up and running in no time flat – here’s how I did it for our existing Test::Unit applications:
Baseline benchmarking
Let’s see how slow our tests are when they are run one! at! a! time!
1 2 3 4 5 6 7 |
$ time rake test:units test:functionals Started .......... (abbreviated... FOR BLOG READING SPEED!!) 415 tests, 898 assertions, 0 failures, 0 errors 29.14s user 3.74s system 87% cpu 37.614 total |
Plugging in
Are you ready? Let’s install a plugin! IN OUR APP!
$ script/plugin install git://github.com/jasonm/parallel_specs.git |
More tests need MORE DATABASES!
More databases… it’s what concurrent environments crave!
1 2 3 4 5 6 7 8 9 10 |
$ rake db:structure:dump
$ echo "create database myapp_test2;" | mysql -uroot
$ mysql -uroot -Dmyapp_test2 < db/development_structure.sql
$ vim config/database.yml # add some ERB to config/database.yml
test:
adapter: mysql
encoding: utf8
database: myapp_test<%= ENV['TEST_ENV_NUMBER'] %>
username: root
|
Now it’s time for testing!
POWER TESTING!!
1 2 3 4 5 6 7 8 |
$ time rake test:parallel (in /Users/jasonmorrison/dev/myapp/trunk) 2 processes for 49 tests, ~ 24 tests per process Loaded suite -e Started .................................. (abbreviated) 29.32s user 3.31s system 143% cpu 22.669 total |
BLAZOW! That’s like 15 more seconds I can spend working out. OR REFACTORING!
USING ALL AVAILABLE HARDWARE, I CAN DELIVER BUSINESS VALUE† FASTER!! USE PARALLEL_SPECS!!
Fine print
parallel_specs does not run cucumber features in parallel. You might look at testjour which is aimed at running cucumber features, although it is intended to run them in a distributed fashion across multiple machines, rather than on one multicore machine.
If you have multiple disks, spreading databases out across them will likely yield even more performance.
The solution isn’t originally mine – the original idea made its way to me from Pivotal Labs via Michael Grosser. Currently, I’ve forked Michael’s code to add Test::Unit support, and plan to send a pull request after I clean up some of the duplication I added.
† When bears say “deliver business value,” they mean eating your whole campsite.
Do not drink parallel_specs.
Comments on this post
Jul 24
Derek said,
This is the best blog post ever written.
Jul 24
Dan Pickett said,
You had me convinced that parallel_specs rocks immediately after I saw the picture of the bear with a tommy gun surfing a shark. You, sir, are a skilled and persuasive writer.
Jul 24
Brian Cardarella said,
Before now I had no idea just how empty my life was without knowing about bears with chainsaws for arms. Thank you.
Jul 24
Jason Morrison said,
Thanks, Dan! I must admit, though, that I have gotten where I am today by standing on the shoulders of giants. I would like to acknowledge Mike Burns for his vast array of subtle and witty writing techniques (e.g. leverging his vasty deep of internet pictures).
Jul 24
Mathieu Mathieu Martin said,
Can I leverage all this bear chainsaws awesomeness with autotest? Cuz that would be equivalent of having a cavalry of these bears at my disposal. The bugs would be pretty much fucked! http://img16.yfrog.com/i/84n.jpg/
Jul 24
Tim Connor said,
Do the rake db:test tasks work?
Jul 24
Jason Seifer said,
TURBOPUNS
Jul 24
Nick G said,
How does it multi-core the tests? Are the split before execution (into groups of similar size) or are the tests delegated on the fly as they are completed?
We have a similar solution here but since they are split, it doesn’t use all the cores for the duration of the process.
Jul 24
Josh Adams said,
What about rake db:test:clone? Does that do anything like working?
Jul 24
max said,
This is insanely awesome! kudos for this.
Now I have less time to drink coffee while running tests =(
Jul 24
Phill Kenoyer said,
parallel_specs feels like having sex with TWO tractor-trailers in a parking lot AT THE SAME TIME! It will make you WIN AT TESTING! You will be KICKING EVERYONES ASS ALL THE TIME!
Jul 25
madlep said,
What? No bears with nukes?
Anyway, current project rspec time went from 1m:06s to 42s. Nice :D
Jul 25
Tim Connor said,
Josh, check the docs on github, for the original project. It covers the parallel prepare rake tasks, for dealing with the additional db[s].
Jul 25
Adam Bair said,
ABNORMALLY FAST
Jul 25
Brent Collier said,
But will it help me run as fast as Kenyans, or have 400 babies?
Jul 26
CLR said,
I’m still not convinced. What can this software do for me? Will it increase my personal brand?
Also, I find it very odd that DBs can’t handle concurrent transactions in such a way that adding additional DBs actually helps. Is that just a stupid MySQL thing, or does it benefit real databases like PostgreSQL as well? I’ll give it a spin w/Postgres and get back to you.
Jul 27
Cássio Marques said,
Nice work!
However I’m having some trouble here! I’m trying to run parallel_specs with a postgresql database, with Rspec.
When I run rake spec:parallel, the first half of my specs runs ok, then the second half runs and when it ends… it just freezes after the results are shown, but without giving me the prompt back.
Do u know what may be causing it?
I tried to run it with—trace to see if there is something else, but go no other output.
Thank you very much!
Jul 27
Nate said,
If you are IO bound, wonder how the performance would be if you moved your test dbs onto a ramdisk. Pretty easy to create in nix environments.
Jul 28
grosser said,
Thanks for that great post, ill merge in those changes asap :D
Jul 28
Alex Pooley said,
In the results of your baseline test you get this:
In the results when running with parallel_spec you get this:
Can the bear put down it’s chain saws for a second and explain why these numbers don’t correlate? I get similar results on my set of tests.
Jul 28
Alex Pooley said,
In the results of your baseline test you get this:
In the results when running with parallel_spec you get this:
Can the bear put down it’s chain saws for a second and explain why these numbers don’t correlate? I get similar results on my set of tests.
Jul 29
Dan Manges said,
There’s also DeepTest: http://github.com/qxjit/deep-test/tree/master
It works with test/unit and RSpec. In addition to running tests in parallel to take advantage of multiple cores, it can run tests distributed to multiple computers.
Jul 29
Dan Manges said,
There’s also DeepTest: http://github.com/qxjit/deep-test/tree/master
It works with test/unit and RSpec. In addition to running tests in parallel to take advantage of multiple cores, it can run tests distributed to multiple computers.
Add a comment
© 2000 - 2009 by thoughtbot, inc.
written by a bushel of tiny robots
Come “ride the toad” on Hoptoad, the app error app.
Thunder Thimble: Brand monitoring for social media.
Widgetfinger: Simple content management for simple websites.
Tee-Bot, funny shirts your friends won't understand!
Umbrella Today: “It’s like totally the simplest weather report ever, Julie.”
Thoughtbot
thoughtbot is a technology consulting firm that provides web application development and design services. We focus on building modern systems, embracing good ideas and delivering elegant solutions.
Interested in learning Rails?
Sign up for our beginning or advanced training.
Archives