Running Selenium Javascript Tests Through Docker Containers
Just writing the title to this blog post makes me shiver. It makes me shiver because I spent 6 hours on a lovely sunny afternoon in London, punishing myself in a coffee shop for being unable to test my Javascript effectively. It got pretty intense in that coffee shop. At one point a tense exchange with an employee occurred:
Employee: “Hey there, can I get you anything el..”
Me: “I just want this f*****g thing to work, can you do that for me!”
Ok that exchange didn’t actually happen, but it’s representative of the stress I was going through to get my tests running. Attempting to learn online about Selenium browser testing also presents you with additional headaches. If you were to ask 10 developers how they did this, you’ll get 137 different solutions. It seems there is no agreed upon standard on how to do this stuff, and every developer seems to require a different setup. The lack of cohesiveness in the world of browser testing makes this an incredibly tough challenge. But a worthwhile one!
I’ve been working on a side project lately, which has only just recently introduced Javascript to the party. Test coverage up until now has been 99.92% with a lot of care taken to avoid test mutants in the process. Now with Javascript playing a role in user interactions, it was time to rethink how I went about testing.
Previously, when users would submit an enquiry they would have to go to a separate page with a separate form to fill out. On successful submission of this form, they would encounter a redirect. All in all, this is very easy to test because you can just make simple assertions in Capybara to say
expect(page).to have_content("Enquiry submitted. Thank you")and your test suite would confirm the user made the enquiry successfully and all is well with the world.
The game starts to change when you want to have an “in-page enquiry” feature, with an AJAX-ified form that submits asynchronously and keeps the user on the same page. It’s time to bring in the Javascript, and I knew things were about to get messy.
As I move forward in my developer career, I shudder at the thought of not writing tests. I used to be very relaxed about the topic, until I concluded from personal experience that the amount of time saved in not writing tests was exceeded by the amount of time wasted debugging and fixing seemingly endless amounts of bugs; on top of the many manual steps one has to go through in the browser to validate their code does the right thing.
I completely understand why inexperienced developers and those just entering the industry will avoid writing tests. It takes some pain to realise that giving up some time in the short term (in the form of writing good, quality rock solid tests) will save you a lot more time in the long term, as well as allowing you to sleep peacefully at night.
Running RSpec / Capybara tests through Docker containers so far had been a dream. But now with Javascript playing a role, it’s time to change strategy.
The world of browser testing isn’t for the faint hearted. It requires bringing together a range of tools to carry out the job, from a test runner like RSpec to a web testing framework like Capybara, to a browser automation tool like Selenium. On top of this, running tests through Docker containers and ensuring they can interact with each other adds another layer of up front complexity for the end goal of reproducibility and portability.
My docker-compose file initially looked like this with a very simple Ruby Docker image to run the tests from.
The main change to get the current system (otherwise known as ‘feature’) tests working was to set a driver in the rails_helper. Rack Test is the fastest and most popular for Rails system tests as it doesn’t need an actual browser to perform the tests. It’s super fast (at least compared to Selenium) as it makes requests at the Rack level, and requires no dependencies as opposed to Poltergeist which requires PhantomJS or Capybara::Webkit which requires a QtWebkit to be installed. For this reason, Rack Test has served me well, and required barely anything in terms of set up as it ships with Capybara. The only change required is to specify the driver as Capybara will attempt to use selenium-webdriver by default. To erase this error message Capybara's selenium driver is unable to load 'selenium-webdriver', please install the gem and add "gem 'selenium-webdriver'" to your Gemfile if you are using bundler , just add the below block to your rails_helper. This error message sucks I feel, because they’re forcing users to install a gem (which in addition will require more installation overhead), when a simple block of configuration is all that’s needed. Shame on you, Capybara!
Now that the user could submit an in page enquiry, with no redirect to another page on submission, the system test did not reflect the true state of the application and how users interacted with it. It was time to add the js: :true flag to the RSpec configuration to allow us to specify “this test requires you to drive this interaction through the browser instead of through a Rack interface”.
In the spirit of keeping everything running through containers, I added a new service to my docker-compose.yaml.
This uses a Selenium image, which uses a real browser window which executes commands against a real Javascript engine. This is very powerful stuff, as now we’re bringing our tests one step closer to how users interact with our application. Users don’t use Rack Test and make requests through a REST client; they use browsers and browsers only. Using Selenium delivers an extra comforting layer of confidence which Rack Test cannot offer. The downside is it’s much more difficult to use, it’s slower and requires more dependencies. On the plus side, you’ve just automated a lot of your Q&A.
There are a lot of benefits to running Selenium tests through Docker containers. The most obvious one is portability and reproducibility. Getting Selenium set up locally on multiple different environments is a pain, with a notorious level of installation overhead. Discrepancies between operating environments lead to inconsistent behaviour, and developers often find themselves having to install the world just to get something running. Docker makes set up extremely comforting and provides ease of set up unrivalled on any other platform.
Now we have a Selenium Docker image to work with, we need to register a driver with Capybara. This instructs Capybara to connect to the remote browser in the Selenium instance we’ll be running in another container with a separate IP.
To register the driver, you need to pass an initial argument which will either be app or if not required, can be anything you want (in my case I passed nil). app in this case is a type of Rack::Builder that can be used to construct a Rack application for the driver to use. After trying to figure out what this was and why every instruction online told me to use it (with zero explanation as to why), I decided to not pass it through because it requires further explanation; it was nothing but a distraction.
A url is required to tell Capybara where the Selenium hub is being run; the hub being the central point in the Selenium grid where we will ultimately send our Javascript tests. The Selenium documentation states: “The hub receives a test to be executed along with information on which browser and ‘platform’ (i.e. WINDOWS, LINUX, etc) where the test should be run”. And the desired_capabilities option specifies what browser to run the test in.
These ENVs are set within the docker-compose file
Now we have a driver registered with Capybara, it’s time to specify a separate configuration for JS tests that are flagged with js: true.
By specifying driven_by: :selenium_remote, we are telling Capybara to use a separate driver for JS tests. Without this, it’ll attempt to use Rack Test as the driver which will run the tests in a headless browser so no Javascript will be executed.
Finally, a Make task that spins up the test container and executes the RSpec tests against the running instance whilst tearing them down afterwards. The $(SPEC) parameter is just so I can pass spec/directory/test_file arguments to the Make command.
This is the part I did not understand and still do not, but was ultimately what got this working. With Docker, you can usually run one time commands against a container using docker-compose run service-name COMMAND , yet attempting to run RSpec in this way (as I had been doing up until that point), gave me a Failed to open TCP connection to test:3005 error, meaning the service in the compose file, test , was not found. So by running docker-compose run test rspec @js:true (that last part to run only JS tests), a TCP connection couldn’t be made by the Capybara server. If anyone knows why this is the case I would appreciate some insight. But what ultimately worked was to spin up the test container in a demonised process and then execute an rspec command against it and subsequently tear down the container plus the Selenium one it automatically spins up due to the linking.
Overall, this was a real pain to set up. I still don’t fully understand all of it, but now I have a higher degree of confidence that when my tests all pass the application is working as expected, from the model layer to the transportation layer and now from the user interface layer. Having tests that fail if an enquiry form is not visible after clicking a button is extremely valuable, as bugs like this cannot be caught with standard Rack tests with a headless browser, and this can save a lot of Q&A clicking around time that ultimately no longer needs to be done. Despite the struggles, I have to say hurray for Selenium and Docker for allowing me to set up a solid and reproducible testing environment for this application.
