How to Leverage On-Demand Environments for Automated Testing

Automated testing in software development is the cornerstone of a strong agile development process. Being able to quickly and easily verify that new functionality doesn’t break existing code, or that it functions as expected, is key to iterating and releasing quickly and with confidence. However, in many organizations, extensive testing — even automated testing — is not practical due to infrastructure constraints or long build times. The solution for this quandary is to leverage on-demand environments for automated testing. A git pull request builder like Tugboat, which generates working versions of your website or web app for every pull request, can replace static and constrained infrastructure in your automated testing process.

Why on-demand environments and automated testing are a match made in heaven

In an ideal world, every pull request should be fully tested before it’s ever merged into development or production. However, in the real world, testing is often constrained by limited server infrastructure. You’ve got a team of four people each testing a few pull requests every day on one shared server? The infrastructure quickly becomes a testing bottleneck. The testing itself takes time, but beyond that, you’ve got time between tests to reset the database, load new code, and build the environment. Only then can you start running tests.

Tugboat solves this problem by providing on-demand environments for every pull request. When your development team makes a pull request for a new piece of functionality or new code, Tugboat automatically spins up a Preview that creates a working version of your website. When you pair this with an automated testing tool, such as Testery.io, the tests can automatically kick off once the environment is ready. By the time a developer is ready to look at a PR for code review, or a QA team member is ready to review the work, the environment is ready and the tests have been run — all without requiring someone to do manual steps. When you’re testing a handful of pull requests daily, and each of those pull requests gets its own automated Tugboat Preview versus having to manually load code onto a shared server, that’s a dramatic time saving for your team.

So how does this work?

How to set up automated testing with Tugboat and Testery

Set up Tugboat to build deploy environments for every pull request

First, you’ll need a Tugboat account. Tugboat’s free tier should be enough to get you up and running and set up a proof-of-concept, but if you’ve got resource-intensive or large builds, you may want to explore a Tugboat plan that offers more CPU, build-time RAM, and storage.

Next, you’ll need to create a project and link Tugboat to your git repository. This allows Tugboat to pull in your code. You’ll need to create a Tugboat config file to set up builds on Tugboat. Our docs offer starter configs to help kick-start simple build processes out of the box.

With all this in place, you should be able to start building Previews on Tugboat; these are our dynamically-generated environments that automatically build when you create or update a pull request.

Set up a testing framework via Testery

Once you have Tugboat Previews building as expected, it’s time to hook up your testing framework. Our friends at Testery have made this simple, so we’ll show you how to set up Tugboat + Testery to solve a lot of your testing needs. If you’d like to see this in action, Testery did a demo for a DevOps meetup, which you can watch on YouTube.

On the Testery side, you’ll need to set up an account; they also offer a free tier that should get you up and running to set up a proof-of-concept.

Follow their instructions to get set up with your testing framework. In the video demo above, Testery is configured for testing with WebdriverIO, but Testery also works with SpecFlow, Cypress, and Nightwatch.js. If you are using a different testing framework, reach out to the Testery folks; they’re adding support for other frameworks as needed.

Create a project in Testery, and select the framework you want to use for your tests. Associate the Testery project with the project containing your tests, and then you’re ready to link Testery and Tugboat to run your tests against the Tugboat environment that’s automatically generated from your PRs.

Linking Testery and Tugboat

You’ll need to install the Testery CLI in your Tugboat build in order to kick off test runs. First, you’ll need to install pip. Add these lines to your Tugboat config.yml in the init steps:

# Install apt-get
apt-get update
# Install pip
apt-get install -y python3 python3-pip

Then run these commands under your build step:

# Install the Testery CLI.
pip3 install testery --upgrade

# Set up environment variables to tell Testery what Tugboat environment to use for testing
testery update-environment --create-if-not-exists
 --token "$TESTERY_TOKEN" --key "${TUGBOAT_PREVIEW}"
 --name "${TUGBOAT_PREVIEW}"
 --variable "TUGBOAT_DEFAULT_SERVICE_URL=${TUGBOAT_DEFAULT_SERVICE_URL}"

You’ll note that we’re using environment variables to store a token to use Testery, and to reference Tugboat Previews to use as environments for Testery test runs.

With these configuration details in place, you can kick off Testery test runs against your Tugboat environments with this command:

# Start a test run.
testery create-test-run --token "$TESTERY_TOKEN" --git-ref "$TUGBOAT_PREVIEW"
 --project "testery" --environment "${TUGBOAT_PREVIEW}"

For a full example config.yml, take a look at our Testery.io Integration page on the documentation website.

The result: on-demand environments and test results right on your pull requests

With this setup, you can get automated builds for every git pull request via Tugboat, and have tests run against those environments via Testery, and have the results of both posted right to your pull requests in your linked git provider.

Here’s what this looks like in GitHub:

You can click into the Tugboat environment to manually verify the build, or click into the Testery details to view more information about the test runs.

Stop waiting for your turn for a testing server, and start leveraging this powerful combination of tools to automate the boring stuff and get back to solving interesting problems with code.

‍