15 minute read
Tugboat increases our velocity. Tugboat has made it possible for us to painlessly have a rapid sprint cycle, delivering on improvements and bug fixes quickly.
Jenna Tollerson, Engineering Lead, Digital Services Georgia
The State of Georgia needed to migrate 80 Georgia agency sites to an all-new digital publishing platform, to be dubbed GovHub. Tugboat provided staging and proofing environments for each agency’s site migration along the way. This meant engineers could develop, test, and deploy migrations using Continuous Delivery, ultimately increasing the velocity and stability of the release process for the entire team.
The State of Georgia needed to migrate over 80 of their agency websites from an aging Drupal 7 platform, to an all-new digital publishing platform designed by IDEO and developed by Lullabot. Georgia agency stakeholders needed to be able to review and audit migrations of these sites continuously, without the bottleneck of a limited number of staging servers. In addition, the fast-moving development team of over 20 engineers needed to be able to develop in parallel without impacting one another’s work.
With Tugboat’s ability to replicate Georgia’s production infrastructure as code (IaC), and instantaneously replicate extremely large datasets, the development team was able to have virtually unlimited staging sites at their fingertips: automatically on any new Pull Requests, and on-demand with a click of a button.
Tugboat has eliminated our previous bottleneck caused by having a limited number of staging environments.
Jenna Tollerson, Engineering Lead
Digital Services Georgia
Before Tugboat, the State of Georgia was stuck in a common software development life cycle (SDLC) anti-pattern: the limitation of only one or two testing environments, resulting in the painfully slow bottleneck of a single workstream. With Tugboat, Georgia was able to test work on production-like infrastructure, using all the same scripts and technologies that would be in use in production, resulting in an exponential increase in both the productivity and stability of the project.
The development for GovHub was a collaborative effort, featuring the combined engineering forces of Digital Services Georgia, Lullabot, and Mediacurrent. While Lullabot focused on the architecture and functionality of the underlying system, Mediacurrent needed to focus on developing migrations for each of the agency sites in an ongoing fashion, without interrupting or being interrupted by other initiatives. Tugboat facilitated an interruption-free workflow by automatically running the in-progress Drupal migrations in Tugboat Previews, but only for the pull requests specifically working on migration code. This meant that the Mediacurrent developers would automatically get previews of their migration code, while Lullabot developers focusing on non-migration work weren’t held up by the long-running and in-progress migration scripts.
Providing an efficient and stable development process on a fast-moving project, while onboarding engineers from multiple vendors, was critical to achieving the aggressive timeline. To achieve that stability, Tugboat’s Technical Account Executive, who was then a Lullabot architect on the project, worked with the development vendor to script a CI workflow to sanitize and import databases directly into custom Docker images. Developers were able to leverage this process to use the same Docker images in their local development stack, providing consistent data on all development and staging environments.
The resulting efficiency and stability of the development process exceeded all expectations. Developers could have a local environment in hours, not days or weeks. Additionally, the process was more secure than traditional workflows, since developers and engineers did not need access to production infrastructure to begin working. And, developers knew that the same data they had on their local environments would also be available on Tugboat, creating stability and environment consistency despite the extremely dynamic and active development process.
Digital Services Georgia and the Lullabot engineering team both know the value of building software that works for everyone. Among other things, that meant ensuring new development met the highest web accessibility standards. To aid in this goal, Lullabot added Google Lighthouse tests for a series of agency sites to run automatically on Tugboat. For every pull request created, the developers, stakeholders, and QA engineers could review those Google Lighthouse test results, ensuring none of the new work introduced accessibility regressions.
Early in the development of the multisite Drupal deployment process, the team recognized an issue that would only get worse: the time to execute all of the deployment scripts was getting longer and longer with every site added. Some quick arithmetic revealed that if the current pace continued, the total deployment process would take several hours to complete once all agency sites were migrated. Lullabot came up with an idea to solve this issue, and then used Tugboat’s infrastructure-as-code approach to create a production-like stack to prototype and benchmark the solution. Tugboat made this possible without risking any other initiatives or adding any additional financial burden that would normally come with provisioning expensive production-like infrastructure.
As a developer, I often prefer to show rather than tell. When I prototype new features for stakeholders, Tugboat allows me to present the idea on a working site, and gather feedback asynchronously.
Jenna Tollerson, Engineering Lead
Digital Services Georgia
Tugboat also supports multisite Drupal, a feature the Georgia team leveraged in production to serve 80 sites from the same codebase. Tugboat’s aliasing feature allowed the Georgia team to preview each site using the same multisite technology that would be in use on production, ensuring the in-progress work on each agency site was fully tested, all before any of that work was merged into the stable branch.
The implementation of Tugboat for the State of Georgia meant considering the tools they would be using for development of their new platform.
While the DSGa and Lullabot teams focused on building out the platform features, Mediacurrent’s developers needed a way to test migration code. Ideally, each team’s work should not disrupt the other. Some clever configuration enabled this.
First, Tugboat needed Drupal 7 source data to be made available within Tugboat’s preview environments, so migrations could be tested and proofed there. Then, the teams added a script to automatically run the migrations on pull request preview environments whose git branches contained migration in the name.
With this configuration, developers who weren’t working on migration wouldn’t need to wait for migrations to complete to have access to Tugboat previews. However, developers who were working on migrations could tell Tugboat to automatically run the migration with a basic branch naming strategy. The result of this win-win was increased productivity for all developers on the project.
During our migration, it was all hands on deck, and Tugboat made it possible for those without the time or ability to spin up a local development environment to participate in the QA process.
Jenna Tollerson, Engineering Lead
Digital Services Georgia
Tugboat’s ability to authenticate to a private Docker registry gave Georgia the ability to create custom Docker images with MySQL databases already available.
To accomplish this, the team wrote a script to copy production databases to an Acquia development server. From there, each database was sanitized of all emails, passwords, form data, or other sensitive information. Then, each database was dumped and copied down to a CI environment, and imported into a Docker image using Tugboat’s Percona image as a starting point. Once imported, the CI worker tagged the image with the date of the import, and pushed the new Docker image to a private Docker registry hosted on Quay.io.
Connecting Tugboat to this private registry was easily accomplished with the “Docker Registry Authentication” form on the Repository settings.
To add automated accessibility testing to Georgia’s Tugboat set up, the Tugboat config.yml was modified to add a new service with Google Lighthouse CI installed. In the build steps of the Tugboat config, the lighthouse script was used to generate HTML audits on specific agency sites and paths on Tugboat Previews.
A very basic NodeJS web service was added to the Tugboat setup to list out these HTML reports.
Clicking into any of these reports would display the Google Lighthouse results for the given page. The State of Georgia’s QA and stakeholders could visit these HTML reports to evaluate and review the in-progress work, ensuring it met or exceeded the desired accessibility guidelines.
When investigating a decline in performance during the deployment process, Lullabot discovered a likely culprit. The majority of the bottleneck appeared to be the result of all the site-specific operations (such as running database updates, importing config changes, clearing caches and the like) being executed synchronously on a single PHP web server. During deployments, the CPU and memory usage on that single webhead would spike, while the remaining 10 webheads sat idle.
If it were possible to leverage the other web servers to asynchronously execute the PHP operations for the agency sites, theoretically the total execution time could drastically decrease. To accomplish this, Lullabot would use a little-used feature of Gnu Parallel: the ability to effectively load-balance a series of commands across multiple servers asynchronously.
Thanks to Tugboat’s infrastructure-as-code approach to configuration, Lullabot was able to create a production-like stack on Tugboat to prototype and benchmark this idea. Multiple identical PHP worker services were added to the Tugboat configuration, and the deployment scripts were modified to make use of Parallel’s ability to distribute jobs to those services. Then, changes to the production deployment scripts were developed and tested in a safe space, without any disruption to production.
Due to Tugboat’s ability to replicate complex network topologies and production infrastructure, the engineers were able to experiment with this idea and prove it’s viability for production, all without placing any other team or initiative at risk or on hold, and without adding any financial burden on the organization. Bugs were discovered early, on non-production infrastructure, resulting in a bug-free, successful deployment to arguably the most critical component of the stack: the very deployment process itself.
Using Tugboat’s aliases configuration option, the GovHub project could specify different URL aliases for agency sites on Tugboat. Tugboat. For example, consider this Tugboat configuration Yaml:
This tells Tugboat to provide different preview URLs for each of these aliases, which Drupal can then read to know which site is which. This is controlled using Drupal’s sites.php file, by reading in the TUGBOAT_DEFAULT_SERVICE_TOKEN environment variable and prefixing it with these aliases.
Once configured, Tugboat prints out each aliased URL as a comment in the linked git repository’s - in this case, a Bitbucket comment - and also on a drop-down from the Preview button in the Tugboat UI.
Adding Tugboat to Digital Services Georgia’s engineering process has increased the velocity of their release cycle, added stability and robustness to a complex migration and replatforming project, and opened possibilities for rapid experimentation. Even now that the replatform is complete, Tugboat continues to be a critical tool for the stakeholders, QA engineers, and support and maintenance staff.
Tugboat increases our velocity. Tugboat has made it possible for us to painlessly have a rapid sprint cycle, delivering on improvements and bug fixes quickly.
Jenna Tollerson, Engineering Lead
Digital Services Georgia
Digital Services Georgia (DSGa) is part of the Georgia Technology Authority, the state’s central IT agency, which promotes an enterprise approach to using technology to serve Georgians. DSGa manages Georgia.gov, the official website for discovering state government information and services, and Georgia GovHub, the state’s official web publishing system for delivering state-of-the-art, secure, accessible, and compliant websites.
Tell us about your project, schedule a demo, or consult with a Tugboat Technical Account Executive.
