Best Minds X Best Tools: DX Solutions & BrowserStack Partnership

Best Minds X Best Tools: DX Solutions & BrowserStack Partnership

May 5, 2020


For website and apps accessed by millions of Australians it is critical to ensure users get the best experience possible. For this to happen, engineering teams must check that releases work well across their entire user base, regardless of browser, device or OS used.

Automating your test scenarios through Browserstack has obvious benefits to your project delivery around quality at speed to expedite your test releases. However, beyond the known benefits of high-coverage test suites, accelerated release frequencies and satisfied customers, let’s not forget our favourite benefit of them all: happy engineers!



Brian Lucas, Senior Staff Software Engineer at Optimizely told us how they lowered ‘developer pain’ by moving testing to the Cloud.


The Challenge:

Engineers frustrated with the on-premise grid


Optimizely is the world’s leading progressive delivery and experimentation platform, enabling businesses to drive up the value of their digital products, commerce and campaigns. Product, engineering, data scientists, and growth marketing teams use Optimizely to quickly and safely rollout and rollback code to subsets of users through enterprise-grade feature flagging to ensure they can ship every new product, feature, or experience with confidence. This makes it critical for the platform to run smoothly for every business, and the engineering team ensures this by deploying high-quality releases at a high velocity.

The engineering team at the time was getting increasingly frustrated due to issues with their automated testing. Optimizely even has an in-house metric for this—Developer Pain Index (DPI). Brian Lucas, Senior Staff Software Engineer at Optimizely explains, “DPI is a mathematical function comprised of the time it takes to run the entire test suite over the average success ratio. The goal is to see it shrinking over time.”

Testing at Optimizely was initially done on their on-premise grid. They started with PhantomJS-based headless browser tests and a limited number of browser-based tests. To avoid this, they began writing an end-to-end test suite for real browsers. The team quickly discovered that real end-to-end tests are more complex and unreliable compared to unit tests for headless browsers, and thus, more prone to failure and non-deterministic results. Creating this end-to-end test suite, therefore, required all available engineering bandwidth for several months.

At the same time, keeping the on-premise grid up and running was a challenge. There were browser-sandbox incompatibilities, a time-consuming discovery process for browser-version updates, and dependencies on package maintainers.

Moreover, the on-premise grid was inefficient in highly parallelized test environments, forcing engineers to run tests sequentially. “You can quickly reach a max capacity and run into physical resource exhaustion issues with on-premise machines,” explains Brian. These bottlenecks, coupled with the added overhead of building and maintaining the on-premise grid, frustrated the engineers.

To resolve these problems, Optimizely chose a competitor cloud-based testing service. “We worried that if we didn’t provide advanced tools to our engineers, we might be sacrificing some aspect of our quality process.” But the service provider’s unreliable infrastructure and frequent downtimes slowed the testing momentum Optimizely needed to achieve their goals.

They eventually moved to BrowserStack, looking for a more stable solution.


The Solution:

Scalable cloud infrastructure for streamlined QA


Moving to cloud infrastructure freed up the engineering team. “A dedicated service provider (like BrowserStack) can maintain the infrastructure better than us. Instead, we could focus on building and running tests to get better coverage,” says Brian.

As a result, the team could focus their attention on crafting a robust end-to-end test suite. They built a highly reliable framework titled Behavely on top of a Gherkin-based system called Behave. They now have multiple CI environments running tests, and a CI/CD pipeline with multiple environments. They leveraged parallelization and other BrowserStack features to deploy bug-free releases more frequently, improving the overall QA process.


The Impact:

Improved quality, no maintenance overheads, happier engineers


“The move to BrowserStack has been a force multiplier across all teams. It is a core and critical part of our build and release infrastructure, and every engineer ultimately depends on BrowserStack in one way or the other,” explains Brian.

BrowserStack’s elastic cloud of browsers allows Optimizely to add capacity when required, without any maintenance overheads or resource constraints on their CI infrastructure. As a result, they now build and test much quicker. Today, 15,000+ automated and parallelized test cases are run on Optimizely’s CI infrastructure (many of these on BrowserStack) every 45 minutes and released to a pre-production environment every 4 hours.

BrowserStack features—such as video and console logs—helped to simplify the bug-detection process, boosting the overall code quality. “Our engineers could perform forensic analysis or diagnosis on their tests in case something went wrong. They were empowered to deploy hotfixes to particular releases before they reached our customers. This way, our engineers could perform comprehensive final reviews to confirm the quality of our code.” Similarly, they could also monitor the total time and effort spent on running tests and use the insights to optimize their test suite, making it more reliable.

Efficiencies brought in by BrowserStack have lowered the Developer Pain Index by a significant margin, particularly where many browser tests would be untenable to run on an individual machine. With BrowserStack, Optimizely is swiftly moving towards the ideal state of software testing with a high-coverage test suite, accelerated release frequency, and happier engineers.

Watch the webinar to hear the story directly from Brian Lucas, Senior Staff Software Engineer at Optimizely.



How can you as a test team make the most of what BrowserStack can offer, and what is the key project ‘legwork’ required to effectively scale your monthly releases to daily releases?

Here’s our roundup of the key take-outs to migrating a manual test team to automated testing practices:


For Waterfall Environments

It’s all in the planning

To successfully migrate manual testing to complete automation, set out with a robust, reliable automation framework (many open source frameworks available with prebuilt patterns) which allows fast adoption and early success, and be sure to drive the approval and willingness of all stakeholders involved. While these planning stages will require an investment for initial development, in the end, it will be worth it.

Start with easy wins

Get your stakeholders comfortable with migrating to new ways of testing by starting with prioritising test scenarios that are easily automatable, and gradually increase automation velocity as you build up your team’s success, putting the right controls in place to mitigate risk.

Get the best minds on the job

Taking advantage of the best tools and platforms like BrowserStack are vital artillery in your engineering toolkit, but when you combine best tools with best minds (us!) now, you’re really onto a winning combination! Ensure your hiring process reviews the right technical talent. Raising defects isn’t what building in quality is about. Ask DX for the ‘why’ and we can work with your team to help you better understand quality.


For Agile Environments

Picking the right tool chain

Contrary to waterfall where stakeholders are generally more conservative, in agile, due to the increased release cadences, automatingtest scenarios is something that should be done pretty quickly. Speed is always of the essence in agile development environments so if you are still manually testing, you need to pick the easiest open sourced testing framework to use. Typically, most people would pick Selenium but in most cases, this is actually a bad choice! Sure Selenium works on web apps but the amount of lines of code required to write as well as the fact that it doesn’t work as well for newer Javascript (JS) frameworks means you should be looking at other JS frameworks. Fortunately, at DX, we’ve written a test base that will make an engineer’s life much easier.

Scaling tests – the right balance

Quality at speed is essential in an agile development environment. However, like everything it comes at a cost. Understanding the right mix of how many tests to run in parallel, the cost associated with a particular tier in subscription and the time it takes for a full cycle to run; is crucial for getting the most return on investment. This is particularly the case for small to medium sized business. If you are unsure, ask a DX consultant and we can help you better understand the complexities associated with running test automation at scale.

Integration with the SDLC management tooling

Reporting is equally as important as getting raw feedback for the tests. Integrating Jira with BrowserStack is essential from a observability, traceability and auditability perspective. If issues are found, it is imperative that evidence is collected and distributed to the right stakeholders. Out of the box, BrowserStack integrates well and as a platform, provides all the evidence you will ever need including videos, screenshots and logging that can be pushed to popular SDLC management tooling.


If you would like to get started and take advantage of DX Solutions’ exclusive BrowserStack partnership discounts, let’s talk about it together.

DX Solutions is a wholly owned and operated Australian private company providing businesses with strategic solutions to optimise, transform and progress their operations. With around 200 employees we are proud to partner with software, project and network engineering teams to deliver quality assurance, test automation, CI/CD, agile project management, business process analysis and automation (RPA), network engineering and cloud infrastructure services for Australia’s major digital transformations.