How we migrated our acceptance tests to use Synthetic Monitoring (opens in new tab)
Datadog’s Frontend Developer Experience team migrated 565 flaky, maintenance-heavy Puppeteer acceptance tests to Synthetic Monitoring. The change replaced manually scripted browser interactions with recorded tests that could run reliably from CI through a dedicated CLI. The year-long migration improved maintainability and built engineer confidence through documentation, gradual adoption, and non-blocking rollout tooling. ## Why the Existing Acceptance Tests Were Failing - Tests ran in Node.js on a custom Puppeteer-based runner. - End-to-end tests were flaky because they depended on browsers, virtual graphics, dedicated machines, navigation, and application timing. - Even simple actions required extensive scripting: - Confirming an element existed - Checking that it was enabled - Performing the interaction - Handling compatibility with Puppeteer - Custom UI elements, such as dropdowns, made reliable automation substantially harder. - Product changes frequently required updates to both tests and the testing infrastructure. - The six CI jobs took up to 14 minutes, with total machine time reaching 35 minutes per commit. ## Synthetic Monitoring as the Replacement - The team adopted Datadog’s own Synthetic Monitoring product to record page interactions rather than manually script them. - They created `synthetics-ci`, a CLI that: - Finds files named `*.synthetics.json` - Accepts configuration overrides - Triggers Synthetic tests - Polls for result statuses - Prints human-readable output - Because the tool represented a broader pattern for using Datadog from CI/CD, it was generalized into `datadog-ci`. ## Scope of the Migration At the start of the migration in June 2021, the frontend repository had: - 300 engineers working in one repository - Approximately 90 new pull requests and 1,120 commits every day - Six acceptance-test CI jobs - 35 minutes of machine time per commit - 84 relevant files - 565 tests - About 100,000 lines of test and infrastructure code The scale of the repository and its rapid development activity meant the migration needed to be gradual and carefully coordinated. ## Building Trust and Adoption - The team wrote documentation covering: - How to write effective Synthetic tests - Which behaviors were worth testing - Which testing patterns to avoid - They demonstrated the system in company-wide and frontend-focused meetings. - Teams learned how to: - Record tests through the UI - Use scheduled tests in CI - Replace existing acceptance tests - Reduce ongoing maintenance - The team worked directly with groups that owned the largest test collections. - Jira tickets tracked the migration of every acceptance test and assigned ownership to the appropriate team. ## Gradual CI Integration - A non-blocking CI job allowed teams to introduce Synthetic tests without risking the entire pipeline. - Failures appeared as pull-request comments rather than blocking merges. - This gave engineers time to understand and trust the new system. - Once tests ran reliably, the team made the pipeline blocking. - The old acceptance-testing platform could then be retired progressively instead of being removed all at once. The migration took roughly one year and succeeded through a combination of better tooling, clear communication, incremental rollout, and shared ownership across frontend teams. For large organizations replacing a critical testing system, introducing the new workflow safely before enforcing it can make adoption far less disruptive.