Automate detection testing with GitLab CI/CD and Duo (opens in new tab)
GitLab’s WATCH framework continuously tests whether security detections still work in real conditions, rather than only verifying that detection rules deploy successfully. It runs simulated attacks in staging, checks alert propagation through logging, SIEM, and SOAR systems, and reports failures automatically. The framework uses GitLab CI/CD to schedule randomized tests, correlate expected alerts, and publish detection-health results. ## The Detection-Validation Gap - Security detections can silently fail because of: - Log schema changes - SIEM updates - Ingestion or pipeline misconfigurations - Other changes between the log source and alerting systems - Reinjecting synthetic logs into a SIEM can test rule logic, but it does not validate real-world behavior or the log-ingestion path. - GitLab’s detections-as-code pipelines confirm that rules can be created and deployed, but not that they fire when the targeted activity occurs. - WATCH fills this gap by validating detections end to end. ## WATCH’s Testing Lifecycle - **Scheduling:** A weekly GitLab CI/CD pipeline discovers active tests and assigns them randomized execution times. - **Heads-up notification:** WATCH creates a dedicated “WATCH Heads Up” SOAR record containing the detections expected to fire. - **Execution:** Scripts perform simulated malicious actions in staging, such as resetting an administrator password or making suspicious API calls. - **Detection:** Activity logs flow through ingestion into the SIEM, where detection rules process them. - **Correlation:** SOAR matches alerts to registered WATCH tests using: - The time window between execution and alerting - Actor identity, such as an IP address or username - The detection rule ID - **Verification:** A follow-up job confirms that all expected detections fired, updates detection metadata, and publishes results to a GitLab Pages dashboard. - Failed tests generate notifications in the team’s Slack channel. - Correlation prevents test alerts from being escalated as genuine incidents while still validating the complete alerting pipeline. ## GitLab CI/CD Implementation WATCH is organized into three pipeline stages: - **`schedule_pipelines`:** - Runs weekly. - Finds active tests and groups them into scheduled pipelines. - Passes the selected tests through the `TESTS_TO_RUN` variable. - **`run_tests`:** - Executes the assigned attack simulations. - Saves execution results in `detection_status.json`. - Records SOAR identifiers needed for later alert correlation. - **`pages`:** - Queries the SOAR to verify alert generation and routing. - Updates `detection_status.json` with test results. - Deploys the latest status data and dashboard assets to GitLab Pages. The example configuration uses Python 3.12, pipeline inputs to enable weekly scheduling or dashboard updates, conditional `rules`, and GitLab Pages artifacts. Scheduled execution is randomized to avoid predictable test patterns and to expose timing-related problems. ## Practical Recommendation Organizations with critical security detections should add continuous behavioral testing alongside detections-as-code validation. A framework like WATCH can provide earlier warning of broken ingestion, rules, or routing while reducing the cost and generic limitations of commercial breach-and-attack simulation tools.