Scaling support with Vagrant and Terraform

Datadog’s Solutions Team uses reproducible virtual environments to investigate customer issues across diverse operating systems, kernels, and integrations. Vagrant simplifies local VM creation, while provisioning scripts eliminate repeated installation and configuration work. Terraform extends the same approach to shared AWS environments, enabling teams to provision, preserve, and collaborate on sandboxes quickly.

Reproducing Customer Environments with Vagrant

• Containers are useful, but virtual machines are better when reproducing specific operating systems, kernels, orchestrators, or complex infrastructure.
• Vagrant provides a simple workflow:
  • vagrant init
  • vagrant up
  • vagrant ssh
• The main challenge is not creating a VM, but installing and configuring the technologies needed to match a customer’s environment.
• With more than 200 integrations, engineers cannot be experts in every technology they may need to troubleshoot.

Standardizing Setup with Provisioning Scripts

• Vagrant provisioning supports tools such as Chef, Puppet, Ansible, and ordinary shell scripts.
• Datadog stores reusable reproduction environments in a shared GitHub repository.
• Each sandbox includes:
  • A Vagrantfile
  • A setup.sh provisioning script
  • A data directory for configuration files and supporting scripts
  • A README.md with usage information
• Engineer-specific values, such as hostnames and tags, are kept in a local .sandbox.conf.sh file.
• Once a sandbox exists, an engineer can run vagrant up and begin reproducing the customer issue within minutes.
• The directory hierarchy organizes sandboxes by operating system, version or provider, and technology—for example, Ubuntu Xenial with Kafka.

Sharing Remote Environments with Terraform

• Terraform provides similar infrastructure management for remote cloud instances, including AWS EC2.
• The team reuses the same setup.sh and data files for both Vagrant and Terraform, avoiding duplicate configuration work.
• Each sandbox adds a .tf file that:
  • Creates an EC2 instance
  • Copies required data files
  • Executes the provisioning script remotely
• A shared Terraform module handles common infrastructure tasks, while a tf.example file helps engineers create new configurations.
• This preserves the same repository structure and workflow while extending sandboxes from local VMs to remote environments.

Benefits for Team Collaboration

• Remote sandboxes can remain available without consuming engineers’ local RAM.
• Proper network security allows teammates to access and share environments.
• Engineers can reproduce previously configured integrations during live customer interactions.
• Investigations can continue across time zones, allowing teams to hand off urgent issues without rebuilding the environment.

The overall recommendation is to treat reproduction environments as reusable infrastructure: encode installation and configuration steps once, store them in version control, and use Vagrant for local testing and Terraform for persistent, shared cloud sandboxes.