datadogOct 27, 2016

Cheering on coworkers: Building culture with Datadog dashboards (opens in new tab)

Christian’s colleagues built a Datadog dashboard to remotely track his progress in a six-day, 850 km ultramarathon. They scraped live race data from the event website, converted it into Datadog metrics, and visualized his distance, ranking, and elapsed time alongside video and other dashboard elements. At publication, Christian was leading by more than 47 km with 44 hours remaining. ## Extracting Race Data - The event website regularly published runners’ statistics and race progress in plain HTML. - A Python crawler using `Requests` retrieved the webpage. - `BeautifulSoup` parsed the HTML to extract: - Current ranking - Total distance run - Elapsed time - Other race information ## Sending Metrics to Datadog - The team used the Datadog Python client and StatsD to emit metrics through the Datadog Agent. - For each runner, the script sent gauge metrics for: - `runner.distance` - `runner.ranking` - `runner.elapsed_time` - Metrics were tagged with each runner’s name, enabling individual tracking and comparisons. ## Building the Dashboard - The collected metrics were combined into a Datadog dashboard. - The dashboard included: - Live race statistics - A live video feed - Animated GIFs for entertainment - Visualizations of meaningful progress metrics - Screens displaying the dashboard were placed in the company’s New York and Paris offices so colleagues could follow and encourage Christian throughout the race. The project demonstrates how a lightweight web scraper, StatsD metrics, and a monitoring dashboard can turn publicly available data into a live, engaging team experience.

beautifulsoupdatadogpythondashboard-design+2

datadogOct 27, 2016

Cheering on coworkers: Building culture with Datadog dashboards | Datadog (opens in new tab)

Datadog engineers developed a real-time tracking dashboard to monitor a colleague’s progress during an 850km, six-day ultra-marathon challenge. By scraping public race statistics and piping the data into their monitoring platform, the team created a centralized visualization tool to provide remote support and office-wide engagement. ### Data Extraction and Parsing The team needed to harvest race data that was only available as plain HTML on the event’s official website. * A crawler was built using the Python `Requests` library to automate the retrieval of the webpage's source code. * The team utilized `BeautifulSoup` to parse the HTML and isolate specific data points, such as the runner's current ranking and total distance covered. ### Ingesting Metrics with StatsD Once the data was structured, it was converted into telemetry using the Datadog agent and the `statsd` Python library. * The script utilized `dog.gauge` to emit three primary metrics: `runner.distance`, `runner.ranking`, and `runner.elapsed_time`. * Each metric was assigned a "name" tag corresponding to the runner, allowing the team to filter data and compare participants within the Datadog interface. * The data was updated periodically to ensure the dashboard reflected the most current race standings. ### Dashboard Visualization and Results The final phase involved synthesizing the metrics into a high-visibility dashboard displayed in the company’s New York and Paris offices. * The dashboard combined technical performance graphs with multimedia elements, including live video feeds and GIFs, to create an interactive cheering station. * The system successfully tracked the athlete's 47km lead in real-time, providing the team with immediate updates on his physical progress and elapsed time over the 144-hour event. This project demonstrates how standard observability tools can be repurposed for creative "life-graphing" applications. By combining simple web scraping with metric ingestion, engineers can quickly build custom monitoring solutions for any public data source.

beautifulsoupdatadogpythonmetrics+3