The trouble with mounting | Datadog (opens in new tab)

The post explains why filesystem mounting becomes surprisingly complex in containerized Linux environments. Datadog’s Agent needs to inspect host filesystems from inside a container, but mount namespaces, bind mounts, and propagation rules can make the host’s view incomplete or inconsistent. The conclusion is that reliable mounting requires understanding namespace boundaries and deliberately configuring mount propagation rather than treating mounts as ordinary directory mappings.

Linux Mount Namespaces

• Each process can have its own mount namespace, isolating its view of mounted filesystems.
• A container therefore sees a filesystem tree that may differ substantially from the host’s tree.
• Bind-mounting a directory into a container does not necessarily expose mounts created beneath that directory.
• This is particularly problematic for paths such as /proc, /sys, /var/lib/docker, and other locations containing nested mounts.

The Problem with Bind Mounts

• A bind mount initially exposes only the directory tree visible at the time it is created.
• Later mounts beneath the source directory may not appear in the container.
• Recursive bind mounts can copy nested mounts, but they introduce their own behavior and compatibility concerns.
• Mounts can also be shared, private, or “slave,” determining whether mount and unmount events propagate between namespaces.

Mount Propagation

• Linux mount propagation controls how changes in one namespace are reflected in another.
• Shared mounts propagate events in both directions, while slave mounts receive changes without sending them back.
• Private mounts isolate changes completely.
• Choosing the wrong propagation mode can cause the Agent to miss newly mounted filesystems or, worse, allow container-side changes to affect the host.

Datadog’s Engineering Challenge

• Datadog needs host-level visibility while keeping the monitoring container isolated and safe.
• The Agent must account for mounts that appear after startup, including dynamically created container and volume mounts.
• Correct behavior depends on both the container runtime configuration and the host’s existing mount topology.
• A robust implementation must inspect mount metadata and handle namespace and propagation details explicitly.

The practical recommendation is to treat mounting as a namespace and event-propagation problem, not merely a path-sharing mechanism. Systems that need host visibility should use carefully selected recursive mounts and propagation modes, validate the resulting mount tree, and test behavior when mounts are created or removed dynamically.