From Silos to Service Topology: Why Netflix Built a Real-Time Service Map
Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting. ## The Observability Problem - Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization. - During incidents, engineers need to determine: - Which services depend on one another - What the potential blast radius is - Whether a failure originates locally or upstream - Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology. - Manually combining information from different tools is slow and error-prone, especially during urgent incidents. ## Why Real-Time Service Mapping Matters - Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete. - Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness. - Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes. - These recurring questions demonstrated the need for accurate, near-real-time dependency information. ## Lessons from Earlier Approaches - Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology. - Key lessons included: - Dependency data must update in near real time. - Storage and query systems must operate at Netflix’s scale. - The solution should integrate with existing observability workflows. - Incorrect or incomplete topology data can mislead engineers during incidents. - No single data source captures every aspect of service relationships. ## Requirements for a Living Map Service Topology was designed to provide: - Real-time updates as services deploy and dependencies change - Sub-second queries for traversing service call graphs - Both network-level and application-level views - Context such as health, availability tiers, ownership, and business domains - A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis ## Combining Multiple Sources of Truth Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times. ### eBPF Network Flows - eBPF captures network activity at the kernel level, recording which services communicate over the network. - This provides broad coverage, including services that lack application instrumentation. - It supports both cluster-level and application-level topology. - Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved. Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.
Read original(opens in new tab)