Metadata Management | Techlist.io

netflixApr 4, 20263 min readCurated summary

Powering Multimodal Intelligence for Video Search

Video search is difficult because it must combine many kinds of information—characters, scenes, dialogue, labels, and embeddings—across enormous volumes of footage. The post argues that solving this problem requires a distributed pipeline that separates reliable ingestion, computationally intensive data fusion, and low-latency search indexing. Temporal bucketing, hybrid ranking, and deduplication turn billions of model outputs into searchable moments for editors. ## Why Video Search Is Complex - Video contains multiple overlapping modalities, each analyzed by specialized models. - Models produce different outputs, including: - Text labels such as characters or objects - Scene classifications - High-dimensional embedding vectors - Time ranges with varying boundaries - Overlapping model timelines must be synchronized into a chronological representation. - A 2,000-hour archive may contain more than 216 million frames, expanding to billions of records after multimodal processing. - Search must avoid returning thousands of redundant clips from continuous shots. - Ranking therefore combines: - Symbolic text matching for precision and interpretability - Semantic vector similarity for contextual relevance - Clustering and deduplication to identify the best moments - Sub-second response times are essential because delays interrupt editors’ creative workflows. ## Three-Stage Ingestion and Fusion Pipeline ### Transactional Persistence - Raw model annotations are ingested through highly available pipelines. - Apache Cassandra stores the annotations with an emphasis on: - Data integrity - Distributed availability - High write throughput - An annotation can include a type, nanosecond time range, embedding vector, label, and confidence score. ### Offline Data Fusion - After persistence, Apache Kafka publishes an event that starts asynchronous processing. - The offline pipeline performs expensive temporal intersections without slowing ingestion or search. - Model outputs are normalized into fixed one-second time buckets. - The fusion process: - Maps continuous detections into discrete intervals - Intersects annotations sharing a bucket - Combines them into unified records - Writes the enriched records back to Cassandra - For example, a “Joey” character detection from seconds 2–8 can be combined with a “kitchen” scene detection from seconds 4–9 to create a fused record for the 4–5 second interval. - Each fused record retains links to the original annotations and source asset. ### Real-Time Search Indexing - Enriched buckets are later sent from Cassandra to Elasticsearch. - Upserts use a composite key consisting of the asset ID and time bucket. - If a bucket already exists, it is updated rather than duplicated. - This creates one consistent record for each second of footage while allowing new model results to be incorporated. The overall recommendation is to treat multimodal video search as a distributed data-fusion problem rather than a single-model retrieval task. Decoupling ingestion, offline processing, and indexing allows the system to handle massive archives while preserving reliable data capture and fast, context-rich search.

Read original

Metadata Management Database Design Apache Kafka Elasticsearch+4

datadogJan 29, 20253 min readCurated summary

Husky: Efficient compaction at Datadog scale

Husky is a distributed event store built on object storage for observability workloads reaching trillions of events per day. Because data is written continuously, rarely updated, and queried both recently and historically, its storage layer must minimize object-store fetches while still supporting high query parallelism. The central design challenge is choosing a compaction and layout strategy that keeps fragments manageable without sacrificing query speed. ## Husky’s Query Execution Model - Ingested events are grouped into files called **fragments** and stored in systems such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. - Metadata for each fragment is stored separately in FoundationDB. - For each query: - Metadata is scanned to identify relevant fragments. - Fragments are distributed among query workers. - Workers scan their assigned data. - Results are merged. - Query cost depends mainly on: - The number of fragments fetched from object storage. - The number of events scanned within those fragments. - Husky therefore focuses on: - Reducing the total number of files through efficient compaction. - Organizing data so queries scan as few irrelevant events as possible. ## The Compaction “Goldilocks” Problem - Compaction combines many small fragments into a larger fragment containing the same data. - FoundationDB transactions atomically replace the old fragments with the compacted one, ensuring queries see a consistent state either before or after compaction. - Ingestion writers buffer events per tenant to avoid producing extremely small files, but they flush periodically to keep newly ingested data queryable quickly. - These flushed fragments may contain only a few thousand events, making queries inefficient when they must fetch thousands of objects and metadata records. ## Balancing Fragment Size Husky must find a fragment size that balances several competing concerns: - **Object storage and metadata overhead** - Fewer, larger fragments reduce the number of fetches and metadata entries. - **Compaction cost** - Larger or more aggressively reorganized fragments require more CPU and more object-storage GET and PUT operations. - **Query parallelism** - Smaller fragments allow more workers to operate concurrently. - Larger fragments reduce distribution overhead but can limit parallelism for large analytical queries. - **Scan efficiency** - Query workers use vectorized execution, which is most effective when scanning sufficiently large batches of rows. - **Data locality and compression** - Compaction can place events with similar timestamps or tags near one another. - This improves compression and allows queries to skip irrelevant data, but requires additional processing and query-pattern analysis. Fragments that are too small create excessive fetch and scheduling overhead. Fragments that are too large reduce parallelism and can make broad queries slower. The goal is a “just right” size suited to typical query patterns. ## Storage Layout and Query Selectivity - Husky organizes events along both: - The time dimension. - Spatial dimensions such as tags. - Keeping commonly queried events close together reduces the amount of data that must be scanned. - Similar data also compresses more effectively. - Achieving this layout increases compaction work, creating a tradeoff between lower query cost and lower maintenance cost. ## Scalable Compaction - Husky’s storage system depends on compaction being efficient enough to run continuously at very large scale. - The design must account not only for the final fragment size, but also for the CPU and object-storage costs required to produce it. - Atomic metadata updates ensure that compaction can occur without exposing partial results or inconsistent table states. Husky’s approach treats compaction as a core part of query performance rather than simple file maintenance. A practical design must tune fragment sizes, merge frequency, and data layout together to minimize total system cost while preserving fast access to both recent events and large historical datasets.

Read original

Metadata Management Database Design Query Optimization Foundationdb+4