The AWS Weekly Roundup highlights major updates in AI pricing, observability, multicloud networking, identity resilience, and data lakes. The biggest change is an up to 80% price reduction for OpenAI GPT‑5.6 Luna models in Amazon Bedrock, alongside several managed services that reduce infrastructure and operational overhead.
## Lower Bedrock Pricing for GPT‑5.6
- Effective July 30, GPT‑5.6 Luna inference prices dropped by 80%.
- New pricing is:
- $0.20 per million input tokens
- $1.20 per million output tokens
- GPT‑5.6 Terra prices decreased by 20%.
- The reductions apply automatically and require no customer action.
## Managed Prometheus Monitoring in CloudWatch
- Amazon CloudWatch now provides fully managed Prometheus collectors.
- Customers can collect metrics from:
- Amazon EKS
- Amazon EC2
- Amazon ECS
- Amazon MSK
- Amazon OpenSearch Service
- This removes the need to deploy and maintain custom Prometheus scraping agents.
## Private Multicloud Connectivity with OCI
- AWS Interconnect for Oracle Cloud Infrastructure is now generally available.
- It enables resilient, scalable private connections between AWS and OCI.
- Traffic avoids the public internet, improving security, performance, and reliability for multicloud workloads.
## Multi-Region IAM Identity Center
- IAM Identity Center can now replicate its built-in Identity Center directory across Regions.
- During a primary-Region disruption, users can continue accessing AWS accounts through provisioned entitlements in additional Regions.
- Previously, multi-Region support was limited to deployments using external identity providers.
## Variant Support in S3 Tables
- Amazon S3 Tables now supports Apache Iceberg V3’s Variant data type.
- Variant provides native, high-performance support for semi-structured data.
- Suitable use cases include IoT sensor data, application logs, and schema-flexible payloads without storing everything as JSON blobs.
## Additional AWS Resources
- New AWS CLI single-line commands simplify installation and upgrades across platforms and CI environments.
- A deployment guide covers running Moonshot AI’s Kimi K3 on SageMaker HyperPod and Amazon EKS.
- Amazon MSK Express brokers can deliver Kafka data to Apache Iceberg streaming tables on S3 Tables, with throughput of up to 10 GB/s.
- AWS Summits and AWS Community Days offer upcoming opportunities for cloud and AI learning and networking.
AWS users should review the new Bedrock pricing, consider managed CloudWatch collectors to reduce monitoring maintenance, and evaluate the multicloud, identity, and Iceberg updates for architectures requiring greater resilience and scalability.
Nebula is Toss’s centralized device farm, turning real-device testing into a simple API call instead of a team-specific infrastructure project. It grew from 15 devices and one developer into a 24/7 platform with more than 100 devices, shared across the company. The team replaced Appium with a faster, stateless custom driver and invested heavily in real-time streaming, reliability, security, and compliance.
## From Team-Owned Farms to a Central Platform
- Before Nebula, teams managed their own small farms using MacBooks or Mac minis.
- Each team repeatedly solved the same problems:
- Appium setup and maintenance
- Device detection and recovery
- OS-version compatibility
- USB and hardware failures
- Security and compliance
- Most teams could manage only five to ten devices, and resources remained isolated.
- Nebula centralized device operations so product teams could focus on testing rather than infrastructure.
## One API for Real Devices
Nebula’s core goal is to let anyone control a physical device from anywhere through a single API.
- Clients reserve a device and invoke actions such as clicks or text input.
- Users do not need to know which host owns the device or configure ADB, Xcode, cables, or test servers.
- The same API supports frontend tools, SDKs, CLIs, direct API clients, and automated systems.
## Four-Layer Architecture
- **Clients:** Web interfaces, SDKs, CLIs, and direct API calls.
- **Server:** Orchestrates device discovery, allocation, and test execution.
- Kafka distributes execution requests.
- Multiple runners consume work horizontally as demand grows.
- `occupy`, `assign`, and `release` provide distributed locking so tests cannot interfere with one another.
- **Agents:** Run on Android/Linux and iOS/Mac hosts, discover local devices, and forward server requests.
- **Devices:** Each device has a controller server and runner that execute actions on the physical phone.
## Why Nebula Replaced Appium
### Faster execution
- Nebula’s click and input operations were more than ten times faster than Appium in common cases.
- Much of Appium’s latency comes from `waitForIdle`, which waits for the screen to stabilize before acting.
- Disabling that wait narrows the difference to roughly two or three times.
- Appium prioritizes robustness against changing screens, while Nebula prioritizes immediate interaction for live remote control.
### Stateless operation
- Appium requires sessions that can take 15–40 seconds to initialize.
- Session startup becomes increasingly fragile and difficult to manage at scale.
- Nebula keeps device controllers pre-warmed and accepts stateless HTTP requests, eliminating session setup and reducing failure points.
### Company-specific customization
Because Nebula owns its driver specification, it can provide:
- A custom IME that preserves Korean and emoji input.
- Toss-specific signal triggers.
- App Center integration for installing pre-release builds.
- Built-in enforcement of internal security policies.
The driver uses Android ADB and UiAutomation, and Swift/XCTest on iOS. Its OpenAPI specification generates Go and TypeScript interfaces.
## Real-Time Interaction and Screen Mirroring
Nebula needed users to watch and control remote devices simultaneously, not merely replay predefined test steps.
### Android
- Instead of using `scrcpy` directly, the team adapted its encoding approach.
- Android’s `SurfaceControl` creates a virtual display.
- `MediaCodec` encodes the output as H.264.
- A broadcaster distributes the stream to multiple browser viewers.
### iOS
- iOS screen capture is constrained by USB access and cannot expose the display as freely as Android.
- Existing approaches such as QVH and Appium MJPEG did not support simultaneous viewing and interaction.
- Nebula developed a capture path inspired by QuickTime’s iOS capture mechanism without exclusively claiming the USB connection.
- Both platforms now use H.264 streaming and broadcasting, allowing the entire device farm to be viewed in a browser.
## Security and Compliance
- Centralization made it possible to apply consistent security requirements across every device.
- The team worked with Toss’s security organization to define mobile-device compliance standards.
- Policies are enforced at the platform and driver levels rather than relying on individual developers.
- Teams can therefore test on devices that already meet the company’s security requirements.
## Operating Hundreds of Devices
Nebula must keep hardware and software running continuously.
- **Hardware operations**
- USB cables, hubs, and power delivery had to be tested and designed for large-scale use.
- Physical failures still require human intervention.
- Redundancy is being introduced to reduce service interruptions.
- **Software operations**
- Controllers and mirroring processes are orchestrated across Mac mini and Linux hosts.
- Dead processes are automatically recovered.
- Server, agent, controller, and mirroring deployments are performed without interrupting user tests.
- Monitoring and observability cover devices, processes, and server performance.
## An Internal Ecosystem Built on the API
Nebula has evolved from a device provider into shared testing infrastructure.
- Web console for no-code interaction and test-step creation.
- SDKs for writing E2E tests.
- CLI support for terminals, CI/CD, and local AI agents.
- Automated log verification during device interaction.
- AI agents that plan and execute tests dynamically.
- Direct API access for teams with specialized needs.
Reported benefits include faster Appium migration, lower barriers to regression testing, and reducing manual verification from 30–40 minutes to under 10 minutes.
Nebula’s main lesson is that a stable, simple API can become the foundation for a much broader testing ecosystem. Centralizing device infrastructure, replacing unsuitable abstractions, and treating real-time operation, compliance, and reliability as first-class requirements enabled Toss to scale physical-device testing across the company.
The post explains how Netflix built a real-time service topology system capable of processing millions of network-flow records per second at production scale. Its core design combines streaming ingestion, reactive backpressure, physically separate data layers, and a distributed aggregation pipeline that resolves network intermediaries into meaningful service dependencies. The system favors slightly delayed but complete updates over stale batch data or incomplete results caused by dropping records.
## The Need for Real-Time Topology
- Traditional topology tools rely on hourly or daily batch processing, making their data outdated during incidents.
- Netflix combines:
- eBPF network flows
- IPC metrics delivered through Server-Sent Events
- Distributed tracing data
- These sources are stored in separate graph or columnar storage layers and can be queried independently or merged.
- The goal is near-real-time freshness, faster incident response, blast-radius analysis, and immediate change validation.
## Backpressure for Reliable Streaming
- Processing millions of flow records per second creates a risk that downstream systems will become overwhelmed.
- Common alternatives are inadequate:
- Unbounded queues eventually exhaust memory.
- Dropping records produces incomplete topology.
- Batch processing introduces unacceptable delays.
- Reactive streams propagate slowdown upstream:
- A graph database signals Stage 2.
- Stage 2 slows Stage 1.
- Stage 1 pauses Kafka consumption.
- Kafka retains the data until capacity returns.
- This allows the system to degrade gracefully during traffic spikes, garbage-collection pauses, or temporary storage slowdowns.
- Updates may be delayed by seconds or minutes, but the data remains substantially more complete than a dropped or hourly-processed stream.
## Physically Separate Topology Layers
Netflix keeps each data source in storage optimized for its characteristics:
- **Network layer:** eBPF flow logs provide broad coverage but limited application context.
- **IPC layer:** Application metrics offer detailed endpoint information but cover only instrumented services.
- **Tracing layer:** Parquet-based distributed traces show actual request paths but are sampled.
- Separate storage enables each layer to evolve and scale independently.
- Queries can run in parallel and merge results while preserving sub-second response times.
## Three-Stage Distributed Aggregation
The network layer uses a distributed pipeline to transform individual network hops into logical service dependencies.
- Cloud traffic commonly passes through load balancers, NAT gateways, API gateways, and proxies.
- Flow logs therefore show relationships such as:
- `App A → Load Balancer`
- `Load Balancer → App B`
- The useful topology must infer the logical dependency: `App A → App B`.
### Stage 1: Initial Flow Aggregation
- Consumes flow logs from Kafka across four regions.
- Filters invalid records.
- Groups data into five-minute windows.
- Creates initial aggregators for each window.
- Uses consistent hashing to distribute aggregators.
- Streams the results to Stage 2 through SSE.
### Stage 2: Intermediary Resolution
- Receives the initial aggregators from Stage 1.
- Groups flows by intermediary components.
- Resolves multi-hop network paths into application-level relationships.
- This prevents infrastructure components from dominating the resulting service graph.
## Engineering Trade-offs
- Streaming provides much fresher data than batch processing but introduces greater operational and conceptual complexity.
- Backpressure is essential for stability at Netflix’s scale, even though reactive pipelines are harder to reason about than synchronous systems.
- The architecture prioritizes reliable, complete topology updates over perfectly immediate processing.
- Production behavior differed substantially from local testing: consumers lagged, memory was exhausted, traffic became unevenly distributed, and garbage collection consumed significant resources.
Netflix’s approach demonstrates that large-scale real-time topology requires streaming ingestion, end-to-end backpressure, specialized storage, and staged aggregation. For similar distributed systems, the practical recommendation is to design explicitly for overload and partial slowdown rather than relying on unbounded buffering, dropped data, or stale batch snapshots.
LINE handles billions of messages daily, including highly sensitive personal data. While Kafka already provides TLS, authentication, and authorization, those controls do not protect message contents stored in brokers from privileged access. LY Corporation therefore introduced Kafka client-to-client end-to-end encryption, keeping payloads encrypted from producers through consumers while supporting large-scale traffic, flexible consumers, and minimal overhead.
## Limits of Kafka’s Existing Security Model
- TLS protects data in transit between clients and brokers.
- SASL authenticates clients before they connect.
- ACLs control which users or groups can publish to or consume from topics.
- These mechanisms primarily control access and communication channels; broker-stored payloads may still exist in plaintext.
- End-to-end encryption adds a defense-in-depth layer by encrypting data at production and decrypting it only at authorized consumers.
## Record-Level Encryption
- LY Corporation chose record-level rather than batch-level encryption.
- Batch encryption offers better compression and lower CPU overhead, but would require modifying Kafka client internals because standard extension points operate at the record level.
- Record encryption works with Kafka interceptors, serializers, and deserializers without modifying existing Kafka clients.
- Using standard APIs also improves compatibility with future Kafka upgrades, despite somewhat larger messages and reduced compression efficiency.
## DEK–KEK Key Architecture
- Payloads are encrypted with a symmetric AES-GCM data encryption key (DEK).
- The DEK is encrypted with an ECC-based key encryption key (KEK), using ECIES and the `secp521r1` curve.
- KEKs are managed through a key management service (KMS).
- Producers use the KEK’s public key, while authorized consumers obtain the private key from KMS.
- This hybrid approach:
- Avoids the high cost of encrypting large payloads with asymmetric cryptography.
- Keeps message size effectively independent of the number of consumers.
- Separates encryption and decryption permissions according to the least-privilege principle.
## Encrypted Kafka Message Structure
- **Key:** The existing Kafka message key remains unchanged for partitioning.
- **Header:** Contains the KEK identifier and the DEK encrypted with that KEK.
- **Body:** Contains the payload encrypted with the DEK.
- Embedding metadata directly in each message avoids dependencies on external databases or caches.
- Consumers identify the appropriate KEK, decrypt the DEK, and then decrypt the payload.
## Producer and Consumer Architecture
### Producer Encryption
- Interceptors generate or select the DEK and place the encrypted DEK in the message header.
- A wrapper serializer encrypts the serialized payload with the DEK.
- The interceptor and serializer share the DEK through `ThreadLocal`, since they run on the same thread.
- DEKs are cached for a limited period rather than regenerated and re-encrypted for every message, reducing asymmetric cryptographic overhead.
### Consumer Decryption
- Consumers retrieve authorized private KEKs from KMS.
- The deserializer reads the encrypted DEK from the header, decrypts it with the private KEK, and decrypts the payload.
- Consumers cache encrypted-DEK/plain-DEK pairs, allowing repeated messages from the same producer to bypass redundant DEK decryption.
- The existing deserialization process is wrapped so decryption occurs before normal deserialization.
### KMS Operations
- Topic owners generate and register KEK key pairs.
- Producers retrieve public keys, while authorized consumers retrieve private keys.
- New consumers must request access to the private key and receive approval from the topic owner.
- KMS manages key distribution, access control, and key rotation.
## Scaling Optimizations
### Shared KEKs
- Assigning a unique KEK to every consumer would cause message headers to grow with the consumer count.
- This would reduce Kafka batch sizes and increase network, CPU, and memory usage, especially for topics reaching up to one million messages per second.
- Multiple consumers therefore share a single KEK, keeping the header size constant.
- The trade-off is reduced per-consumer key isolation, mitigated through:
- KMS authorization controls.
- Mandatory periodic key rotation.
- Centralized key management by the topic owner.
### Zero-Downtime Migration
- During migration, encrypted and plaintext messages must coexist.
- The consumer deserializer checks whether encryption metadata exists:
- If headers are present, it decrypts the message.
- If headers are absent, it processes the message using the existing plaintext path.
- The migration sequence is:
- Deploy compatible consumers first.
- Enable producer encryption after all consumers support both formats.
- Monitor the plaintext-message ratio and complete the migration once it reaches zero.
- Producer encryption is intended to be enabled progressively rather than switched to 100% immediately, reducing the risk of unexpected performance or cryptographic failures.
## Practical Conclusion
Kafka’s built-in security controls should be supplemented with payload-level encryption when brokers handle highly sensitive data. A record-level AES-GCM design combined with DEK–KEK key wrapping, KMS authorization, caching, shared KEKs, fallback processing, and gradual rollout provides a practical balance between confidentiality, scalability, and operational continuity.
Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness.
## Defining Completeness at Datadog’s Scale
- Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers.
- The system must measure completeness:
- Across hundreds of services and ingestion paths
- For each individual customer
- In real time
- With enough detail to identify where degradation occurred
- Customer traffic may take different routes because of partitioning, isolation, and traffic patterns.
- Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points.
- The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents.
## Tracking Completeness by Pipeline Segment
- Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical.
- Pipelines are divided into segments representing steps within or between services.
- For example, intake-in to intake-out is one segment.
- Intake-out to processing-in is another.
- Each segment is measured independently, allowing engineers to locate degradation within a service or between services.
- Segment-level tracking also adapts to pipelines whose branches appear or disappear over time.
## Counting Creates and Acknowledgments
- When a payload enters a segment, the system records a create event.
- When it exits, the system records an acknowledgment using the payload’s unique identifier.
- Comparing creates with acknowledgments reveals whether payloads were lost in that segment.
- Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock.
- Each identifier has a state per segment:
- Created
- Acknowledged
- Acknowledged before the create event arrived
- Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering.
## Calculating End-to-End Completeness
- Segment completeness is the ratio of payloads exiting a segment to those entering it.
- For sequential services, overall completeness is calculated by multiplying segment ratios.
- Parallel branches require a different approach:
- Treating branches as one pipeline would make completeness wait for the slowest branch.
- Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes.
- In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%.
- Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower.
## Practical Conclusion
Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.
The AWS Weekly Roundup highlights tools aimed at making AI, infrastructure management, and cloud operations faster and more accessible. The main announcement is Amazon Connect Customer’s no-code Agentic CX designer, which lets business teams create governed AI customer experiences without relying on lengthy engineering backlogs. Other updates cover isolated serverless compute, AMI governance, guided migrations, AI-assisted security investigations, and broader community initiatives.
## Agentic Customer Experience Design
- Amazon Connect Customer launched the Agentic CX designer (NLX) in preview.
- The no-code canvas enables business teams to design, test, simulate, and deploy voice and digital self-service experiences.
- It combines agentic and deterministic AI within a governed workflow.
- AWS also introduced Live Sync in preview, allowing web or mobile interfaces to update in real time as customers speak or type.
- Customers could, for example, complete forms or open product pages while continuing a voice conversation.
## New AWS Infrastructure and Operations Features
- **AWS Lambda MicroVMs**
- Provides VM-level isolation with near-instant startup and resume times.
- Supports suspending and resuming execution for up to eight hours.
- Targets multi-tenant applications running user-generated or AI-generated code.
- **Amazon EC2 AMI Watermarks**
- Embeds custom identifiers in private AMIs.
- Watermarks persist across copies, Regions, and account shares.
- Works with Allowed AMIs and Declarative Policies to enforce approved-image usage.
- **AWS Outposts lifecycle management**
- Adds self-service configuration, quoting, ordering, subscription management, renewal, and decommissioning.
- A new quoting tool provides rapid cost estimates and identifies account or regional constraints.
## AI-Assisted Developer and Migration Tools
- **Amazon MSK AI Agent Skills** gives coding assistants such as Kiro, Claude Code, and Cursor operational guidance for Amazon MSK.
- It supports Kafka sizing, configuration, troubleshooting, monitoring, and migrations to MSK Express.
- **Amazon OpenSearch Service Migration Assistant** now offers agent-guided migrations from Solr, Elasticsearch, and OpenSearch to managed clusters or OpenSearch Serverless.
- The migration tooling adds live traffic capture and replay for Solr workloads.
## AI-Powered Security Investigations
- Amazon GuardDuty’s AI-powered investigations entered preview.
- It analyzes findings, account context, related activity from the previous 90 days, knowledge graphs, and threat intelligence.
- Investigations produce confidence-scored assessments, MITRE ATT&CK classifications, and recommended actions to help distinguish real threats from benign activity.
## Open Governance and AWS Community Updates
- Oracle announced a community governance model for MySQL, including four non-Oracle seats on a new Steering Committee and a public GitHub presence.
- AWS supports the initiative and contributes fixes upstream.
- AWS Certification holders can renew eligible Associate and Professional certifications for an additional year through selected Skill Builder training and hands-on labs instead of retaking an exam.
- The 2026 All Builders Welcome Grant offers selected early-career builders conference admission, airfare, and lodging for AWS re:Invent.
AWS’s latest releases broadly point toward more self-service cloud management: business users can design AI experiences, developers can receive operational guidance from coding assistants, and teams can apply stronger controls to infrastructure and security workflows.
Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic.
## Stream Router’s Role in Datadog’s Metrics Pipeline
- Datadog processes more than a hundred trillion events per day.
- Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint.
- It serves both producers and queriers but does not process Kafka messages itself.
- Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential.
## From Configuration File to Control Plane
- In 2016, routing was managed through a small configuration file distributed to services.
- As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts.
- Stream Router replaced this workflow with:
- A centralized gRPC service
- API-managed routes
- Automated, gradual rollouts
- The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory.
- This eventually became a bottleneck as routing tables and operational changes grew larger.
## Why the Key-Value Model Stopped Scaling
- Routes reference streams and sharding strategies, while rules reference routes.
- These relationships are inherently relational and require cross-entity validation.
- The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code.
- Some operations exceeded FoundationDB transaction-size limits.
- Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips.
- The fundamental problem was the data model and application logic, not simply the choice of database.
## Designing the New Storage Architecture
- The team redesigned the schema manually before using AI tools.
- The relational model introduced explicit foreign keys between:
- Streams
- Sharding strategies
- Routes
- Rules
- PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model.
- DuckDB was selected for the read path because:
- It is embeddable and suitable for snapshot-based serving
- It supports array columns
- Its SQL dialect is closely compatible with PostgreSQL
- Shared query logic could therefore work across both storage engines.
## AI-Assisted Refactoring
- Claude and Cursor were used to accelerate a systematic, test-driven migration.
- For each method, developers supplied:
- The old implementation
- The new schema
- A failing test
- AI generated an initial implementation, while tests determined whether it was correct.
- The models assisted with method-level refactoring rather than autonomously designing the architecture.
- Human expertise remained central to schema design, migration strategy, and evaluating system-level risks.
## Foundations for a Safe Migration
- The migration benefited from infrastructure already present at Datadog.
- Stream Router’s storage layer was isolated behind an internal `Controller` interface.
- This modularity helped contain storage changes and enabled incremental refactoring.
- Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued.
The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.
Security Insights needed a 10x throughput increase to scan all customers more frequently and detect risks sooner. The existing system was overwhelmed by Kafka backlogs, slow processing, database inefficiencies, and API timeouts. Cloudflare improved capacity by introducing parallel and lane-based processing, optimizing bulk database writes, and addressing regional latency between its API and database.
## Scaling Kafka Processing
- Scans are scheduled and published to Apache Kafka.
- Go-based checker services consume these messages, inspect accounts, zones, and DNS records, and send findings to an internal API.
- Kafka’s partition ordering limits each consumer group to one active consumer per partition.
- Slow messages could block all subsequent messages in the same partition.
- Adding partitions was avoided because it would increase resource usage for shared Kafka brokers.
## Introducing Parallel Processing
- Checkers were changed to consume messages in batches.
- Each message in a batch is processed concurrently in its own goroutine.
- This increased throughput without requiring additional Kafka partitions.
- The trade-offs were higher memory usage and potentially more work to repeat after a process crash.
## Separating Slow and Fast Work
- Some scans took seconds or milliseconds, while unusually large accounts or zones could take minutes or hours.
- These slow messages caused head-of-line blocking for faster work.
- Consumer groups and checkers were split into:
- A fast lane for predictable, short-running scans
- A slow lane for messages expected to require substantially more time
- Fast-lane consumers skipped slow messages, allowing normal scans to continue without delay.
## Optimizing Postgres Writes
- The API originally executed one insert/upsert transaction per insight.
- A request containing up to 500,000 insights could therefore generate hundreds of thousands of database round trips.
- Bulk insertion with `COPY` into a temporary table was tested but caused bloat in Postgres system tables.
- The final hybrid approach used:
- `UNNEST` for smaller batches
- `COPY` for batches above a configured threshold
- This delivered millisecond-level performance for small writes and completion within seconds for very large writes.
## Diagnosing API Timeouts
- Client-side timeouts increased as scan volume grew.
- Checkers sometimes spent 20–90% of their processing time waiting on a single API call.
- Throughput initially rose but then deteriorated under heavy load.
- The root cause was network latency:
- Postgres was hosted in Portland, Oregon.
- The API ran active-active in Portland and Amsterdam.
- Requests routed to Amsterdam incurred roughly 50 milliseconds of network round-trip latency.
- Amsterdam database queries held client connection-pool connections much longer—nearly three seconds on average versus about 10 milliseconds in Portland.
- The connection pool became exhausted, causing requests to wait for available connections and creating uneven Kafka lag across partitions.
Cloudflare’s results came from improving the full processing pipeline rather than relying on a single infrastructure change. Parallelize message handling, isolate slow workloads, batch database writes, and place latency-sensitive services close to their databases to achieve large throughput gains and more frequent security scanning.
Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level.
## Cassandra and the Wide-Partition Problem
- Cassandra provides:
- High-throughput, low-latency reads and writes
- Cost-effective operation at scale
- Strong operational familiarity within Netflix
- TimeSeries datasets accumulate events over time, creating potentially very wide partitions.
- Wide partitions can lead to:
- Read latencies increasing from milliseconds to seconds
- Request timeouts
- Garbage-collection pauses
- High CPU utilization and thread queueing
- Scaling Cassandra clusters can help, but Netflix sought more targeted solutions.
## Initial Time-Based Partitioning
- TimeSeries divides data into discrete time slices to keep partitions manageable.
- This structure also makes it efficient to:
- Query data by time
- Drop old data without creating large tombstone problems
- At dataset creation, users provide expected workload characteristics.
- Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings.
## Why Static Provisioning Falls Short
- Workloads may be unknown or inaccurately estimated during initial provisioning.
- Traffic patterns, client behavior, and product needs can change over time.
- A small number of TimeSeries IDs may generate far more events than the rest.
- Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical.
## Repartitioning Entire Time Slices
- Cassandra introspection tools, such as `nodetool tablehistograms`, expose partition-size distributions.
- Netflix added a background worker that:
- Monitors partition histograms for time slices
- Publishes observations through a Cassandra virtual table
- Detects partitions that are too large or too small
- Calculates a new partitioning adjustment factor
- Target partition density is typically between 2 MiB and 10 MiB, depending on workload.
- The worker updates the strategy for future time slices. For example, it may expand a `time_bucket` interval from 60 seconds to 604,800 seconds when partitions are too small.
- This approach reduced read latency and timeouts caused by thread queueing.
- Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions.
## Handling Isolated Problem IDs
Netflix considers several responses when only some IDs are problematic:
- **Do nothing:** Appropriate when wide partitions do not affect application-level metrics.
- **Partial returns:** Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness.
- **Block IDs:** Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system.
- These options are inadequate when valid, important IDs must return all their data despite generating large partitions.
## Dynamic Partitioning per ID
Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table.
The asynchronous pipeline has three stages:
- **Detection:** The read path identifies partitions that exceed a configured size threshold.
- **Planning and splitting:** The system asynchronously plans and executes splits into appropriately sized partitions.
- **Serving reads:** Once splits are available, read requests are transparently rerouted to them.
During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as:
- The Cassandra time-slice table
- The affected TimeSeries ID
- The existing time and event bucket
- Whether the partition is immutable
- A version identifier
## Practical Recommendation
Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.
Netflix’s growing use of machine learning across personalization, Studio, payments, advertising, and other domains has created a fragmented ecosystem of tools and metadata. The Metadata Service (MDS) addresses this problem by building a Model Lifecycle Graph that connects models, features, pipelines, experiments, datasets, and ownership information. Its goal is to make ML assets discoverable, understandable, and reusable across organizational boundaries.
## A Fragmented Machine Learning Landscape
- Netflix ML has expanded from personalization into areas such as:
- Studio production and post-production
- Fraud detection and payment optimization
- Advertising and real-time targeting
- Each domain uses different technologies, metrics, and organizational structures.
- Valuable assets often remain isolated in specialized systems.
- For example, Studio-generated content embeddings could support:
- Contextual ad matching
- Episodic merchandising
- Recommendations based on tone, topic, or mood
- Practitioners struggle to answer basic questions because relevant information is split across:
- Model registries
- Pipeline orchestrators
- Experimentation platforms
- Feature stores
- Dataset systems
- This fragmentation makes discovery, lineage tracking, impact analysis, and ownership difficult.
## The Challenge of Connecting ML Infrastructure
- MDS must unify metadata from many independent systems, including:
- Pipeline execution and transformation data
- Model versions, artifacts, deployments, and staleness
- A/B test configurations
- Feature definitions and usage
- Dataset creation and discovery
- User, team, and organization information
- These systems use different identifiers, formats, and conceptual models.
- The core challenge is transforming heterogeneous metadata into a common entity model and connected graph—not merely creating a consolidated user interface.
## The Model Lifecycle Graph
- Netflix’s Metadata Service indexes ML-related assets and materializes relationships between them.
- It supports real-time metadata ingestion and cross-domain questions such as:
- Which experiments use a particular model?
- Which models depend on a feature?
- What data sources feed a model?
- Who owns each part of the workflow?
- The graph is intended to make every ML asset discoverable and reusable regardless of its originating team or business domain.
## Core Concepts and Vocabulary
- **Component:** Any uniquely addressable object identified by an AIP URI, such as:
- `aip://model/registry/ranking-v5`
- `aip://user/identity/alice`
- `aip://pipeline/orchestrator/weekly-training`
- **Entity:** A component enriched with properties such as name, description, creation date, and ownership.
- **Entity type:** A group of entities sharing the same data shape and required properties.
- **Domain:** An abstract interface for a category of ML assets, such as Models or Pipelines.
- **Provider:** A concrete backend implementation of a domain, such as Netflix’s internal model registry.
- Separating domains from providers allows multiple systems to implement the same interface without changing how consumers interact with MDS.
- URI-based addressing gives services a consistent way to reference assets and resolve them to connected metadata.
## From Events to a Queryable Graph
- MDS receives metadata events through Kafka and AWS SNS/SQS.
- Source systems emit lightweight events containing an event type and resource identifier.
- For example, a model registry might emit a `model_instance_created` event with the new instance’s ID.
- This keeps event producers simple while allowing MDS to enrich events, construct entities, and infer relationships such as connections between models and A/B tests.
The Model Lifecycle Graph provides Netflix with a common layer for connecting previously isolated ML systems. By standardizing identifiers, entities, domains, and providers, MDS can support cross-domain discovery, lineage, impact analysis, and collaboration at scale.
Bits AI SRE improved in isolated scenarios but lacked a way to detect regressions across the broader range of production incidents. The team found that tool-level tests and live replays could not capture failures caused by multi-step reasoning or changing telemetry. They built a replayable evaluation platform combining realistic investigation labels, scalable orchestration, and longitudinal performance tracking.
## Subtle Regressions from Well-Intentioned Features
- Adding a monitor’s service name to the agent’s initial context improved some internal investigations.
- Across broader scenarios, it introduced irrelevant signals that confused the agent and degraded unrelated investigations.
- Because there was no representative evaluation set, the team could not measure the change’s wider impact before internal misses exposed it.
- The incident demonstrated the need to evaluate every change across diverse investigation types.
## Limits of Tool Tests and Live Replay
- Testing tools individually failed to capture errors caused by incorrect interactions between valid tool outputs.
- Live investigation replay was difficult to scale because:
- Results were not consistently aggregated.
- Production environments changed.
- Telemetry expired, making investigations unreplayable.
- Standard evaluation frameworks assumed clean inputs and static datasets, unlike agents operating over production telemetry.
- The team needed controlled, offline replay of realistic end-to-end investigations.
## Evaluation Labels and World Snapshots
- Each label represents one production-style investigation.
- It contains:
- **Ground truth:** the issue’s actual root cause.
- **World snapshot:** the queries and signals available when the issue occurred.
- The agent is never shown the root cause directly; it must reason from the preserved signals.
- Labels must cover varied technologies and failure modes, including:
- Kubernetes pod failures
- Kafka lag
- Bad-code deployments
- Complex multi-service business failures
- A narrow or overly clean dataset would make performance appear better than it really is.
## Orchestrating Evaluations at Scale
- The platform runs Bits against labels, scores the outcomes, and tracks quality over time.
- It supports comparisons across:
- Investigation categories
- Model variants
- Configuration versions
- Evaluation runs
- The architecture consists of a shared label set, an orchestration layer, and reporting infrastructure.
- This allows teams to determine whether improvements in one domain, such as Kafka, regress another, such as Kubernetes.
## Scaling Label Creation
- The team initially created labels manually from Datadog alerts.
- Manual labeling provided early coverage but consumed engineering time and remained far from representative.
- They embedded label generation into Bits itself:
- Customer feedback and investigation data are used to derive root causes.
- Relevant queries are preserved as the world snapshot.
- Each user interaction becomes a potential evaluation case.
- This increased label creation rates by an order of magnitude and allowed coverage to grow with product usage.
## Agent-Assisted Validation
- Early labels required extensive human review, especially when feedback was ambiguous or reconstructed signals were uncertain.
- As ingestion grew, manual review became a bottleneck.
- Bits was then used to assist with validation by aggregating related signals, identifying relationships, and resolving ambiguous feedback before human review.
## Practical Conclusion
Reliable agent improvement requires more than testing individual tools or replaying live incidents. A representative, production-derived label set combined with reproducible end-to-end evaluations makes regressions visible and enables safer iteration.
Video search is difficult because it must combine many kinds of information—characters, scenes, dialogue, labels, and embeddings—across enormous volumes of footage. The post argues that solving this problem requires a distributed pipeline that separates reliable ingestion, computationally intensive data fusion, and low-latency search indexing. Temporal bucketing, hybrid ranking, and deduplication turn billions of model outputs into searchable moments for editors.
## Why Video Search Is Complex
- Video contains multiple overlapping modalities, each analyzed by specialized models.
- Models produce different outputs, including:
- Text labels such as characters or objects
- Scene classifications
- High-dimensional embedding vectors
- Time ranges with varying boundaries
- Overlapping model timelines must be synchronized into a chronological representation.
- A 2,000-hour archive may contain more than 216 million frames, expanding to billions of records after multimodal processing.
- Search must avoid returning thousands of redundant clips from continuous shots.
- Ranking therefore combines:
- Symbolic text matching for precision and interpretability
- Semantic vector similarity for contextual relevance
- Clustering and deduplication to identify the best moments
- Sub-second response times are essential because delays interrupt editors’ creative workflows.
## Three-Stage Ingestion and Fusion Pipeline
### Transactional Persistence
- Raw model annotations are ingested through highly available pipelines.
- Apache Cassandra stores the annotations with an emphasis on:
- Data integrity
- Distributed availability
- High write throughput
- An annotation can include a type, nanosecond time range, embedding vector, label, and confidence score.
### Offline Data Fusion
- After persistence, Apache Kafka publishes an event that starts asynchronous processing.
- The offline pipeline performs expensive temporal intersections without slowing ingestion or search.
- Model outputs are normalized into fixed one-second time buckets.
- The fusion process:
- Maps continuous detections into discrete intervals
- Intersects annotations sharing a bucket
- Combines them into unified records
- Writes the enriched records back to Cassandra
- For example, a “Joey” character detection from seconds 2–8 can be combined with a “kitchen” scene detection from seconds 4–9 to create a fused record for the 4–5 second interval.
- Each fused record retains links to the original annotations and source asset.
### Real-Time Search Indexing
- Enriched buckets are later sent from Cassandra to Elasticsearch.
- Upserts use a composite key consisting of the asset ID and time bucket.
- If a bucket already exists, it is updated rather than duplicated.
- This creates one consistent record for each second of footage while allowing new model results to be incorporated.
The overall recommendation is to treat multimodal video search as a distributed data-fusion problem rather than a single-model retrieval task. Decoupling ingestion, offline processing, and indexing allows the system to handle massive archives while preserving reliable data capture and fast, context-rich search.
LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation.
## Large-Scale LINE Ads Data Pipelines
- The data pipeline supports:
- Real-time advertising-event processing
- Abuse and validity checks
- Machine-learning systems and model training
- Analytics and system integration
- Advertiser reporting
- The platform must handle hundreds of billions of events per day and hundreds of thousands per second.
- It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery.
- The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added.
## Limitations of Spark on YARN
- Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components.
- Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity.
- JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features.
- Applications became tightly coupled to the Hadoop infrastructure.
## How Spark on Kubernetes Works
- Kubernetes replaces YARN as the cluster manager.
- Spark drivers and executors run as separate Kubernetes pods.
- In cluster mode:
- `spark-submit` requests a driver pod.
- Kubernetes schedules the driver on an appropriate node.
- The driver creates a `SparkContext`, builds the DAG, and requests executors.
- Executors run as independent pods with individually allocated CPU and memory.
- The driver divides the DAG into stages and distributes tasks to executors.
- Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured.
## Advantages over YARN
- **Containerized execution:** Docker images package application dependencies, improving reproducibility and CI/CD integration.
- **Infrastructure independence:** Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster.
- **Simpler autoscaling:** Kubernetes can scale pods and integrate with cloud VM autoscalers.
- **Unified platform:** Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster.
- **Governance and isolation:** Namespaces, `ResourceQuota`, and RBAC provide flexible team-level controls.
- **Operational automation:** Helm, ArgoCD, GitOps, and rolling updates enable more automated application management.
## LINE Ads’ Kubernetes-Based System
The platform is organized into four layers:
- **Deployment layer**
- GitHub Actions runs CI workflows based on repository events.
- ArgoCD monitors desired and deployed states and supports easier rollback and synchronization.
- **Compute layer**
- Kubeflow’s Spark Operator deploys applications through the `SparkApplication` Kubernetes custom resource.
- Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling.
- LogSender forwards pod logs to OpenSearch.
- ClusterMonitoring sends Prometheus metrics to the company’s monitoring system.
- **Storage layer**
- Kafka provides high-throughput, low-latency storage for real-time advertising actions.
- Hadoop remains available for large-scale, long-term analysis.
- **Monitoring layer**
- Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging.
The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.
The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability.
## Domain: Products, Catalogs, and Data Ingestion
- **Products** are individual seller-listed items, potentially with different prices and shipping conditions.
- **Catalogs** group products representing the same model or product type and provide derived value such as:
- Real-time lowest prices
- Unit-price metrics such as price per 100 ml
- **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data.
- Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service.
## The Verification Loop
- The goal was not merely to find errors, but to help developers understand and correct them quickly.
- Inputs had to be identical for both systems, such as:
- The same IDs
- The same time-based snapshot
- The same product files
- Outputs were compared according to system type:
- API response objects
- Database update values
- Final registered product data
- The general loop consisted of:
- **Trigger:** Database changes, developer requests, or file arrivals
- **Execution:** Send identical inputs to legacy and new systems
- **Comparison:** Apply logic suited to reads, updates, or end-to-end flows
- **Processing:** Store detailed differences and produce real-time statistics
- **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat
## Query Logic Verification
- The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior.
- CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states.
- The verifier made dual API calls and compared legacy and new responses field by field.
- Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class.
- If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison.
- This helped distinguish real implementation defects from harmless ordering differences.
- Kafka isolated verification traffic from production services while handling large event volumes.
- Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation.
- ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared.
- Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute.
- Because both APIs were called in parallel, the same pipeline also measured and compared their response times.
## Update Logic Verification
- The second case involved recalculating catalog statistics whenever product or catalog data changed.
- Unlike read verification, this process tested state transitions and asynchronous updates.
- When CDC detected a relevant change:
- The new statistics logic calculated an expected result.
- The verifier compared it with the result actually written by the legacy logic.
- Recursive Map-based comparison checked deeply nested statistics fields.
- To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module.
## Handling Asynchronous Lag
- Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed.
- The team introduced an **N-attempt retry queue**:
- Temporarily inconsistent events were requeued.
- Only differences that remained after several retries were treated as genuine defects.
- The verifier remained a separate process rather than being embedded in the production statistics stream.
- This avoided adding load or latency to the existing processing pipeline while preserving independent verification.
## ETL Batch Verification for Missing Triggers
- Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred.
- During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition.
- As a result, some statistics remained stale without generating any comparison event.
- To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks.
The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.
Karrot’s Taxonomy team built an LLM-powered system to classify marketplace posts, group activities, and local businesses into a shared category and attribute structure. After finding that manually managed taxonomies and event-only pipelines were difficult to scale, they created a configurable Taxonomy Management System using Dataflow/Beam, BigQuery, Kafka, and multiple LLM strategies. The system emphasizes scalable inference, rapid evaluation, multilingual support, and continuous taxonomy expansion.
## What a Taxonomy Is and Why It Matters
- A taxonomy is a hierarchical category system, such as `Outerwear > Padding/Down > Long Padding`.
- It can also include attributes that describe an item’s characteristics:
- Category: long padding
- Attributes: brand=Nike, color=black, material=polyester
- A consistent taxonomy acts as a shared language across:
- Search, including parent and child-category expansion
- Recommendations and diversity controls
- Advertising and targeting segments
- Analytics and machine-learning features
## Karrot’s Taxonomy Challenges
- Karrot manages roughly 1,400 marketplace categories across up to three levels.
- Users are not required to manually select highly detailed categories because that would increase posting friction and produce unreliable labels.
- Earlier systems used a Golang Kafka consumer to receive posting events and extract categories with an LLM.
- This approach had several limitations:
- Taxonomy definitions were managed separately by different teams.
- Categories alone could not express useful properties such as season or material.
- Batch processing and backfilling were difficult.
- Expanding to data sources outside Kafka was inconvenient.
- Quality monitoring and failure handling were insufficient.
- Changes to prompts or models required slow offline and online experiments.
## The Taxonomy Management System
- The new system centrally manages taxonomies, performs LLM-based classification, delivers category and attribute results, and monitors quality.
- Dataflow with Apache Beam was selected because it supports:
- Parallel, high-throughput LLM inference
- Both streaming and large-scale batch processing
- Existing team expertise compared with alternatives such as Spark or Flink
- BigQuery serves as the source of truth for inference results.
- Analysts and data scientists can query results directly.
- Online consumers can receive results through Kafka sinks into the internal feature platform.
## Configuration-Driven and Extensible Design
- Taxonomy definitions are stored in YAML, allowing different services and category trees to use the same framework.
- Pipeline settings, worker sizing, Kafka topics, and BigQuery destinations are also configured through YAML.
- LLM models and inference strategies can be selected through configuration, including:
- Primary and evaluation models
- Single-shot or two-stage categorization
- Attribute extraction modes
- Evaluation sampling ratios
- The system is designed for multilingual taxonomies.
- Large translation jobs are divided into chunks.
- One LLM generates translations and another validates consistency and naturalness.
- A depth-first traversal carries parent-category translations into child-category prompts to maintain terminology consistency.
## Creating and Expanding Taxonomies with LLMs
- New taxonomies are developed by researching established taxonomies and generating candidate trees from real data.
- Existing taxonomies are expanded by:
- Classifying sampled data against the current taxonomy
- Asking the LLM to suggest categories for unsuitable examples
- Merging similar suggestions using LLM similarity judgments
- Promoting sufficiently strong candidates for review
- Candidates undergo two evaluations:
- Whether the originating examples are correctly assigned to the new category
- Regression testing comparing classifications under the old and new taxonomies
- This process enabled the team to move beyond the existing 1,400 three-level categories and create taxonomies with more than 10,000 categories and six or more levels.
## LLM Categorization Strategies
The team supports multiple strategies because the best approach depends on the model and taxonomy size:
- **Single shot:** Provide all categories and ask the model to choose one.
- **Hierarchical classification:** Select the best category at each depth, then continue through the chosen branch.
- **Two-stage tournament:** Split categories into chunks, select candidates from each chunk, and run a second selection among those candidates.
- Categorization and attribute assignment are separate modular Beam `DoFn` stages:
- `Article → Category inference → Attribute inference`
- New approaches can be added as interchangeable strategies without redesigning the whole pipeline.
## Evaluation with LLM-as-a-Judge
- A sample of production data is processed by multiple different models.
- Their labels are combined into a ground-truth label, generally through majority voting.
- Each model’s output is compared against that ground truth.
- Accuracy changes are tracked whenever the team modifies:
- The LLM model
- Prompts
- Pipeline structure
- Categorization or attribute strategies
- The ground-truth method varies depending on whether the task involves:
- A single category
- Multiple categories
- Multi-label attributes
- Category quality is measured as a precision-at-one-style accuracy: the primary model’s category must match the ground-truth category.
- Attributes are evaluated with precision and recall because a post can legitimately contain multiple attribute-value pairs.
The main recommendation is to treat LLM classification as a production data pipeline rather than a one-off prompt: centralize taxonomy management, support both batch and streaming execution, make inference strategies configurable, and build automated evaluation and monitoring into the system from the beginning.
