Apache Iceberg | Techlist.io

awsAug 3, 20262 min readCurated summary

AWS Weekly Roundup: Price reduction of GPT models in Bedrock, CloudWatch managed collectors for Prometheus metrics, and more (August 3, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights major updates in AI pricing, observability, multicloud networking, identity resilience, and data lakes. The biggest change is an up to 80% price reduction for OpenAI GPT‑5.6 Luna models in Amazon Bedrock, alongside several managed services that reduce infrastructure and operational overhead. ## Lower Bedrock Pricing for GPT‑5.6 - Effective July 30, GPT‑5.6 Luna inference prices dropped by 80%. - New pricing is: - $0.20 per million input tokens - $1.20 per million output tokens - GPT‑5.6 Terra prices decreased by 20%. - The reductions apply automatically and require no customer action. ## Managed Prometheus Monitoring in CloudWatch - Amazon CloudWatch now provides fully managed Prometheus collectors. - Customers can collect metrics from: - Amazon EKS - Amazon EC2 - Amazon ECS - Amazon MSK - Amazon OpenSearch Service - This removes the need to deploy and maintain custom Prometheus scraping agents. ## Private Multicloud Connectivity with OCI - AWS Interconnect for Oracle Cloud Infrastructure is now generally available. - It enables resilient, scalable private connections between AWS and OCI. - Traffic avoids the public internet, improving security, performance, and reliability for multicloud workloads. ## Multi-Region IAM Identity Center - IAM Identity Center can now replicate its built-in Identity Center directory across Regions. - During a primary-Region disruption, users can continue accessing AWS accounts through provisioned entitlements in additional Regions. - Previously, multi-Region support was limited to deployments using external identity providers. ## Variant Support in S3 Tables - Amazon S3 Tables now supports Apache Iceberg V3’s Variant data type. - Variant provides native, high-performance support for semi-structured data. - Suitable use cases include IoT sensor data, application logs, and schema-flexible payloads without storing everything as JSON blobs. ## Additional AWS Resources - New AWS CLI single-line commands simplify installation and upgrades across platforms and CI environments. - A deployment guide covers running Moonshot AI’s Kimi K3 on SageMaker HyperPod and Amazon EKS. - Amazon MSK Express brokers can deliver Kafka data to Apache Iceberg streaming tables on S3 Tables, with throughput of up to 10 GB/s. - AWS Summits and AWS Community Days offer upcoming opportunities for cloud and AI learning and networking. AWS users should review the new Bedrock pricing, consider managed CloudWatch collectors to reduce monitoring maintenance, and evaluate the multicloud, identity, and Iceberg updates for architectures requiring greater resilience and scalability.

Read original

Apache Iceberg Amazon Bedrock Apache Kafka Amazon Cloudwatch+4

netflixJun 19, 20263 min readCurated summary

The Evolution of Cassandra Data Movement at Netflix

Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency. ## Casspactor’s Role and Limitations - Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups. - It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions. - Its metadata view depended on several independent systems, each with different failure modes and update schedules. - Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed. - Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second. ## Constraints for Higher-Level Data Abstractions - Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations. - Large or skewed partitions caused executor memory failures and out-of-memory crashes. - Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing. - Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity. - Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes. - The monolithic connector could not serve as a reusable foundation for specialized connectors. ## Direct S3 Metadata as the Source of Truth - The new design reads backup metadata directly from the S3 storage layer. - This removes the chain of external metadata dependencies. - Backup existence and completeness are determined from the files that actually contain the data. - Direct backup access also enables restoration of historical backup states. ## A Layered Connector Architecture - The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format. - It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames. - A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector. - Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics. - Improvements to the shared reading engine automatically benefit every connector. ## Performance and Operational Improvements - Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions. - Reduced data shuffling helps prevent memory failures on large datasets. - Direct DataFrame output eliminates costly intermediary Iceberg tables. - Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning. - Fewer dependencies improve reliability and make the system easier to maintain. Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.

Read original

Apache Iceberg Amazon S3 Apache Spark Apache Cassandra+3

cloudflareMay 28, 20263 min readCurated summary

How we built Cloudflare's data platform and an AI agent on top of it

Cloudflare built Town Lake to unify data scattered across production databases, analytics systems, streams, and object storage behind one governed SQL interface. The platform combines Trino, Iceberg on R2, DataHub, and custom access-control and PII-detection services to make data fresher, more discoverable, and safer to use. Skipper extends Town Lake with a natural-language AI interface intended to provide fast, accurate, and auditable answers without requiring users to write SQL. ## The Data Sprawl Problem - Cloudflare processes over a billion events per second across a network spanning more than 330 cities and 120 countries. - Relevant data was distributed across: - Postgres - ClickHouse - BigQuery - Kafka - Google Cloud Storage and R2 - Numerous pipelines and production databases - Users needed separate credentials, query languages, retention expectations, and system knowledge for each source. - Sampled analytics data worked for dashboards but was unsuitable for billing, usage calculations, and security investigations. - External vendors created cost and dependency concerns. - Important data was difficult to discover because table locations, schemas, joins, and customer-ID mappings depended on tribal knowledge. - Data infrastructure had historically been treated as a back-office service rather than core company infrastructure. ## Goals for the New Platform Cloudflare wanted a single place where authorized employees could answer questions about customers, traffic, billing, security events, and support activity. - Support both: - Fresh, accurate, unsampled data for billing and investigations - Fast, downsampled data for dashboards and exploration - Provide built-in governance: - Automatic PII detection - Sensitive tables locked down by default - Auditable access - Time-limited permission grants - Build the system using Cloudflare’s own products, including R2, Workers, Access, and Workflows. - Eventually let employees ask questions in plain English rather than requiring SQL knowledge. - That natural-language interface became Skipper. ## Town Lake’s Lakehouse Architecture Town Lake is a lakehouse: a query engine combines data from object storage and operational systems while a metadata layer makes the data behave like a unified database. - **Trino** serves as the query engine. - A single query can join Postgres, ClickHouse, and Iceberg tables stored on R2. - Trino pushes filters into source systems and combines results without requiring intermediate materialization. - **R2 Data Catalog and Apache Iceberg** store warm and cold data. - Iceberg provides schema evolution, time travel, partition evolution, and compaction. - Data can be rolled from per-minute to hourly and eventually daily granularity as it ages. - Older data becomes cheaper to store while remaining queryable. - Parquet files on R2 cost less than retaining equivalent data in an OLAP database. - **DataHub** provides the metadata catalog. - It stores table and column descriptions, owners, lineage, and glossary terms. - Users can discover what a table contains, which teams maintain it, and how it relates to upstream and downstream data. ## Access Control and Privacy - **Lifeguard** manages access policies. - Rules are stored in D1. - User and group memberships are retrieved dynamically from Cloudflare’s internal access-management system. - Lifeguard produces JSON policies that Trino reads over HTTP. - It also supplies access information to Skipper and the Gateway, allowing users to be blocked before queries execute. - **Skimmer** continuously scans tables for PII. - It samples rows from columns across the data platform. - Workers AI classifies whether columns contain personally identifiable information. Cloudflare’s overall approach is to combine unified querying, durable low-cost storage, rich metadata, and policy enforcement so data can be broadly useful without sacrificing accuracy or governance.

Read original

Apache Iceberg AI Agents Sql Data Governance+4

awsMay 18, 20263 min readCurated summary

AWS Weekly Roundup: AWS Transform at 1 year, Claude Platform on AWS, EC2 M3 Ultra Mac instances, and more (May 18, 2026) | Amazon Web Services

AWS’s May 18, 2026 roundup highlights the rapid expansion of AI-assisted modernization, cloud infrastructure, and developer tooling. AWS Transform reached its first anniversary after processing billions of lines of code and migrating hundreds of thousands of servers, while new services provide direct Claude access, stronger Mac development hardware, improved analytics, and automated security remediation. The roundup also covers multicloud connectivity, AI research funding, community events, and startup credits. ## AWS Transform Reaches Its First Anniversary - AWS Transform launched a year earlier for modernizing .NET, mainframe, and VMware workloads. - New capabilities include: - Custom AWS-managed or user-defined code transformations - Language and framework upgrades - Performance optimization and code analysis - Full-stack Windows modernization - Mainframe reimagining and automated testing - In its first 12 months, customers: - Migrated hundreds of thousands of servers - Saved more than 1.6 million hours - Processed over 4.5 billion lines of code - AWS Transform agents are now available in Kiro, Claude, Cursor, and Codex. ## Major Product Launches - **Claude Platform on AWS:** Provides Anthropic’s native Claude APIs, console, and beta features through existing AWS accounts and billing. Anthropic operates the service, and customer data is processed outside the AWS security boundary. - **EC2 M3 Ultra Mac instances:** Use Apple M3 Ultra Mac Studio hardware with a 28-core CPU, 60-core GPU, 32-core Neural Engine, and 256 GB of unified memory. Compared with M4 Max instances, they offer more CPU, GPU, Neural Engine cores, and memory for parallel Xcode simulators and on-device machine learning. - **Redshift RG instances:** AWS Graviton-based instances run workloads up to 2.4 times faster than previous RA3 instances while costing 30% less per vCPU. They include a vectorized engine for Apache Iceberg and Parquet data. - **Bedrock Advanced Prompt Optimization:** Compares original and optimized prompts across as many as five models, helping users improve performance or migrate between models. - **AWS Security Agent repository scanning:** The preview feature analyzes entire codebases and generates fixes linked to specific files and lines. It is free for existing Security Agent customers during the preview. - **AWS Interconnect for OCI:** Enables private, resilient multicloud connections to Oracle Cloud Infrastructure. The same open specification already supports Google Cloud and is planned for Azure later in 2026. ## Research, Community, and Startup Updates - AWS is investing $110 million in the Build on Trainium program to give university researchers access to purpose-built AI chips. - Research at institutions including UC Berkeley, MIT, and Carnegie Mellon is open source, allowing improvements to benefit the broader developer community. - AWS Community Days 2026 will take place globally, with agendas organized by local developer communities. - The Kiro Startups Credit program has reopened, offering eligible organizations up to one year of Kiro Pro+ credits applied through AWS accounts. AWS’s latest updates emphasize managed AI tools, faster specialized hardware, automated modernization, and broader support for developers and researchers. Organizations can particularly benefit by evaluating AWS Transform, Bedrock prompt optimization, new Mac instances, and the expanding multicloud connectivity options.

Read original

Apache Iceberg Amazon Bedrock Amazon Ec2 Claude+3

awsMay 12, 20262 min readCurated summary

Amazon Redshift introduces AWS Graviton-based RG instances with an integrated data lake query engine | Amazon Web Services

Amazon Redshift introduces RG instances powered by AWS Graviton, targeting lower-cost, higher-volume analytics for both human users and AI agents. RG instances can run warehouse workloads up to 2.2× faster than RA3 at 30% lower price per vCPU, while an integrated data lake engine enables faster SQL queries across warehouse tables and S3 data. The architecture also removes the need for Redshift Spectrum and its per-terabyte scanning fees. ## Performance and Cost Improvements - RG instances deliver: - Up to 2.2× faster data warehouse workloads than RA3. - 30% lower pricing per vCPU. - Up to 2.4× faster queries on Apache Iceberg data. - Up to 1.5× faster queries on Apache Parquet data. - The improvements are designed for: - Low-latency BI dashboards. - ETL pipelines and near-real-time analytics. - High-volume queries generated by autonomous AI agents. - AWS recommends using the AWS Pricing Calculator to estimate savings for specific workloads. ## Integrated Data Lake Query Engine - RG instances query warehouse tables and S3 data lakes through one engine. - Data lake queries run directly on Redshift cluster nodes rather than through Redshift Spectrum. - Existing external tables, schemas, Spectrum queries, and SQL syntax remain unchanged. - Customers do not need to recreate external tables or modify application code. - Queries remain inside the customer’s VPC, use existing IAM roles, and avoid Spectrum’s former $5-per-terabyte scanning charge. ## Migration and Setup - RG clusters can be created or migrated through: - The AWS Management Console. - AWS CLI. - AWS API. - The integrated data lake engine is enabled by default. - Migration options include: - **Elastic Resize:** In-place migration with approximately 10–15 minutes of downtime for compatible configurations. - **Snapshot and Restore:** Creates an RG cluster from an RA3 snapshot and is useful when configuration changes are needed. ## Availability and Pricing Options - RG instances are available across numerous AWS Regions in North America, Europe, Asia-Pacific, Canada, and South America. - Redshift Provisioned customers can choose: - On-Demand Instances with hourly billing and no commitment. - Reserved Instances for additional savings. RG instances are intended for organizations combining data warehouse and data lake workloads, especially those needing lower costs and fast response times at high query volumes. Customers should test compatibility and use workload-specific pricing estimates before migrating.

Read original

Apache Iceberg AI Agents Amazon S3 Sql+4

lineApr 3, 20264 min readCurated summary

From Hive to Iceberg: The Secret to 12x Faster Data Reflection

LINE Plus replaced a full-dump ETL pipeline for product data with incremental processing using Apache Iceberg and Apache Flink. The previous HBase/Hive workflow rewrote hundreds of millions of rows for every update, causing high compute costs and delays that left data up to an hour out of date. With the new architecture, update intervals were reduced from 60 minutes to 5 minutes—roughly a 12× improvement—while preserving consistency and fault tolerance. ## Limitations of Full-Data ETL - The existing HBase and Hive pipeline continuously collected CDC data in HDFS but had to merge it with existing data and rewrite the entire table before changes became queryable. - This caused: - High compute and storage costs - Dependence on limited shared Hadoop resources - Delayed updates and stale data - Snapshot-based extraction provides consistency, but large snapshots can take hours and retain old versions through MVCC, increasing system overhead. - Processing only the changed rows would reduce the workload from hundreds of millions of records to tens of thousands, separating update cost from total dataset size. ## Introducing Apache Iceberg - Iceberg manages data through metadata and table snapshots rather than relying solely on directory structures like traditional Hive tables. - It supports row-level `upsert` and `delete` operations. - This allows incremental changes to be written without rewriting the entire table, making much shorter ETL intervals possible. ## Requirements for the Streaming Pipeline The team evaluated Spark and Flink against three essential requirements: - **Data freshness:** Late-arriving compensation or replay data must not overwrite newer records. - **End-to-end exactly-once processing:** Iceberg updates and Kafka status messages must not partially succeed. - **Fault tolerance and state management:** Processing state must survive failures and restarts. A Kafka message indicating that all CDC data through a specific timestamp—such as 13:03—has been applied serves as the signal that a bulk extraction can safely begin. This requires complete confidence that the message accurately represents the Iceberg table’s committed state. ## Why Two-Phase Commit Was Necessary - Iceberg and Kafka are independent systems, so writing to one while failing to write to the other could create inconsistent state. - Two-phase commit (2PC) prevents partial success: - Both systems prepare their writes. - They commit only when all required operations succeed. - Any failure causes the operation to roll back. - Exactly-once processing also prevents duplicate or missing records during retries, network failures, or node restarts. - Together, these guarantees make Kafka status messages a reliable representation of the Iceberg table’s state. ## Choosing Flink over Spark - Spark Structured Streaming uses a micro-batch model, which makes fine-grained event-time and state control more difficult. - Flink provides native event-by-event streaming and better support for the required consistency model. - The team used Flink state to track each record’s `updatedate`: - Older late-arriving events are ignored. - Replayed historical data cannot overwrite newer values. - Flink checkpoints: - Persist streaming state externally. - Enable recovery from the latest consistent point. - Integrate with the Kafka sink’s 2PC mechanism. - Kafka messages remain in a pre-commit state until the Iceberg write and checkpoint both succeed. ## Kubernetes Deployment Options - The team compared: - **Native Kubernetes:** Requires manually configuring roles, service accounts, services, routing, deployments, slots, and jobs. - **Flink Kubernetes Operator:** Represents Flink infrastructure and jobs as custom resources, automating configuration such as routing and the web UI through Helm values. - Although Flink has greater operational complexity and a steeper learning curve than Spark, it was selected because it was the only option that satisfied all three core requirements at the engine level. The recommended architecture is an incremental Iceberg pipeline powered by Flink, with stateful processing, checkpoints, and two-phase commit between Iceberg and Kafka. This approach keeps data current, avoids expensive full-table rewrites, and provides reliable recovery and consistency at a five-minute update interval.

Read original

Apache Iceberg Database Design Kubernetes Apache Spark+4

pinterestFeb 24, 20263 min readCurated summary

Piqama: Pinterest Quota Management Ecosystem

Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.

Read original

Apache Iceberg Amazon S3 Rate Limiting Resource Scheduling+4

pinterestFeb 5, 20263 min readCurated summary

Next Generation DB Ingestion at Pinterest

Pinterest replaced fragmented, batch-oriented database ingestion with a unified Change Data Capture (CDC) framework. The new architecture uses Debezium/TiCDC, Kafka, Flink, Spark, and Iceberg to process only changed records, reducing latency from over 24 hours to minutes while lowering infrastructure costs. It also provides native row-level deletion, scalable operations, and improved compliance. ## Problems with the Legacy System - Batch workflows often delayed updates by more than 24 hours. - Full-table processing was inefficient because many tables changed by less than 5% each day. - Lack of row-level deletion support complicated data compliance. - Multiple independently maintained pipelines created operational complexity and inconsistent data quality. ## Unified CDC-Based Architecture - Supports MySQL, TiDB, and KVStore. - Captures database changes through a generic CDC service and publishes them to Kafka, typically in under one second. - Flink processes events in near real time and stores them in append-only CDC Iceberg tables on S3. - Spark jobs run periodically—often every 15 minutes—to merge recent changes into base Iceberg tables. - A bootstrap pipeline initializes base tables from historical database dumps. - Maintenance jobs handle compaction and snapshot expiration. - The framework is designed for at-least-once processing, petabyte-scale data, thousands of pipelines, and YAML-based configuration. ## CDC Tables and Base Tables - CDC tables act as time-series ledgers containing every change event. - CDC data typically becomes available within five minutes. - Base tables mirror the current state of the source database while retaining historical records. - Base-table latency is generally between 15 minutes and one hour. ## Upserting Changes into Base Tables - Spark first identifies the newest event for each primary key. - Events are ranked by timestamp and GTID, then deduplicated. - Iceberg’s `MERGE INTO` applies the resulting changes: - Deletes matching records when the event represents a deletion. - Updates existing records. - Inserts new records unless the event is a deletion. - The process uses a recent CDC window and a processing watermark to avoid reprocessing unnecessary data. ## Choosing Merge-on-Read - Pinterest standardized on Iceberg’s Merge-on-Read (MOR) strategy. - Copy-on-Write (COW) was rejected for most workloads because: - It requires more computation during writes. - It produces substantially larger replacement files, increasing storage costs. - MOR better balances update performance and storage efficiency for frequent incremental changes. ## Partitioning for Faster Upserts - Large base tables can be partitioned using a hash bucket of the primary key. - For example, `bucket(100, id)` distributes records across 100 partitions. - This allows Spark to process partitions in parallel and reduces the data scanned or rewritten during merges. - Iceberg tables are configured with format version 2, identifier fields, merge-on-read update and delete modes, and target file sizes. ## Small-File Challenge - Bucketing improved parallelism but caused each upsert to generate many small files within partitions. - The article indicates that Pinterest investigated this bottleneck and introduced further optimizations, though the supplied excerpt ends before describing them. Pinterest’s CDC-based design provides a substantially faster and more efficient alternative to full-table batch ingestion. Teams adopting a similar system should combine incremental CDC processing with partitioning, merge-on-read storage, bootstrapping, and ongoing file-maintenance strategies.

Read original

Apache Iceberg Apache Kafka Apache Spark Mysql+4

awsDec 2, 2025Original article

Announcing replication support and Intelligent-Tiering for Amazon S3 Tables (opens in new tab)

AWS has expanded the capabilities of Amazon S3 Tables by introducing Intelligent-Tiering for automated cost optimization and cross-region replication for enhanced data availability. These updates address the operational overhead of managing large-scale Apache Iceberg datasets by automating storage lifecycle management and simplifying the architecture required for global data distribution. By integrating these features, organizations can reduce storage costs without manual intervention while ensuring consistent data access across multiple AWS Regions and accounts. ### Cost Optimization with S3 Tables Intelligent-Tiering This feature automatically shifts data between storage tiers based on access frequency to maximize cost efficiency without impacting application performance. * The system utilizes three low-latency tiers: Frequent Access, Infrequent Access (offering 40% lower costs), and Archive Instant Access (offering 68% lower costs than Infrequent Access). * Data transitions are automated, moving to Infrequent Access after 30 days of inactivity and to Archive Instant Access after 90 days. * Automated table maintenance tasks, such as compaction and snapshot expiration, are optimized to skip colder files; for example, compaction only processes data in the Frequent Access tier to minimize unnecessary compute and storage costs. * Users can configure Intelligent-Tiering as the default storage class at the table bucket level using the AWS CLI commands `put-table-bucket-storage-class` and `get-table-bucket-storage-class`. ### Cross-Region and Cross-Account Replication New replication support allows users to maintain synchronized, read-only replicas of their S3 Tables across different geographic locations and ownership boundaries. * Replication maintains chronological consistency and preserves parent-child snapshot relationships, ensuring that replicas remain identical to the source for query purposes. * Replica tables are typically updated within minutes of changes to the source table and support independent encryption and retention policies to meet specific regional compliance requirements. * The service eliminates the need for complex, custom-built architectures to track metadata transformations or manually sync objects between Iceberg tables. * This functionality is primarily designed to reduce query latency for geographically distributed teams and provide robust data protection for disaster recovery scenarios. ### Practical Implementation To maximize the benefits of these new features, organizations should consider setting Intelligent-Tiering as the default storage class at the bucket level for all new datasets to ensure immediate cost savings. For global operations, setting up read-only replicas in regions closest to end-users will significantly improve query performance for analytics tools like Amazon Athena and Amazon SageMaker.

Apache Iceberg Amazon S3 Data Replication Amazon S3 Tables+3

awsDec 2, 2025Original article

Amazon CloudWatch introduces unified data management and analytics for operations, security, and compliance (opens in new tab)

Amazon CloudWatch has evolved into a unified platform for managing operational, security, and compliance log data, significantly reducing the need for redundant data stores and complex ETL pipelines. By standardizing ingestion through industry-standard formats like OCSF and OpenTelemetry, the service enables seamless cross-source analytics while lowering operational overhead and storage costs. This update allows organizations to move away from fragmented data silos toward a centralized, Iceberg-compatible architecture for deeper technical and business insights. **Data Ingestion and Schema Normalization** * Automatically collects AWS-vended logs across accounts and regions via AWS Organizations, including CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs. * Includes pre-built connectors for a wide range of third-party sources, such as endpoint security (CrowdStrike, SentinelOne), identity providers (Okta, Entra ID), and network security (Zscaler, Palo Alto Networks). * Utilizes managed Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry (OTel) conversion to ensure data consistency across disparate sources. * Provides built-in processors, such as Grok for custom parsing and field-level operations, to transform and manipulate strings during the ingestion phase. **Unified Architecture and Cost Optimization** * Consolidates log management into a single service with built-in governance, eliminating the need to store and maintain duplicate copies of data across different tools. * Introduces Apache Iceberg-compatible access via Amazon S3 Tables, allowing data to be queried in place by external tools. * Removes the requirement for complex ETL pipelines by providing a unified data store that is accessible to Amazon Athena, Amazon SageMaker Unified Studio, and other Iceberg-compatible analytics engines. **Advanced Analytics and Discovery Tools** * Supports multiple query interfaces, allowing users to interact with logs using natural language, SQL, LogsQL, or PPL (Piped Processing Language). * The new "Facets" interface enables intuitive filtering by application, account, region, and log type, featuring intelligent parameter inference for cross-account queries. * Enables the correlation of operational logs with business data from third-party tools like ServiceNow CMDB or GitHub to provide a more comprehensive view of organizational health. Organizations should leverage these unified management features to consolidate their security and operational monitoring into a single source of truth. By adopting OCSF normalization and the new S3 Tables integration, teams can reduce the technical debt associated with managing multiple log silos while improving their ability to run cross-functional analytics.

Apache Iceberg Log Management Amazon S3 Data Analytics+4

naverDec 1, 2025Original article

Naver TV (opens in new tab)

This technical session from NAVER ENGINEERING DAY 2025 explores the architectural journey of building a low-latency query system for real-time transaction reports. The project focuses on resolving the tension between high data freshness, massive scalability, and rapid response times for complex, multi-dimensional filtering. By leveraging Apache Iceberg in conjunction with StarRocks’ materialized views, the team established a performant data pipeline that meets the demands of modern business intelligence. ### Challenges in Real-Time Transaction Reporting * **Query Latency vs. Data Freshness:** Traditional architectures often struggle to provide immediate visibility into transaction data while maintaining sub-second query speeds across diverse filter conditions. * **High-Dimensional Filtering:** Users require the ability to query reports based on numerous variables, necessitating an engine that can handle complex aggregations without pre-defining every possible index. * **Scalability Requirements:** The system must handle increasing transaction volumes without degrading performance or requiring significant manual intervention in the underlying storage layer. ### Optimized Architecture with Iceberg and StarRocks * **Apache Iceberg Integration:** Iceberg serves as the open table format, providing a reliable foundation for managing large-scale data snapshots and ensuring consistency during concurrent reads and writes. * **StarRocks for Query Acceleration:** The team selected StarRocks as the primary OLAP engine to take advantage of its high-speed vectorized execution and native support for Iceberg tables. * **Spark-Based Processing:** Apache Spark is utilized for the initial data ingestion and transformation phases, preparing the transaction data for efficient storage and downstream consumption. ### Enhancing Performance via Materialized Views * **Pre-computed Aggregations:** By implementing Materialized Views, the system pre-calculates intensive transaction summaries, significantly reducing the computational load during active user queries. * **Automatic Query Rewrite:** The architecture utilizes StarRocks' ability to automatically route queries to the most efficient materialized view, ensuring that even ad-hoc reports benefit from pre-computed results. * **Balanced Refresh Strategies:** The research focused on optimizing the refresh intervals of these views to maintain high "freshness" while minimizing the overhead on the cluster resources. The adoption of a modern lakehouse architecture combining Apache Iceberg with a high-performance OLAP engine like StarRocks is a recommended strategy for organizations dealing with high-volume, real-time reporting. This approach effectively decouples storage and compute while providing the low-latency response times necessary for interactive data analysis.

Apache Iceberg Apache Spark Data Architecture Starrocks+3