aws·Jun 16, 2026·3 min readCurated summary

Amazon S3 annotations: attach rich, queryable context directly to your objects | Amazon Web Services

Amazon S3 annotations let organizations attach rich, mutable context directly to objects without rewriting the underlying data. Each object can have up to 1,000 named annotations, with a combined limit of 1 GB, in formats such as JSON, XML, YAML, or plain text. The annotations move with objects during copying and replication, can be queried through S3 Metadata tables, and are intended to support large-scale analytics and AI-driven workflows. ## Richer Metadata at Object Scale - Annotations address limitations of existing S3 metadata features: - System metadata describes properties such as size and storage class. - Object tags support operational tasks but are limited to 10 immutable tags. - User-defined metadata is limited to approximately 2 KB of headers and must be supplied during upload. - Annotations can be modified or deleted independently without rewriting the object. - Each annotation has a unique name and can hold up to 1 MB, with up to 1 GB total per object. - Supported content includes structured formats such as JSON, XML, and YAML, as well as plain text. ## Supporting AI and Industry Workflows - AI-generated transcripts, summaries, classifications, ratings, and technical specifications can remain attached to the relevant data. - Media companies can store transcripts, moderation results, subtitles, licensing information, and media specifications alongside video assets. - Financial institutions can attach investment summaries and sentiment analysis to research documents for autonomous discovery. - Life sciences organizations can add regulatory status, cohort information, and approval chains to clinical-trial data. - Annotations remain available for objects in archival storage classes without restoring the objects or paying retrieval charges. ## Object Lifecycle and Synchronization - Annotations automatically move with objects during copy, replication, and cross-Region transfers. - S3 removes annotations when the associated object is deleted. - This reduces dependence on separate metadata databases or sidecar files and avoids synchronization overhead. - Different teams or enrichment processes can maintain separate annotations on the same object without overwriting one another. ## Working with Annotations - IAM or bucket policies must grant: - `s3:PutObjectAnnotation` - `s3:GetObjectAnnotation` - The main APIs are: - `PutObjectAnnotation` to create or update an annotation - `GetObjectAnnotation` to retrieve one annotation - `ListObjectAnnotations` to view all annotations on an object - `DeleteObjectAnnotation` to remove an annotation - Reusing an annotation name with `PutObjectAnnotation` updates its content. - For multipart uploads, annotations are added after the upload is completed. - The example uses separate `mediainfo` JSON and `ai_summary` text annotations on a video object. ## Querying Through S3 Metadata - When S3 Metadata is enabled, annotations automatically populate managed annotation tables. - These tables can be queried with Amazon Athena and other analytics engines. - The S3 Tables MCP server provides a standardized interface for AI models and agents to query annotation data using natural-language workflows. - This enables discovery across large collections of S3 objects without retrieving the objects themselves. S3 annotations are most useful when object context is large, frequently updated, and needed for cross-object search or AI automation. Organizations should use them to keep business and machine-generated metadata directly associated with the data while using S3 Metadata tables for scalable querying.

Read original(opens in new tab)

Amazon AthenaAI AgentsAmazon S3Json+3

aws·Dec 2, 2025Original article

Amazon CloudWatch introduces unified data management and analytics for operations, security, and compliance (opens in new tab)

Amazon CloudWatch has evolved into a unified platform for managing operational, security, and compliance log data, significantly reducing the need for redundant data stores and complex ETL pipelines. By standardizing ingestion through industry-standard formats like OCSF and OpenTelemetry, the service enables seamless cross-source analytics while lowering operational overhead and storage costs. This update allows organizations to move away from fragmented data silos toward a centralized, Iceberg-compatible architecture for deeper technical and business insights. **Data Ingestion and Schema Normalization** * Automatically collects AWS-vended logs across accounts and regions via AWS Organizations, including CloudTrail, VPC Flow Logs, WAF access logs, and Route 53 resolver logs. * Includes pre-built connectors for a wide range of third-party sources, such as endpoint security (CrowdStrike, SentinelOne), identity providers (Okta, Entra ID), and network security (Zscaler, Palo Alto Networks). * Utilizes managed Open Cybersecurity Schema Framework (OCSF) and OpenTelemetry (OTel) conversion to ensure data consistency across disparate sources. * Provides built-in processors, such as Grok for custom parsing and field-level operations, to transform and manipulate strings during the ingestion phase. **Unified Architecture and Cost Optimization** * Consolidates log management into a single service with built-in governance, eliminating the need to store and maintain duplicate copies of data across different tools. * Introduces Apache Iceberg-compatible access via Amazon S3 Tables, allowing data to be queried in place by external tools. * Removes the requirement for complex ETL pipelines by providing a unified data store that is accessible to Amazon Athena, Amazon SageMaker Unified Studio, and other Iceberg-compatible analytics engines. **Advanced Analytics and Discovery Tools** * Supports multiple query interfaces, allowing users to interact with logs using natural language, SQL, LogsQL, or PPL (Piped Processing Language). * The new "Facets" interface enables intuitive filtering by application, account, region, and log type, featuring intelligent parameter inference for cross-account queries. * Enables the correlation of operational logs with business data from third-party tools like ServiceNow CMDB or GitHub to provide a more comprehensive view of organizational health. Organizations should leverage these unified management features to consolidate their security and operational monitoring into a single source of truth. By adopting OCSF normalization and the new S3 Tables integration, teams can reduce the technical debt associated with managing multiple log silos while improving their ability to run cross-functional analytics.

Amazon AthenaLog ManagementAmazon S3Apache Iceberg+4

coupang·Nov 15, 2024Original article

Optimizing Operating Costs through Cloud Service (opens in new tab)

Coupang’s Finance and Engineering teams collaborated to optimize cloud expenditures by focusing on resource efficiency and the company's "Hate Waste" leadership principle. Through a dedicated optimization project team and the implementation of data-driven analytics, the company successfully reduced on-demand costs by millions of dollars without compromising business growth. This initiative transformed cloud management from a reactive expense into a proactive engineering culture centered on financial accountability and technical efficiency. ### Forming the Optimization Project Team * A specialized team consisting of Cloud Infrastructure Engineers and Technical Program Managers (TPMs) was established to bridge the gap between finance and engineering. * The project team focused on educating domain teams about the variable cost model of cloud services, moving away from a fixed-cost mindset. * Technical experts helped domain teams identify opportunities to use cost-efficient technologies, such as ARM-based AWS Graviton processors and AWS Spot Instances for data processing. * The initiative established clear ownership, ensuring that each domain team understood and managed their specific cloud resource usage. ### Analytics and Dashboards for Visibility * Engineers developed custom dashboards using Amazon Athena to process Amazon CloudWatch data, providing deep insights into resource performance. * The team utilized AWS Cost & Usage Reports (CUR) within internal Business Intelligence (BI) tools to provide granular visibility into spending patterns. * Finance teams worked alongside engineers to align technical roadmaps with monthly and quarterly budget goals, making cost management a shared responsibility. ### Strategies for Usage and Cost Reduction * **Spend Less (Usage Reduction):** Coupang implemented automation to ensure that non-production environment resources were only active when needed, resulting in a 25% cost saving for those environments. * **Pay Less (Right-sizing):** The team analyzed usage patterns to manually identify and decommission unused EC2 resources across all domain teams. * **Instance and Storage Optimization:** The project prioritized migrating workloads to the latest instance generations and optimizing Amazon S3 storage structures to reduce costs for data at rest. To achieve sustainable cloud efficiency, organizations should move beyond simple monitoring and foster an engineering culture where resource management is a core technical discipline. Prioritizing automated resource scheduling and adopting modern, high-efficiency hardware like Graviton instances are essential steps for any large-scale cloud operation looking to maximize its return on investment.

Amazon AthenaAWSAmazon Ec2Amazon S3+3

coupang·Nov 14, 2024Original article

Cloud expenditure optimization for cost efficiency (opens in new tab)

Coupang addressed rising cloud costs by establishing a cross-functional Central team to bridge the gap between engineering usage and financial accountability. Through a data-driven approach involving custom analytics and automated resource management, the company successfully reduced on-demand expenditure by millions of dollars. This initiative demonstrates that aligning technical infrastructure with financial governance is essential for maintaining growth without unnecessary waste. **The Central Team and Data-Driven Governance** * Coupang formed a specialized Central team consisting of infrastructure engineers and technical program managers to identify efficiency opportunities across the organization. * The team developed custom BI dashboards utilizing Amazon CloudWatch, AWS Cost and Usage Reports (CUR), and Amazon Athena to provide domain teams with actionable insights into their spending. * The finance department partnered with engineering to enforce strict budget compliance, ensuring that domain teams managed their resources within assigned monthly and quarterly limits. **Strategies for Spending and Paying Less** * The company implemented "Spending Less" strategies by automating the launch of resources in non-production environments only when needed, resulting in a 25% cost reduction for those areas. * "Paying Less" initiatives focused on rightsizing, where the Central team worked with domain owners to manually identify and eliminate unutilized or underutilized EC2 resources. * Workloads were migrated to more efficient hardware and pricing models, specifically leveraging ARM-based AWS Graviton processors and AWS Spot Instances for data processing and storage. **Targeted Infrastructure Optimization** * Engineering teams focused on instance generation alignment, ensuring that services were running on the most cost-effective hardware generations available. * Storage costs were reduced by optimizing Amazon S3 structures at rest, improving how data is organized and stored. * The team refined Amazon EMR (Elastic MapReduce) configurations to enhance processing efficiency, significantly lowering the cost of large-scale data analysis. To achieve sustainable cloud efficiency, engineering organizations should move beyond viewing cloud costs as a purely financial concern and instead treat resource management as a core technical metric. By integrating financial accountability directly into the engineering workflow through shared analytics and automated resource controls, companies can foster a culture of efficiency that supports long-term scalability.

Amazon AthenaAWSAmazon S3Finops+4