Amazon S3 | Techlist.io

awsJul 20, 20263 min readCurated summary

AWS Weekly Roundup: One-click Lambda setup prompt, OpenAI GPT-5.6 models on Bedrock, and more (July 20, 2026) | Amazon Web Services

The AWS Weekly Roundup highlights new tools for AI-assisted serverless development, including a one-click Lambda setup prompt that configures coding agents with AWS Serverless skills and MCP. It also covers major service updates such as OpenAI models on Bedrock, faster S3 storage-class transitions, self-managed Lambda code storage, and Cognito password-hash imports. Additional stories include SQS’s 20th anniversary, open agent protocols, DynamoDB bulk operations, and a resolved Cost Explorer billing-data incident. ## One-Click Lambda Setup for Coding Agents - The Lambda console now provides a prompt that configures AI coding agents with: - AWS Serverless skills - The Serverless Model Context Protocol (MCP) server - Embedded serverless best practices - The setup guide supports Claude Code, Kiro, Cursor, GitHub Copilot, Codex, Devin Desktop, and OpenCode. - Developers can copy the setup URL into their preferred agent: ```text fetch https://docs.aws.amazon.com/lambda/latest/dg/samples/aws-lambda-agent-setup.md ``` - AWS’s Agent Toolkit can also install the AWS MCP Server, providing current AWS knowledge and controlled resource access. ## Major AWS Service Launches - **OpenAI GPT-5.6 models on Amazon Bedrock** - Sol: flagship reasoning - Terra: balanced performance - Luna: faster, lower-cost inference - All are available through Bedrock’s Responses API and its high-performance inference engine. - **Same-day S3 transitions** - Objects can transition to S3 Standard-IA or S3 One Zone-IA on the day they are created. - The previous 30-day minimum retention period in S3 Standard no longer applies. - These classes can reduce storage costs by up to 40% while retaining millisecond access. - Suitable for backups, log analytics, and compliance data that becomes cold quickly. - **Self-managed Lambda code storage** - Lambda can reference code directly from customer-owned S3 buckets. - Lambda no longer needs to create intermediate copies. - This removes code-storage limits and can shorten activation times after deployments. - **Cognito password-hash imports** - CSV user imports can now include password hashes. - Users can sign in immediately with existing credentials instead of resetting passwords. - Import configuration specifies the source system’s hashing algorithm. ## Additional AWS Updates - **Amazon SQS at 20** - SQS continues to provide scalable decoupling between message producers and consumers, two decades after its public launch. - **Open protocols with Strands Agents SDK** - An example demonstrates how MCP, A2A, UTCP, AG-UI, and x402 can work together when building AI agents. - **Open-source DynamoDB Bulk Executor** - Performs large-scale table operations without custom code. - Supports `count`, `find`, `delete`, and `update` commands. - **Kiro CLI for AWS Support** - MCP integration combines investigation, documentation lookup, and support-case creation. - Examples cover Glue failures, Lambda cold starts, and WAF false positives. ## Cost Explorer Incident - Some customers saw inaccurate estimated billing and usage data in Cost Explorer. - The issue generated erroneous budget and cost-anomaly alerts. - AWS resolved the incident and is conducting a retrospective to improve billing-incident prevention and response. AWS’s latest releases emphasize faster serverless development, more capable AI tooling, lower-cost storage, and easier automation of operational tasks. Developers should explore the Lambda agent setup and Agent Toolkit while reviewing the new storage, identity, and bulk-operation capabilities for relevant workloads.

Read original

Amazon S3 AI Agents Amazon Bedrock Aws Lambda+4

awsJul 13, 20263 min readCurated summary

Amazon SQS turns 20: Two decades of reliable messaging at scale | Amazon Web Services

Amazon SQS has spent two decades helping distributed systems communicate asynchronously without tightly coupling services. While its core purpose remains unchanged—decoupling producers and consumers, buffering traffic, and isolating failures—its scale, security, integrations, and workload support have expanded significantly. Recent improvements also make SQS suitable for high-throughput, multi-tenant, and AI-driven architectures. ## SQS’s Core Role in Distributed Systems - Producers place messages in queues and continue processing without waiting for consumers. - Consumers process messages when they are ready, preventing slow or unavailable services from causing cascading failures. - Customers use SQS to: - Decouple application components - Absorb traffic bursts - Improve resilience when individual services fail - Coordinate independent services and AI agents ## Higher Throughput for FIFO Queues - High-throughput FIFO mode launched in 2021 at up to 3,000 transactions per second per API action. - Capacity increased progressively to: - 6,000 TPS in 2022 - 9,000 TPS in 2023 - 18,000 TPS later in 2023 - Up to 70,000 TPS per API action in select Regions - The FIFO in-flight message limit grew from 20,000 to 120,000 in 2024, enabling more concurrent processing. ## Stronger Security and Access Controls - SSE-SQS launched in 2021, providing server-side encryption with AWS-managed keys and eliminating customer key-management requirements. - Encryption became the default for newly created queues in 2022. - Attribute-based access control was introduced in 2022, allowing permissions to be based on queue tags rather than static resource policies. ## Improved Message Recovery and Integration - Dead-letter queue redrive became available in the SQS console in 2021. - SDK and CLI APIs—including `StartMessageMoveTask`, `CancelMessageMoveTask`, and `ListMessageMoveTasks`—followed in 2023. - FIFO queue redrive support was added later that year. - JSON protocol support reduced processing latency by up to 23% for 5 KB payloads while lowering client CPU and memory use. - SQS queues can connect directly to EventBridge Pipes, enabling routing to many AWS services without custom integration code. ## Larger Messages and Fairer Queuing - The Extended Client Library for Python allows payloads up to 2 GB by storing message data in Amazon S3 and sending a reference through SQS. - In 2025, the native maximum message size increased from 256 KiB to 1 MiB for standard and FIFO queues. - Fair queues help prevent one tenant in a shared standard queue from delaying others. Producers provide a message group ID, while consumers require no changes. ## SQS for AI Workloads - SQS can buffer requests to large language models and regulate inference throughput. - Queues also help coordinate autonomous AI agents that operate as separate services. - These use cases apply the same established messaging model to more complex, distributed AI systems. Amazon SQS’s recommendation remains straightforward: use asynchronous queues when systems need loose coupling, burst management, and resilience. Its newer throughput, security, recovery, integration, and fairness features extend that pattern to larger and more demanding applications.

Read original

Amazon S3 AWS Distributed Systems Amazon Sqs+4

awsJun 22, 20263 min readCurated summary

Run isolated sandboxes with full lifecycle control: AWS Lambda introduces MicroVMs | Amazon Web Services

AWS Lambda MicroVMs provide isolated, stateful execution environments for running untrusted user- or AI-generated code without managing virtual machine infrastructure. Built on Firecracker, they combine VM-level isolation, near-instant startup and resume, and persistent memory and disk state. The post concludes that MicroVMs fill the gap between slow, isolated VMs, less-secure containers, and stateless event-driven Lambda functions. ## The Need for Isolated, Stateful Execution - AI coding assistants, online development environments, analytics tools, vulnerability scanners, and game servers increasingly need a dedicated environment for each user or session. - Traditional options involve tradeoffs: - VMs provide strong isolation but often take minutes to start. - Containers launch quickly but share a kernel and require extensive hardening for untrusted workloads. - Standard serverless functions are designed for short, request-response workloads rather than long-running interactive sessions. - Building custom virtualization infrastructure requires significant security, operations, and virtualization expertise. ## What Lambda MicroVMs Provide - Each user or session receives its own Firecracker-powered MicroVM. - MicroVMs offer: - Dedicated VM-level isolation with no shared kernel between users. - Rapid launch and resume from a pre-initialized snapshot. - Persistent memory, disk state, and running processes during a session. - Automatic suspension during inactivity to reduce idle costs. - Automatic resume when new traffic arrives. - Firecracker already powers AWS Lambda at large scale, providing an established virtualization foundation. ## Creating a MicroVM Image - The example packages a Flask application and Dockerfile into a ZIP archive and uploads it to Amazon S3. - The Dockerfile uses: ```dockerfile FROM public.ecr.aws/lambda/microvms:al2023-minimal ``` - It installs Python and dependencies, copies the Flask application, and starts it with Gunicorn on port 5000. - An image is created with the `aws lambda-microvms create-microvm-image` command, specifying: - The S3 code artifact - An image name - An AWS-provided base image ARN - An IAM build role - Lambda builds the image, initializes the application, and captures its memory and disk state in a Firecracker snapshot. - Build logs are available in CloudWatch under `/aws/lambda/microvms/<image-name>`. ## Launching and Managing a MicroVM - A MicroVM is launched from the image ARN with `run-microvm`. - The example configures an idle policy that: - Suspends the MicroVM after 15 minutes of inactivity. - Keeps it suspended for up to 5 minutes. - Automatically resumes it when traffic returns. - Lambda assigns a unique MicroVM ID and provides a dedicated HTTPS endpoint. - No separate networking setup is required. - The application is already running when the MicroVM becomes available because it resumes from the image snapshot. ## Request Handling and State Preservation - Clients authenticate requests using a short-lived token in the `X-aws-proxy-auth` header. - The Flask API responds immediately after launch. - When the MicroVM becomes idle, Lambda snapshots and stores its memory and disk state. - A later request resumes the environment with the application state intact, making suspension effectively invisible to the client. ## Underlying Execution Model - Lambda MicroVMs use an image-then-launch workflow: - Build and initialize an environment once. - Snapshot the initialized state. - Launch future MicroVMs by resuming that snapshot. - This avoids repeating operating-system and application startup work. - The combination of Firecracker isolation, snapshot-based startup, and suspend/resume lifecycle control makes MicroVMs suitable for secure, interactive, multi-tenant workloads. For applications that must safely execute untrusted code while preserving session state and responsive startup times, Lambda MicroVMs offer a managed alternative to building custom VM infrastructure.

Read original

Amazon S3 Python Docker & Containers Flask+3

netflixJun 19, 20263 min readCurated summary

The Evolution of Cassandra Data Movement at Netflix

Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency. ## Casspactor’s Role and Limitations - Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups. - It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions. - Its metadata view depended on several independent systems, each with different failure modes and update schedules. - Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed. - Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second. ## Constraints for Higher-Level Data Abstractions - Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations. - Large or skewed partitions caused executor memory failures and out-of-memory crashes. - Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing. - Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity. - Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes. - The monolithic connector could not serve as a reusable foundation for specialized connectors. ## Direct S3 Metadata as the Source of Truth - The new design reads backup metadata directly from the S3 storage layer. - This removes the chain of external metadata dependencies. - Backup existence and completeness are determined from the files that actually contain the data. - Direct backup access also enables restoration of historical backup states. ## A Layered Connector Architecture - The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format. - It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames. - A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector. - Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics. - Improvements to the shared reading engine automatically benefit every connector. ## Performance and Operational Improvements - Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions. - Reduced data shuffling helps prevent memory failures on large datasets. - Direct DataFrame output eliminates costly intermediary Iceberg tables. - Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning. - Fewer dependencies improve reliability and make the system easier to maintain. Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.

Read original

Amazon S3 Apache Spark Apache Iceberg Apache Cassandra+3

awsJun 17, 20263 min readCurated summary

Top announcements of the AWS Summit in New York, 2026 | Amazon Web Services

AWS Summit New York 2026 focused on making AI agents more capable, secure, autonomous, and easier to operate in production. Announcements spanned Amazon Bedrock AgentCore, security automation, developer tooling, workplace agents, and S3 data management. The overall direction is toward agents that can access governed knowledge, act independently, and continuously improve while remaining subject to enterprise controls. ## Building More Capable Agents - **Amazon Bedrock Managed Knowledge Base** simplifies enterprise RAG with native data connectors, Smart Parsing for multi-format data, and an Agentic Retriever for complex, multi-step queries. - **Web Search for Bedrock AgentCore** provides managed, current, cited web knowledge while keeping data within the customer’s secured AWS environment. - **Bedrock AgentCore harness** is now generally available, allowing developers to define an agent’s model, tools, skills, and instructions through configuration rather than custom orchestration loops. - **AWS Context**, coming soon, will map relationships across organizational data into a knowledge graph. Agents will be able to use governed relationships, business rules, and domain knowledge at runtime. - **AWS WAF Bot Control** lets publishers and content owners price, meter, and collect payment from AI bots accessing content and APIs, with scoped access enforced at the edge. ## Securing Agents and Applications - **AWS Continuum**, available in gated preview, aggregates vulnerability findings, ranks them by business impact, verifies exploitability, and routes fixes through existing development processes. - **AWS Security Agent**, now part of Continuum, adds: - Threat modeling based on the STRIDE framework - Pull-request code scanning and remediation across major Git platforms - IDE integrations through Kiro, Claude Code, and MCP - These tools are intended to let developers perform security reviews and address vulnerabilities without leaving their normal workflows. ## AI-Assisted Software Development - **Kiro for iOS**, in gated preview, lets developers start, monitor, steer, and approve Kiro sessions from a phone, including reviewing diffs and approving changes without keeping a laptop running. - **AWS DevOps Agent** adds release readiness reviews and autonomous release testing. It evaluates changes against natural-language standards and tests them in production-like environments. - **AWS Transform continuous modernization**, in preview, scans repositories against configurable technical-debt baselines and can autonomously generate remediation pull requests. ## Autonomous Workplace Agents - New **Amazon Quick autonomous agents** can work in the background with defined expertise, tone, permissions, and tools. - Example use cases include: - A finance agent processing incoming orders - A sales agent analyzing CRM, email, and Slack activity - Agents drafting follow-ups, identifying risks, and recommending next steps - A new **activity feed** combines email, messaging, calendars, and tasks into a prioritized view that adapts to the user’s communication and work patterns. ## Richer Metadata for Amazon S3 - **S3 annotations** allow up to 1 GB of mutable, queryable context to be attached directly to an object. - The feature targets AI agents and autonomous workflows that need to discover and interpret data without maintaining separate metadata systems. AWS’s announcements point toward an agent ecosystem that combines managed knowledge retrieval, web access, security automation, autonomous development workflows, and persistent workplace assistance. Organizations adopting these capabilities should pair autonomy with strong governance, scoped permissions, and continuous validation in production.

Read original

Amazon S3 AI Agents Amazon Bedrock RAG+4

awsJun 16, 20263 min readCurated summary

Amazon S3 annotations: attach rich, queryable context directly to your objects | Amazon Web Services

Amazon S3 annotations let organizations attach rich, mutable context directly to objects without rewriting the underlying data. Each object can have up to 1,000 named annotations, with a combined limit of 1 GB, in formats such as JSON, XML, YAML, or plain text. The annotations move with objects during copying and replication, can be queried through S3 Metadata tables, and are intended to support large-scale analytics and AI-driven workflows. ## Richer Metadata at Object Scale - Annotations address limitations of existing S3 metadata features: - System metadata describes properties such as size and storage class. - Object tags support operational tasks but are limited to 10 immutable tags. - User-defined metadata is limited to approximately 2 KB of headers and must be supplied during upload. - Annotations can be modified or deleted independently without rewriting the object. - Each annotation has a unique name and can hold up to 1 MB, with up to 1 GB total per object. - Supported content includes structured formats such as JSON, XML, and YAML, as well as plain text. ## Supporting AI and Industry Workflows - AI-generated transcripts, summaries, classifications, ratings, and technical specifications can remain attached to the relevant data. - Media companies can store transcripts, moderation results, subtitles, licensing information, and media specifications alongside video assets. - Financial institutions can attach investment summaries and sentiment analysis to research documents for autonomous discovery. - Life sciences organizations can add regulatory status, cohort information, and approval chains to clinical-trial data. - Annotations remain available for objects in archival storage classes without restoring the objects or paying retrieval charges. ## Object Lifecycle and Synchronization - Annotations automatically move with objects during copy, replication, and cross-Region transfers. - S3 removes annotations when the associated object is deleted. - This reduces dependence on separate metadata databases or sidecar files and avoids synchronization overhead. - Different teams or enrichment processes can maintain separate annotations on the same object without overwriting one another. ## Working with Annotations - IAM or bucket policies must grant: - `s3:PutObjectAnnotation` - `s3:GetObjectAnnotation` - The main APIs are: - `PutObjectAnnotation` to create or update an annotation - `GetObjectAnnotation` to retrieve one annotation - `ListObjectAnnotations` to view all annotations on an object - `DeleteObjectAnnotation` to remove an annotation - Reusing an annotation name with `PutObjectAnnotation` updates its content. - For multipart uploads, annotations are added after the upload is completed. - The example uses separate `mediainfo` JSON and `ai_summary` text annotations on a video object. ## Querying Through S3 Metadata - When S3 Metadata is enabled, annotations automatically populate managed annotation tables. - These tables can be queried with Amazon Athena and other analytics engines. - The S3 Tables MCP server provides a standardized interface for AI models and agents to query annotation data using natural-language workflows. - This enables discovery across large collections of S3 objects without retrieving the objects themselves. S3 annotations are most useful when object context is large, frequently updated, and needed for cross-object search or AI automation. Organizations should use them to keep business and machine-generated metadata directly associated with the data while using S3 Metadata tables for scalable querying.

Read original

Amazon S3 AI Agents Json Aws Cli+3

awsMay 14, 20262 min readCurated summary

Amazon Bedrock introduces new advanced prompt optimization and migration tool | Amazon Web Services

Amazon Bedrock Advanced Prompt Optimization automatically rewrites prompts to improve performance or support migration between models. It compares original and optimized prompts across up to five Bedrock models using example inputs, reference answers, and configurable evaluation methods. The tool supports text and multimodal inputs, then reports quality scores, estimated costs, and latency. ## Prompt Optimization and Model Migration - Users can optimize prompts for their current model or compare a baseline model with up to four alternatives. - Bedrock runs prompts against supplied examples, evaluates the responses, and iteratively rewrites the prompt. - Results include: - Original and optimized prompt templates - Evaluation scores - Cost estimates - Latency information - The process can identify regressions in established use cases and improve weaker tasks. ## Multimodal and Dataset Support - Prompt templates and evaluation examples are supplied in JSONL format. - Each JSON object must appear on a single line. - Supported multimodal inputs include: - PNG and JPG images - PDF documents - Files can be uploaded directly or imported from Amazon S3. - Optimization results and evaluation data can also be written to an S3 location. ## Evaluation Methods Users can select one evaluation approach per prompt template: - **Lambda-based evaluation** - Uses custom Python scoring logic for metrics such as accuracy, F1, execution accuracy, or structured JSON matching. - The function compares model outputs with reference responses. - **LLM-as-a-judge** - Uses a custom rubric with named metrics, structured instructions, and rating scales. - Bedrock’s judge model evaluates responses and provides scores with reasoning. - Claude Sonnet 4.6 is the default judge, though other supported judge models can be selected. - **Natural-language steering criteria** - Describes desired qualities such as brand voice, formatting, or safety constraints. - A default LLM judge evaluates responses holistically against those criteria. ## Getting Started and Availability - Start from **Create prompt optimization** in the Amazon Bedrock console or use the `CreateAdvancedPromptOptimizationJob` API. - Optimization is available in selected AWS Regions across North America, Europe, Asia-Pacific, Canada, and South America. - Pricing is based on Bedrock inference tokens consumed during optimization, using standard per-token inference rates. The tool is most useful when teams need measurable prompt improvements or want to migrate applications to another model. Preparing representative evaluation examples and choosing an appropriate scoring method will be essential for producing reliable optimization results.

Read original

Amazon S3 Large Language Models Amazon Bedrock Multimodal AI+3

awsMay 12, 20262 min readCurated summary

Amazon Redshift introduces AWS Graviton-based RG instances with an integrated data lake query engine | Amazon Web Services

Amazon Redshift introduces RG instances powered by AWS Graviton, targeting lower-cost, higher-volume analytics for both human users and AI agents. RG instances can run warehouse workloads up to 2.2× faster than RA3 at 30% lower price per vCPU, while an integrated data lake engine enables faster SQL queries across warehouse tables and S3 data. The architecture also removes the need for Redshift Spectrum and its per-terabyte scanning fees. ## Performance and Cost Improvements - RG instances deliver: - Up to 2.2× faster data warehouse workloads than RA3. - 30% lower pricing per vCPU. - Up to 2.4× faster queries on Apache Iceberg data. - Up to 1.5× faster queries on Apache Parquet data. - The improvements are designed for: - Low-latency BI dashboards. - ETL pipelines and near-real-time analytics. - High-volume queries generated by autonomous AI agents. - AWS recommends using the AWS Pricing Calculator to estimate savings for specific workloads. ## Integrated Data Lake Query Engine - RG instances query warehouse tables and S3 data lakes through one engine. - Data lake queries run directly on Redshift cluster nodes rather than through Redshift Spectrum. - Existing external tables, schemas, Spectrum queries, and SQL syntax remain unchanged. - Customers do not need to recreate external tables or modify application code. - Queries remain inside the customer’s VPC, use existing IAM roles, and avoid Spectrum’s former $5-per-terabyte scanning charge. ## Migration and Setup - RG clusters can be created or migrated through: - The AWS Management Console. - AWS CLI. - AWS API. - The integrated data lake engine is enabled by default. - Migration options include: - **Elastic Resize:** In-place migration with approximately 10–15 minutes of downtime for compatible configurations. - **Snapshot and Restore:** Creates an RG cluster from an RA3 snapshot and is useful when configuration changes are needed. ## Availability and Pricing Options - RG instances are available across numerous AWS Regions in North America, Europe, Asia-Pacific, Canada, and South America. - Redshift Provisioned customers can choose: - On-Demand Instances with hourly billing and no commitment. - Reserved Instances for additional savings. RG instances are intended for organizations combining data warehouse and data lake workloads, especially those needing lower costs and fast response times at high query volumes. Customers should test compatibility and use workload-specific pricing estimates before migrating.

Read original

Amazon S3 AI Agents Sql Apache Iceberg+4

awsApr 7, 20263 min readCurated summary

Launching S3 Files, making S3 buckets accessible as file systems | Amazon Web Services

Amazon S3 Files makes general-purpose S3 buckets accessible through a native NFS-based file system. It combines S3’s durability, cost, and broad service integration with interactive file operations, shared access, and low-latency performance. The post concludes that this reduces the need to choose between object storage and traditional file systems for many AWS workloads. ## Bridging Object Storage and File Systems - S3 Files presents S3 objects as files and directories. - Applications can use standard NFS v4.1+ operations, including creating, reading, updating, and deleting files. - Changes made through the file system are synchronized back to S3 as new objects or object versions. - Changes made directly in S3 generally appear in the file system within seconds, though synchronization can sometimes take longer. - Multiple compute resources can mount the same file system and share data without duplicating it. ## Performance and Data Access - S3 Files uses Amazon EFS underneath and provides approximately 1 ms latency for active data. - Frequently accessed metadata and file contents are placed on high-performance storage. - Large sequential reads can be served directly from S3 to maximize throughput. - Byte-range reads transfer only the requested portion of a file, reducing data movement and cost. - Intelligent prefetching anticipates access patterns. - Administrators can choose whether to cache complete files or metadata only. - NFS close-to-open consistency supports concurrent, interactive workloads such as ML pipelines and collaborative AI agents. ## Supported AWS Compute Services S3 Files can expose buckets to: - Amazon EC2 instances - Amazon ECS and EKS containers - AWS Fargate workloads - AWS Lambda functions This allows production applications, machine-learning systems, and agentic AI tools to access shared S3 data using ordinary file-system interfaces. ## Creating and Mounting an S3 File System The demonstration uses an EC2 instance and a general-purpose S3 bucket: - Create an S3 file system from the S3 console, AWS CLI, or infrastructure-as-code tools. - Configure or discover a mount target inside the relevant VPC. - Mount the file system on EC2 with commands such as: ```bash sudo mkdir /home/ec2-user/s3files sudo mount -t s3files fs-...:/ /home/ec2-user/s3files ``` - Files created in the mounted directory become visible in the S3 bucket after synchronization. - Standard commands such as `ls`, `echo`, and AWS CLI operations can verify that file contents are consistent between the mount and S3. ## Security, Permissions, and Monitoring - IAM identity and resource policies control access at both the file-system and object levels. - Data is encrypted in transit with TLS 1.3. - Data at rest uses SSE-S3 or customer-managed AWS KMS keys. - POSIX permissions rely on user IDs and group IDs stored as object metadata. - CloudWatch provides performance and update metrics. - CloudTrail records management events. - EC2 instances should use the latest `amazon-efs-utils` package, which is included in AWS-provided AMIs. S3 Files is best suited to workloads requiring shared, interactive file access while retaining data in S3. Teams should still evaluate access patterns and latency requirements, but the service offers a practical way to use familiar file operations without giving up S3’s centralized, durable storage model.

Read original

Amazon S3 Aws Cli Aws Ec2 Aws Eks+4

awsMar 16, 20263 min readCurated summary

AWS Weekly Roundup: Amazon S3 turns 20, Amazon Route 53 Global Resolver general availability, and more (March 16, 2026) | Amazon Web Services

Amazon S3 marked its 20th anniversary with major milestones in scale, performance, and cost reduction, while AWS introduced account regional namespaces for improved bucket-name control. The week’s featured launch was the general availability of Amazon Route 53 Global Resolver, providing secure, globally accessible DNS resolution across 30 AWS Regions. Other updates covered stateful AI agent infrastructure, Windows Server 2025, simplified AWS identity access, and reusable Redshift ingestion templates. ## Amazon S3 Reaches 20 Years - Launched publicly on March 14, 2006, S3 has grown from object storage into a foundational cloud service. - As of March 2026, it stores: - More than 500 trillion objects - Hundreds of exabytes of data - Over 200 million requests per second globally - Storage prices have fallen by approximately 85% since launch, to just over $0.02 per gigabyte. - New account regional namespaces let organizations reserve bucket names within their own account namespace by adding an account-specific suffix. - Adoption can be enforced with IAM and AWS Organizations service control policies using the `s3:x-amz-bucket-namespace` condition key. ## Route 53 Global Resolver Becomes Generally Available - Amazon Route 53 Global Resolver is an internet-reachable, anycast DNS resolver for authorized clients anywhere. - It is available across 30 AWS Regions and supports IPv4 and IPv6 DNS queries. - It resolves: - Public internet domains - Private domains associated with Route 53 private hosted zones - Security features include filtering for malicious, unsafe, DNS tunneling, and Domain Generation Algorithm (DGA) domains. - General availability adds protection against Dictionary DGA threats. - Centralized DNS query logging is also included. ## Additional AWS Service Updates - **Bedrock AgentCore Runtime** - Adds stateful MCP server support through the `Mcp-Session-Id` header. - Dedicated microVMs isolate each user session and preserve context across interactions. - MCP servers can use elicitation, sampling, and progress notifications in addition to resources, prompts, and tools. - **Amazon WorkSpaces** - Adds Windows Server 2025 bundles for WorkSpaces Personal and WorkSpaces Core. - Security features include TPM 2.0, UEFI Secure Boot, Credential Guard, HVCI, Secured-core server, and DNS-over-HTTPS. - Existing Windows Server 2016, 2019, and 2022 bundles remain supported. - **AWS Builder ID** - Adds GitHub and Amazon as sign-in options alongside Google and Apple. - Users can access AWS Builder Center, Training and Certification, and Kiro without maintaining separate credentials. - **Amazon Redshift** - Introduces reusable templates for `COPY` operations. - Templates centralize frequently used parameters, improve consistency, and automatically apply future updates to subsequent loads. - The feature is available in commercial and AWS GovCloud Regions. ## Upcoming AWS Events - AWS Summits are scheduled for Paris, London, and Bengaluru. - AWS Community Days are planned in Pune, San Francisco, and Romania. - AWS will participate in NVIDIA GTC 2026 in San Jose. - AWS Community GameDay Europe will offer hands-on troubleshooting challenges across more than 50 cities. For practitioners, the most significant developments are Route 53 Global Resolver for centralized global DNS security, S3 namespaces for organizational naming governance, and AgentCore’s stateful MCP support for more capable AI applications.

Read original

Amazon S3 Database Design Aws Iam Amazon Redshift+4

awsMar 13, 20263 min readCurated summary

Twenty years of Amazon S3 and building what’s next | Amazon Web Services

Amazon S3 began in 2006 as a simple web service for storing and retrieving objects, but its emphasis on security, durability, availability, performance, and elasticity enabled it to become foundational infrastructure. Over two decades, it scaled from roughly one petabyte to hundreds of exabytes while preserving API compatibility, reducing prices, and expanding beyond object storage. Amazon’s long-term vision is for S3 to serve as a universal foundation for data, analytics, and AI workloads. ## The Original S3 Philosophy - S3 introduced two basic operations: - `PUT` to store an object - `GET` to retrieve it - The service abstracted away complex infrastructure so developers could focus on applications. - Its five enduring design principles are: - **Security:** Data is protected by default. - **Durability:** Designed for 11 nines of durability, with a lossless operating model. - **Availability:** Failure is assumed and handled throughout the system. - **Performance:** Storage capacity can grow without degrading performance. - **Elasticity:** Capacity expands and contracts automatically. ## From One Petabyte to Hundreds of Exabytes - At launch, S3 had approximately: - One petabyte of capacity - 400 storage nodes across 15 racks and three data centers - 15 Gbps of bandwidth - A maximum object size of 5 GB - A price of $0.15 per GB - Today, S3: - Stores more than 500 trillion objects. - Serves over 200 million requests per second. - Operates across 123 Availability Zones in 39 AWS Regions. - Supports objects up to 50 TB—10,000 times larger than the original limit. - Storage prices have fallen by roughly 85%, to slightly above 2 cents per GB. - S3 Intelligent-Tiering has saved customers more than $6 billion in storage costs. - The S3 API has become an industry standard, with many other storage systems offering compatible interfaces. ## Backward Compatibility and Long-Term Reliability - Code written against S3 in 2006 still works without modification. - AWS has repeatedly replaced disks, storage systems, and request-processing code while preserving access to older data. - This compatibility reflects S3’s goal of remaining infrastructure that “just works” despite continuous internal change. ## Engineering for Durability and Scale - Microservices continuously inspect every byte across the fleet. - Auditor services detect degradation and automatically trigger repair and re-replication. - Automated formal methods mathematically verify correctness in areas such as: - The index subsystem - Cross-Region replication - Access policies - AWS has progressively rewritten performance-critical components in Rust over the past eight years. - Rust improves performance while preventing memory-safety bugs and other classes of errors at compile time. - S3 follows the principle that scale should improve the service: larger, more distributed workloads become increasingly decorrelated, improving reliability for all customers. ## S3 as a Foundation for Data and AI Amazon’s future vision is for customers to store data once in S3 and work with it directly, avoiding costly copies and specialized systems. - **S3 Tables** provides managed Apache Iceberg tables with automated maintenance to improve query performance and reduce storage costs. - **S3 Vectors** supports semantic search and retrieval-augmented generation, with up to 2 billion vectors per index and sub-100 ms query latency. - Within five months of launch, customers created over 250,000 indexes, ingested more than 40 billion vectors, and executed over 1 billion queries. - **S3 Metadata** enables centralized, faster data discovery without recursively listing large buckets. These additions extend S3 from inexpensive object storage into a broader platform for analytics, search, and AI while retaining its scale and cost advantages.

Read original

Amazon S3 Database Design Rust API+4

awsMar 12, 20262 min readCurated summary

Introducing account regional namespaces for Amazon S3 general purpose buckets | Amazon Web Services

Amazon S3 now lets customers create general purpose buckets in an account regional namespace, making bucket names predictable and reusable across AWS Regions. Names combine a customer-selected prefix with an account-, Region-, and namespace-specific suffix, preventing other accounts from claiming them. The feature preserves existing general purpose bucket capabilities while improving governance and automation. ## Account Regional Bucket Namespaces - Bucket names use a format such as `mybucket-123456789012-us-east-1-an`. - The suffix identifies the AWS account and Region, ensuring that other accounts cannot create buckets using it. - The combined prefix and suffix must be between 3 and 63 characters. - Buckets support the same features as general purpose buckets in the global namespace. ## Governance and Policy Controls - IAM policies and AWS Organizations service control policies can enforce namespace usage. - The new `s3:x-amz-bucket-namespace` condition key allows organizations to require account regional bucket creation. ## Creating Buckets - In the S3 console, select **Account regional namespace** when creating a bucket. - AWS CLI requests use the `--bucket-namespace account-regional` option. - SDKs can pass `BucketNamespace: "account-regional"` to the `CreateBucket` API. - Applications can use STS to retrieve the account ID and the SDK’s Region to construct compliant names. ## Infrastructure as Code - CloudFormation templates can use `AWS::AccountId` and `AWS::Region` to construct bucket names. - The `BucketNamespace: "account-regional"` property enables the feature. - `BucketNamePrefix` can be used when only the customer-defined prefix should appear in the template; AWS adds the account regional suffix automatically. ## Limitations and Availability - Existing global-namespace buckets cannot be renamed into the account regional namespace; new buckets must be created. - The feature applies only to S3 general purpose buckets. - S3 table and vector buckets use account-level namespaces, while directory buckets use zonal namespaces. - It is available in 37 AWS Regions, including AWS China and GovCloud Regions, with no additional cost. Organizations can adopt account regional namespaces to simplify bucket provisioning, prevent naming conflicts, and enforce consistent naming through IAM, Organizations policies, and infrastructure-as-code tools.

Read original

Amazon S3 Aws Cli Aws Cloudformation Aws Iam+3

pinterestFeb 24, 20263 min readCurated summary

Piqama: Pinterest Quota Management Ecosystem

Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.

Read original

Amazon S3 Apache Iceberg Rate Limiting Resource Scheduling+4

awsDec 2, 2025Original article

Announcing replication support and Intelligent-Tiering for Amazon S3 Tables (opens in new tab)

AWS has expanded the capabilities of Amazon S3 Tables by introducing Intelligent-Tiering for automated cost optimization and cross-region replication for enhanced data availability. These updates address the operational overhead of managing large-scale Apache Iceberg datasets by automating storage lifecycle management and simplifying the architecture required for global data distribution. By integrating these features, organizations can reduce storage costs without manual intervention while ensuring consistent data access across multiple AWS Regions and accounts. ### Cost Optimization with S3 Tables Intelligent-Tiering This feature automatically shifts data between storage tiers based on access frequency to maximize cost efficiency without impacting application performance. * The system utilizes three low-latency tiers: Frequent Access, Infrequent Access (offering 40% lower costs), and Archive Instant Access (offering 68% lower costs than Infrequent Access). * Data transitions are automated, moving to Infrequent Access after 30 days of inactivity and to Archive Instant Access after 90 days. * Automated table maintenance tasks, such as compaction and snapshot expiration, are optimized to skip colder files; for example, compaction only processes data in the Frequent Access tier to minimize unnecessary compute and storage costs. * Users can configure Intelligent-Tiering as the default storage class at the table bucket level using the AWS CLI commands `put-table-bucket-storage-class` and `get-table-bucket-storage-class`. ### Cross-Region and Cross-Account Replication New replication support allows users to maintain synchronized, read-only replicas of their S3 Tables across different geographic locations and ownership boundaries. * Replication maintains chronological consistency and preserves parent-child snapshot relationships, ensuring that replicas remain identical to the source for query purposes. * Replica tables are typically updated within minutes of changes to the source table and support independent encryption and retention policies to meet specific regional compliance requirements. * The service eliminates the need for complex, custom-built architectures to track metadata transformations or manually sync objects between Iceberg tables. * This functionality is primarily designed to reduce query latency for geographically distributed teams and provide robust data protection for disaster recovery scenarios. ### Practical Implementation To maximize the benefits of these new features, organizations should consider setting Intelligent-Tiering as the default storage class at the bucket level for all new datasets to ensure immediate cost savings. For global operations, setting up read-only replicas in regions closest to end-users will significantly improve query performance for analytics tools like Amazon Athena and Amazon SageMaker.

Amazon S3 Apache Iceberg Data Replication Amazon S3 Tables+3

awsDec 2, 2025Original article

Amazon S3 Storage Lens adds performance metrics, support for billions of prefixes, and export to S3 Tables (opens in new tab)

Amazon S3 Storage Lens has introduced three significant updates designed to provide deeper visibility into storage performance and usage patterns at scale. By adding dedicated performance metrics, support for billions of prefixes, and direct export capabilities to Amazon S3 Tables, AWS enables organizations to better optimize application latency and storage costs. These enhancements allow for more granular data-driven decisions across entire AWS organizations or specific high-performance workloads. ## Enhanced Performance Metric Categories The update introduces eight new performance-related metric categories available through the S3 Storage Lens advanced tier. These metrics are designed to pinpoint specific architectural bottlenecks that could impact application speed. * **Request and Storage Distributions:** New metrics track the distribution of read/write request sizes and object sizes, helping identify small-object patterns that might be better suited for Amazon S3 Express One Zone. * **Error and Latency Tracking:** Users can now monitor concurrent PUT 503 errors to identify throttling and analyze FirstByteLatency and TotalRequestLatency to measure end-to-end request performance. * **Data Transfer Efficiency:** Metrics for cross-Region data transfer help identify high-cost or high-latency data access patterns, suggesting where compute resources should be co-located with storage. * **Access Patterns:** Tracking unique objects accessed per day identifies "hot" datasets that could benefit from higher-performance storage tiers or caching solutions. ## Support for Billions of Prefixes S3 Storage Lens has expanded its analytical scale to support the monitoring of billions of prefixes. This allows organizations with massive, complex data structures to maintain granular visibility without sacrificing performance or detail. * **Granular Visibility:** Users can drill down into massive datasets to find specific prefixes causing performance degradation or cost spikes. * **Scalable Analysis:** This expansion ensures that even the largest data lakes can be monitored at a level of detail previously limited to smaller buckets. ## Integration with Amazon S3 Tables The service now supports direct export of storage metrics to Amazon S3 Tables, a feature optimized for high-performance analytics. This integration streamlines the workflow for administrators who need to perform complex queries on their storage metadata. * **Analytical Readiness:** Exporting to S3 Tables makes it easier to use SQL-based tools to query storage trends and performance over time. * **Automation:** This capability allows for the creation of automated reporting pipelines that can handle the massive volume of data generated by prefix-level monitoring. To take full advantage of these features, users should enable the S3 Storage Lens advanced tier and configure prefix-level monitoring for buckets containing mission-critical or high-throughput data. Organizations experiencing latency issues should specifically review the new request size distribution metrics to determine if batching objects or migrating to S3 Express One Zone would improve performance.

Amazon S3 Cloud Monitoring Amazon S3 Tables Storage Optimization+2