Aws Lambda

21 posts

awsOriginal article

Build multi-step applications and AI workflows with AWS Lambda durable functions (opens in new tab)

AWS Lambda durable functions introduce a simplified way to manage complex, long-running workflows directly within the standard Lambda experience. By utilizing a checkpoint and replay mechanism, developers can now write sequential code for multi-step processes that automatically handle state management and retries without the need for external orchestration services. This feature significantly reduces the cost of long-running tasks by allowing functions to suspend execution for up to one year without incurring compute charges during idle periods. ### Durable Execution Mechanism * The system uses a "durable execution" model based on checkpointing and replay to maintain state across function restarts. * When a function is interrupted or resumes from a pause, Lambda re-executes the handler from the beginning but skips already-completed operations by referencing saved checkpoints. * This architecture ensures that business logic remains resilient to failures and can survive execution environment recycles. * The execution state can be maintained for extended periods, supporting workflows that require human intervention or long-duration external processes. ### Programming Primitives and SDK * The feature requires the inclusion of a new open-source durable execution SDK in the function code. * **Steps:** The `context.step()` method defines specific blocks of logic that the system checkpoints and automatically retries upon failure. * **Wait:** The `context.wait()` primitive allows the function to terminate and release compute resources while waiting for a specified duration, resuming only when the time elapses. * **Callbacks:** Developers can use `create_callback()` to pause execution until an external event, such as an API response or a manual approval, is received. * **Advanced Control:** The SDK includes `wait_for_condition()` for polling external statuses and `parallel()` or `map()` operations for managing concurrent execution paths. ### Configuration and Setup * Durable execution must be enabled at the time of the Lambda function's creation; it cannot be retroactively enabled for existing functions. * Once enabled, the function maintains the same event handler structure and service integrations as a standard Lambda function. * The environment is specifically optimized for high-reliability use cases like payment processing, AI agent orchestration, and complex order management. AWS Lambda durable functions represent a major shift for developers who need the power of stateful orchestration but prefer to keep their logic within a single code-based environment. It is highly recommended for building AI workflows and multi-step business processes where state persistence and cost-efficiency are critical requirements.

datadog3 min readCurated summary

Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust

Datadog rewrote its AWS Lambda extension from Go into Rust to overcome the performance limits of adapting its large, host-oriented Datadog Agent to Lambda’s constrained environment. The redesign reduced cold-start latency by 82%, memory usage by 40%, and binary size from 55 MB to 7 MB. The project succeeded by narrowing the problem, enforcing performance budgets from the beginning, and designing specifically for Lambda’s execution model. ## Why the Original Extension Needed to Change - The Lambda extension runs as a sidecar process, collecting logs, metrics, traces, profiles, and process data asynchronously. - It was originally based on the Datadog Agent, which is designed for hosts, containers, and clusters. - The Agent’s fairness, buffering, caching, and high-throughput features introduced unnecessary overhead in Lambda. - Optimization attempts included: - Removing dependencies with build tags - Compressing binaries with UPX - Eliminating unnecessary `init` methods - Exploring Go plugins for lazy loading - These changes could not reduce additional cold-start latency below roughly 450–500 milliseconds. ## Why a Rewrite—and Why Rust - Rewrites are risky because they can lose undocumented invariants, reproduce subtle bugs, and create the burden of supporting two systems. - The team concluded that Lambda represented a fundamentally different scale and workload from the general-purpose Datadog Agent. - Rust was well suited because: - Memory safety reduces the risk of crashes and data races. - Extension crashes also terminate the Lambda function and trigger another cold start. - Rust produces small binaries with limited runtime overhead. - Lambda targets a narrow platform set: Amazon Linux on x86 and Arm. - Compile-time concurrency guarantees support reliable multithreaded code. - A hackathon prototype demonstrated enough potential to begin the full rewrite, named Project Bottlecap. ## Project Bottlecap’s Design Constraints - The extension had to minimize interference with the function handler, especially because many Lambda functions serve latency-sensitive APIs. - Telemetry work should occur after the handler returns whenever possible. - The team also minimized post-runtime duration—the CPU time added after normal function execution. - Performance was monitored from the start: - Dashboards and alerts tracked cold-start overhead. - Every pull request was benchmarked. - Regressions were investigated before merging. - The team accepted targeted tradeoffs for speed, including manually implementing AWS API calls and request signing instead of using SDKs that added too much overhead. - The design emphasized optionality because Lambda workloads range from small API functions to large asynchronous batch jobs. - Planned flush strategies included: - Flushing at the end of an invocation for infrequently called or CPU-constrained functions - Periodic or in-invocation flushing for workloads needing different latency and resource tradeoffs The practical lesson is that software optimized for large, long-running systems may be fundamentally unsuitable for serverless runtimes. When optimization reaches a hard performance floor, a focused rewrite—constrained by the target environment and measured continuously—can deliver major gains.

Read original(opens in new tab)
datadog1 min readCurated summary

Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust | Datadog

The provided text does not contain the blog post’s main article body. It mainly includes Datadog’s navigation links and a promotional announcement that Gartner named Datadog a Leader in the 2026 Magic Quadrant for Observability Platforms. The linked page URL suggests the intended post concerns Datadog’s AWS Lambda extension and Rust, but no technical details are included. ## Datadog’s Observability Platform Recognition - Datadog promotes its recognition as a Gartner Magic Quadrant Leader. - The surrounding navigation highlights products for: - Infrastructure and cloud monitoring - Application performance monitoring - Logs and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI and platform capabilities ## Referenced Lambda and Rust Article - The navigation links to an engineering post titled around the Datadog Lambda Extension and Rust. - However, the supplied content contains no explanation of: - Why Rust was selected - The extension’s architecture - Performance or resource improvements - Deployment and compatibility considerations The article body is needed for a substantive technical summary.

Read original(opens in new tab)
datadog3 min readCurated summary

How Datadog's IT team automated account inactivity and SaaS spend management

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

Read original(opens in new tab)
figma2 min readCurated summary

Inside Figma: securing internal web apps | Figma Blog

Figma built a reusable system for securely exposing internal web applications while preserving a smooth employee experience. The design combines AWS Application Load Balancers, Cognito, Okta, SAML, Lambda, and Terraform to enforce zero-trust authentication and centralized authorization. Its main conclusion is that carefully integrated, managed cloud components can provide strong security without creating excessive operational work for a small security team. ## Security Requirements for Internal Applications Figma’s internal web tools support critical workflows such as software deployment and customer support, making them attractive targets for attackers. The system was designed around five requirements: - **Smooth user experience:** Authentication should be fast, reliable, and convenient. - **Zero-trust access:** Network location alone should not establish trust. - **Modern authentication:** Applications should be able to use protections such as WebAuthn. - **Centralized authorization:** IT and security teams should centrally assign, monitor, and revoke permissions. - **Low operational overhead:** The system should minimize ongoing SRE and security-team toil. ## Technologies Used The architecture relies primarily on managed services and infrastructure-as-code: - **SAML:** Exchanges identity, group membership, and role assertions between services. - **AWS Application Load Balancer:** Acts as a managed reverse proxy and routes authenticated HTTP/HTTPS traffic. - **AWS Cognito:** Provides user-pool functionality and integrates with federated SAML identity providers. - **AWS Lambda:** Runs code in response to configured events without managing servers. - **Terraform:** Defines and automates AWS and Okta configuration through reusable modules. ## Application Load Balancers, Cognito, and Okta Figma uses AWS for infrastructure and Okta for employee authentication and authorization. - ALBs can authenticate traffic using OIDC, but Okta charges extra for OIDC support. - Figma instead uses the ALB’s SAML authentication capability together with an AWS Cognito user pool. - Terraform modules automate the creation and configuration of the required ALB and Cognito resources. - These modules allow infrastructure engineers to quickly deploy internal applications protected by the company’s Okta environment. - The Cognito user pool disables self-registration. - Each pool connects to a Cognito identity provider backed by a SAML Okta application created for the specific internal application. - Attribute mappings are configured so user information such as `email` and `profile` is passed through the authentication flow.

Read original(opens in new tab)
datadog2 min readCurated summary

How Datadog's IT team automated monitoring third-party accounts

Datadog built “Clarity,” an automated system for auditing SaaS accounts against Workday’s employee records. It regularly identifies accounts that do not belong to active employees, then logs, tickets, stores, and communicates findings through Datadog, Freshservice, Slack, and DynamoDB. The system replaces infrequent, manual reviews with continuous visibility and faster remediation of security and cost risks. ## The Need for Automated SaaS Audits - Modern companies rely on dozens or hundreds of external applications. - Manual account reviews are difficult to scale and may fail to detect unauthorized or abandoned accounts promptly. - An unexpected account in an identity provider or SaaS application could give a bad actor access to sensitive systems. - Datadog needed recurring audits as its SaaS portfolio continued to expand. ## Clarity’s Requirements - Use a single source of truth for employee status: - Datadog uses Workday. - Other organizations could use Okta, OneLogin, ADP, or Active Directory. - Run frequently enough to provide timely visibility. - Support manual execution when needed. - Integrate with existing communication, ticketing, and observability tools, such as Slack, Freshservice, and Datadog. - Minimize disruption to IT workflows and encourage adoption across a globally distributed organization. ## Audit Pipeline - A CloudWatch Event Rule triggers the audit Monday through Friday at 10 a.m. EST. - Clarity concurrently retrieves: - Active employees from Workday. - Active users from primary SaaS applications such as Slack, GitHub, and Zoom. - It compares SaaS user email addresses with active employee records. - Accounts without a matching active employee are flagged. - Results are: - Sent to Datadog as logs and metrics. - Added to DynamoDB for historical tracking. - Converted into Freshservice tickets. - Reported through Slack notifications with an audit summary. ## Datadog Metrics and Investigation - Clarity sends a metric for every flagged account using the Datadog Metrics API and Python SDK. - It uses a gauge metric to track flagged accounts over time. - Metrics include tags such as: - Environment, such as production. - Responsible team. - SaaS service. - Flagged user’s email address. - These tags provide the context needed to investigate the account and support alerting and visualization within Datadog. ## Practical Outcome Clarity provides a repeatable, automated control for SaaS account governance. Organizations implementing a similar system should connect an authoritative employee directory to their SaaS inventory, run audits regularly, and integrate findings with their existing monitoring, ticketing, and notification workflows.

Read original(opens in new tab)