Agentic Workflows

14 posts

gitlab4 min readCurated summary

How I built a demo generator with GitLab Duo Agent Platform

Click-through demos communicate product value more effectively than documentation, videos, or live environments, but traditionally required days of manual work. The author built a GitLab Duo Agent Platform workflow that turns screenshots and optional context into a narrated, branded HTML demo, opens a merge request, and publishes the result to GitLab Pages. The approach makes demos repeatable, reviewable, versioned, and easier to update when the product changes. ## Why Click-Through Demos Work - Users understand features faster by seeing them in action. - Click-through demos let viewers control the pace without requiring: - Access to a live product environment - Expertise navigating the product - External video-editing tools - Demos are reusable links that can be shared consistently through channels such as the GitLab Demo Hub. ## From Virtual Machines to Agents The author describes three stages in demo creation: - **Virtual machines:** Product and demo content were packaged into VMs, later hosted on centralized ESX servers. - **External SaaS tools:** Demo platforms added analytics and convenience but remained manual and dependent on vendor infrastructure. - **Agentic workflows:** Agents generate demos from prompts and deploy them within the same project, removing vendor dependency. Each stage improved accessibility, but agents make demos more sustainable by reducing repetitive work and simplifying updates. ## The Demo Creation Workflow A complete demo typically requires: - Learning the feature through documentation, issues, Slack, and discussions with product and engineering teams. - Developing a customer-focused story and scenario. - Creating a project and realistic demo data. - Capturing screenshots or video clips. - Writing step-by-step narration. - Assembling the demo in a clickable format. - Gathering stakeholder feedback and incorporating revisions. - Publishing to GitLab Docs, the Demo Hub, or other destinations. - Announcing the demo internally and externally. These activities can take days, and product changes may force the process to restart. The agent automates much of the predictable work while leaving feature selection, quality review, and final approval to humans. ## How the Click-Through Demo Generator Works - Screenshots are placed in a project folder. - An optional `context.md` file provides feature background. - An issue comment triggers the workflow. - The agent: - Renames and reorders screenshots - Writes the narration - Applies a branded HTML template - Creates a merge request - After review and merging, the demo is automatically published to GitLab Pages. - Follow-up changes can be requested from the agent. A Custom Flow invokes a custom tool that assembles the HTML, places narration panels over screenshots, and posts the merge request link back to the issue. ## Outcome-Focused Narration A project skill guides the agent’s writing so the demo explains customer value rather than merely walking through the interface. The required structure has three acts: - Open with the customer’s problem. - Show the product solving that problem step by step. - Close by explaining the value the customer gains. Localization can be added by creating a language-specific skill, allowing future demos to be translated automatically. ## Automated Publishing Publishing is also handled through an agent skill: - A demo is tagged as ready for production in the development gallery. - The agent enters the Demo Hub project. - It writes a short description, adds the current date, applies a “new” tag, and opens a merge request for the demo card. - A human still reviews and merges the request before the demo goes live. ## Demos as Code Because a click-through demo consists primarily of HTML, screenshots, and configuration, it can live in a GitLab repository rather than an isolated external tool. This enables teams to: - Review demos through merge requests. - Improve them collaboratively. - Track changes and versions over time. - Keep demos alongside the workflows that generate and publish them. The practical recommendation is to identify repetitive, well-defined work in your own organization and use GitLab Duo Agent Platform to automate its groundwork, while retaining human control over strategy, review, and release.

Read original(opens in new tab)
github3 min readCurated summary

Copilot vs. raw API access: What are you actually paying for?

GitHub Copilot and direct model APIs serve different purposes rather than competing at the same layer. Copilot bundles model access with development workflows—repositories, editors, terminals, issues, pull requests, and organizational controls—while APIs give teams the primitives to build their own systems. The best choice depends on whether you want to own the surrounding infrastructure or use GitHub’s integrated tooling. ## Copilot as Development Tooling - Copilot supports workflows from GitHub Issues through code changes, testing, pull requests, and review. - Its value includes integration with: - Editors and repositories - Terminals and permitted commands - Repository instructions - Pull requests and organizational policies - Paid plans include code completions and Next Edit Suggestions, while more intensive chat and agentic tasks consume AI Credits. - Actual cost depends on context selection, input/output/cached tokens, tool calls, retries, and task complexity. - Organization plans pool credits and provide budgets and usage tracking through the billing dashboard. ## Raw APIs for Systems You Control - Direct API access is suited to product features, internal agent platforms, evaluation systems, and automation pipelines. - Teams control prompts, retrieval, model routing, retries, logging, security, credentials, and billing. - Production agents still require substantial engineering, including: - Selecting relevant repository or document context - Preserving instructions - Handling failed tool calls - Storing traces and audit records - Defining data boundaries and approval points - Agent SDKs can provide orchestration, tools, sessions, and streaming. GitHub’s Copilot SDK exposes the runtime used by Copilot CLI and can run with either a Copilot subscription or a provider key. ## BYOK: Keeping Copilot’s Workflow - Copilot’s public-preview Bring Your Own Key feature lets teams use supported external models in Copilot Chat, CLI, and VS Code. - Supported providers include Anthropic, AWS Bedrock, Google AI Studio, Microsoft Foundry, OpenAI, OpenAI-compatible services, and xAI. - GitHub continues to provide the Copilot harness and integrations, while the customer pays the model provider directly. - BYOK can preserve existing cloud contracts or provider commitments while maintaining a familiar Copilot workflow. - Administrators can control which GitHub-hosted or BYOK models teams may use. - Because BYOK is still in public preview, teams should consult the current documentation before making purchasing or architecture decisions. ## Choosing the Right Layer - Choose raw API access when you need custom integrations, behavior, security controls, auditing, or billing. - Choose Copilot when developers primarily need to work faster within existing repositories, editors, terminals, issues, pull requests, reviews, and security processes. - BYOK is a middle option for teams that want GitHub’s development workflow but prefer to pay for models through an existing provider relationship. The practical decision is not simply about token price. It is about whether your team needs to build and operate the surrounding AI system or wants an integrated development workflow managed through Copilot.

Read original(opens in new tab)
gitlab2 min readCurated summary

Turn multi-step software delivery into agentic flows you can trust

GitLab Duo Agent Platform’s Custom Flows are now generally available, enabling teams to automate multi-step software delivery rather than relying on chat responses, scripts, or manual runbooks. Flows can start from GitLab events or Agentic Chat, execute tasks across projects, and include human approval checkpoints. The goal is dependable, repeatable automation for work such as implementing issues, repairing pipelines, and reviewing merge requests. ## Why Software Delivery Remains Manual - Real development tasks involve multiple steps: gathering context, changing code, opening a merge request, waiting for CI, and responding to reviews. - Without workflows, people must manually coordinate each handoff and remember undocumented team practices. - Teams previously hesitated to formalize these processes because Custom Flows lacked production readiness and sufficient event coverage. ## What Agentic Flows Provide - Automate trusted sequences triggered by events such as: - Mentions and assignments - Pipeline activity - Merge request lifecycle changes - Work item updates - Run under a composite identity, keeping permissions bounded and actions attributable. - Launch specialist workflows from Agentic Chat, including: - Developer Flow for implementing changes - Code Review Flow for reviewing merge requests - Fix CI/CD Pipeline Flow for diagnosing and repairing failures - Require user approval before handing work from chat to a flow, while displaying progress inline. - Use review exclusions to avoid automatically reviewing bot-authored or branch-pattern merge requests. - Apply custom review instructions so automated reviews match team standards. ## Custom Flow Configuration - Create flows from a project or the AI Catalog. - Set visibility, enable flows for selected projects, and attach GitLab event triggers. - Add human-in-the-loop checkpoints for sensitive operations. - GitLab 19.2 adds: - A “work item status changed” trigger - Bulk enablement of public flows across up to 100 projects - A future Flow Creation Agent is planned to generate runnable flow definitions from plain-language descriptions. ## Costs and Adoption - Event-driven flows consume GitLab credits according to the work they perform. - GitLab recommends testing flows in a small number of projects before broad deployment. - Access is available through a GitLab Duo Agent Platform trial or through included credits for Premium and Ultimate subscribers. Teams should begin by encoding a few well-understood delivery processes as Custom Flows, adding approval gates where necessary, and expanding gradually while monitoring credit usage and automation quality.

Read original(opens in new tab)
figma2 min readCurated summary

GPT-5.6 is Now Available in Figma Make | Figma Blog

GPT-5.6 is now available in Figma Make, where Figma says it improves both the speed and quality of AI-generated prototypes. The model is designed to produce stronger first passes, preserve existing designs more faithfully, and recover from errors without stopping. Figma’s examples suggest it can move teams from prompts or static designs to functional, responsive prototypes with less iteration. ## Faster exploration and error recovery - GPT-5.6 can turn complex prompts into working prototypes quickly, helping teams explore multiple ideas in one session. - In Figma’s stock-tracking app evaluation, it created: - An interactive dashboard - Sample prices and performance data - Keyboard shortcuts and search - A dark, gothic visual style - The model is described as more token-efficient, helping users make better use of Figma Make credits. - When builds encounter errors, GPT-5.6 can investigate and self-heal instead of stopping. Figma reports that it independently diagnosed and fixed a blank build. ## Faithful design-to-code conversion - GPT-5.6 can build prototypes from existing design specifications or Figma Design files. - In a nature sound player test, it preserved: - Layout and visual hierarchy - Spacing, proportions, and styling - A multi-track timeline and sound library - It also implemented functional interactions, including: - Play, pause, and skip controls - Working audio playback - Multiple playable tracks - Audio-responsive visual effects ## Higher-quality first passes - Figma says GPT-5.6 produces polished initial prototypes with functional interactions and responsive layouts. - A bookshelf e-commerce example included: - Product descriptions, measurements, and care information - Populated information dropdowns - An interactive product photo library - A clickable navigation menu - The prototype adapted reliably across different screen sizes without additional prompting. - Stronger first passes allow teams to spend more time refining ideas collaboratively rather than repairing basic implementation problems. GPT-5.6 is available through Figma Make’s model selector. Users can select it directly in Make and consult Figma’s help center for guidance on choosing and using AI models.

Read original(opens in new tab)
gitlab2 min readCurated summary

AI Catalog updates for governance and operations

GitLab 19.1 expands the AI Catalog from a manually triggered tool into a governed automation platform. New event-driven Duo Flow triggers, configuration validation, agent restrictions, and model allowlists help enterprises run AI workflows continuously while maintaining security and operational control. The release is designed to make agentic automation safer and more reliable at production scale. ## Event-Driven Duo Flow Automation - Four new triggers let flows respond automatically to GitLab events: - Merge request code conflicts - Draft merge requests becoming ready for review - Merge request approvals - New work item creation - These triggers enable automated conflict summaries, compliance checks, deployment readiness checks, notifications, triage, labeling, and routing. - Pipeline triggers can now filter for specific states: failure, success, or cancellation. - Conflict-detected and draft-to-ready triggers are enabled by default. - A beta pattern-based approval tier lets developers approve repeated tool uses for an entire session instead of approving each invocation individually. ## Governance for Agents and Flows - Instance administrators and top-level group owners gain controls over which AI content users can run. - **Disable custom agents and flows** prevents users from creating or enabling custom-built content. - **Restrict the AI Catalog to your group hierarchy** blocks AI Catalog items from outside the organization’s namespace, including community and third-party contributions. - These controls help prevent unapproved agents and workflows from entering regulated or production environments. ## Pre-Save Flow Configuration Validation - GitLab now validates AI Catalog flow configurations against the Duo Workflow Service before saving them. - Errors such as missing inputs or invalid tool parameters appear directly in the UI. - This moves troubleshooting to configuration time, reducing the risk of broken or overly active flows running in production. ## Approved AI Model Controls - A public beta lets administrators define an allowlist of approved AI models. - Organizations can also set an organization-wide default model. - Teams retain flexibility within approved boundaries while meeting provider, compliance, or data-residency requirements. - The first version applies to GitLab Duo Agentic Chat, with broader coverage planned. Overall, GitLab 19.1 recommends treating AI workflows like production automation: trigger them from real events, validate them before deployment, and govern both the agents and models they use. Enterprises adopting the AI Catalog should configure the new restrictions and model policies before enabling widespread automated flows.

Read original(opens in new tab)
gitlab2 min readCurated summary

Mythos-class Claude Fable 5 arrives on GitLab Duo Agent Platform

Claude Fable 5, Anthropic’s Mythos-class model, is now available through GitLab Duo Agent Platform across all tiers and deployment models. The post presents it as a major step beyond incremental model improvements, emphasizing stronger first-attempt accuracy, sustained autonomous workflows, and better code review and incident analysis. Its main conclusion is that teams should apply it to complex, previously difficult engineering problems rather than routine tasks. ## More Accurate First-Pass Results - Produces more correct implementations for complex, well-defined problems with fewer iterations. - Reduces back-and-forth in Duo Agentic Chat. - Performs especially well on: - Multi-file refactoring - Incident investigation - Infrastructure-as-code definitions - Interprets technical images, web applications, and detailed screenshots more accurately while often using fewer output tokens. ## Longer and More Reliable Agent Workflows - Sustains goal-directed work across extended, multi-day runs and millions of tokens. - Maintains instructions and task focus without frequent manual checkpoints or re-prompting. - Uses verification loops to identify and correct its own mistakes. - Handles parallel sub-agents more reliably across multiple repositories or services. - Reduces the amount of human oversight required per agent run, allowing teams to review results asynchronously. ## Improved Bug Detection and Incident Response - Provides higher bug-finding recall than previous models. - Improves outage triage, root-cause analysis, and repository-history investigation. - Finds deeper code-path issues and edge cases during merge request reviews. - Produces more actionable review comments instead of generic feedback. - Helps reduce production defects and improve mean time to resolution. ## Recommended Use Cases - The post recommends testing the model on difficult, unsolved engineering problems rather than routine work. - Suggested applications include: - Complex multi-file refactors - Production incident investigations - Implementations developers would normally write manually - Teams should allow agents to scope work, ask clarifying questions, and execute multi-step solutions. Claude Fable 5 became available on GitLab Duo Agent Platform on June 9, 2026. Teams can access it through a free trial, the GitLab free tier, or included GitLab Credits for Premium and Ultimate subscribers.

Read original(opens in new tab)
github3 min readCurated summary

GitHub recognized as a Leader in the Gartner® Magic Quadrant™ for Enterprise AI Coding Agents for the third year in a row

GitHub argues that AI coding has made code generation easier, shifting the main bottleneck to reviewing, securing, governing, and deploying software. It presents GitHub Copilot as an agentic platform spanning the full software development lifecycle, enabling developers to assign issues to agents and focus on reviewing and approving results. Gartner named GitHub a Leader in the 2026 Magic Quadrant for Enterprise AI Coding Agents, placing it highest for ability to execute for the third consecutive year. ## The Shift from Code Generation to Software Delivery - AI coding agents are increasingly expected to handle more than writing functions. - The harder problems now involve: - Code review - Security - Governance - Testing - Deployment - GitHub describes the new workflow as “orchestrating outcomes”: developers assign work to agents, then return to steer, review, and approve it. - Gartner projects that asynchronous AI coding-agent workflows could improve engineering productivity by 30%–50% by 2028, compared with 0%–20% gains from code assistants in 2025. ## Enterprise Adoption of GitHub Copilot - Copilot is used by 140,000 organizations, nearly three times the number reported a year earlier. - Overall growth exceeded 100% year over year. - Most users work with multiple AI models. - GitHub Copilot CLI usage nearly doubled month over month. - GitHub says these figures indicate that enterprises are adopting increasingly sophisticated, agent-driven workflows. ## Gartner’s 2026 Evaluation - Gartner evaluated 12 enterprise AI coding-agent vendors according to: - Ability to execute - Completeness of vision - GitHub was positioned as a Leader and ranked highest in ability to execute. - Gartner describes Leaders as vendors combining strong execution, market-shaping vision, rapid innovation, broad software-engineering relevance, and enterprise-grade security and governance. - The report’s Leader quadrant also includes Anthropic, Cursor, and OpenAI. ## GitHub’s Claimed Differentiators - **Developer choice:** Copilot supports multiple models and providers. - **Broad availability:** It works across editors, IDEs, CLIs, and GitHub’s web, desktop, and mobile applications. - **Full-lifecycle integration:** Copilot operates across issues, pull requests, code reviews, and GitHub Actions—not only inside the editor. - **Enterprise governance:** Teams can observe, audit, and secure how AI is used in engineering workflows. ## What GitHub Plans to Build Next - GitHub says it will expand agentic workflows across more developer-facing surfaces. - Planned investments include: - Greater model choice and intelligent model routing - Deeper integrations throughout the software lifecycle - Performance improvements based on how software is actually built and maintained on GitHub GitHub’s central recommendation is to treat AI coding agents as part of an end-to-end engineering platform rather than isolated code-generation tools. The post also notes that Gartner’s recognition is not an endorsement and that its findings should be considered alongside the full research report.

Read original(opens in new tab)
figma2 min readCurated summary

How to Design Agentic Tools for Work | Figma Blog

Gemini Enterprise is designed to make complex, multi-agent business workflows feel simple without hiding AI’s role. Its core principle is to keep users focused on goals while making intervention, accountability, and data context visible. The result is an agentic system that supports not only individual productivity but shared team intelligence. ## A Familiar Brand with Business-Specific Capabilities - Gemini Enterprise shares Gemini’s visual language, including the sparkle icon, gradients, rounded shapes, and motion. - Its enterprise experience emphasizes integrations with tools such as Google Workspace, Jira, and Notion. - Connectors are made prominent in the prompt experience so agents can access the business context needed to produce useful results. ## Moving Beyond Chat with the AI Inbox - Enterprise work often involves multiple tools, data sources, deadlines, and agents working simultaneously. - The AI Inbox provides a visual overview of: - Tasks agents are currently handling - Completed work - Items requiring human intervention - Deliverables awaiting review - This dashboard is intended to feel more like a team status check-in than a sequence of chat messages. ## Collaborative Projects as Shared Workspaces - Gemini Enterprise replaces isolated chat threads with persistent, shared project spaces. - AI participates as a visible team member by: - Performing tasks - Summarizing discussions - Finding project files - Answering questions about shared material - Requests are attributed to individual team members, improving accountability and helping others understand the context behind an agent’s actions. - Shared spaces reduce information silos by allowing teammates to discover and use one another’s uploaded materials. - The assistant becomes a single source of truth and a “team intelligence amplifier,” rather than merely a personal productivity tool. ## Multiple Modes of Team Interaction - Teams can communicate with AI in group chats within Collaborative Projects. - In Canvas Mode, the assistant can generate and edit documents. - These modes allow AI to remain embedded in ongoing team workflows instead of being limited to isolated prompts. Gemini Enterprise’s design recommendation is to combine powerful orchestration with clear visibility and human control. Agents should work proactively, but their actions, sources, status, and opportunities for intervention must remain understandable to the people responsible for the outcome.

Read original(opens in new tab)
github3 min readCurated summary

Continuous AI for accessibility: How GitHub transforms feedback into inclusion

GitHub built a continuous, AI-assisted accessibility feedback system to replace scattered reports, unclear ownership, and unresolved “phase two” promises. The workflow combines GitHub Actions, Copilot, and GitHub Models to turn user feedback into tracked, prioritized issues while preserving human judgment. Its goal is continuous follow-through: every accessibility barrier is captured, routed, reviewed, and acted upon. ## Accessibility as a Living System - GitHub treats accessibility as an ongoing methodology rather than a one-time audit or standalone product. - The approach combines: - Automation - Artificial intelligence - Human expertise - Real user feedback is considered more valuable than automated code scans because it reveals barriers experienced in real workflows. - The system supports GitHub’s 2025 Global Accessibility Awareness Day pledge to improve accessibility across the open source ecosystem. - Technology helps process feedback at scale, turning unstructured reports into clearer, implementation-ready work. ## Designing for Different Users The workflow was designed around three primary groups: - **Issue submitters** - Community managers, support agents, and sales representatives submit reports for users and customers. - Since they may not be accessibility specialists, the system guides them and teaches accessibility concepts during submission. - **Accessibility and service teams** - Engineers and designers need actionable reports containing reproducible steps, WCAG references, severity ratings, and ownership information. - **Program and product managers** - Leaders need trend data, issue categories, and progress visibility to prioritize investments. The design treats feedback as data moving through a pipeline and allows the process to evolve over time. ## Event-Driven Feedback Workflow - Each workflow stage triggers a GitHub Action that determines what happens next. - Key events include: - New issues launching Copilot analysis through the GitHub Models API - Status changes initiating hand-offs between teams - Resolutions triggering follow-up with the original submitter - Actions can be started manually or rerun, allowing humans to intervene whenever necessary. - GitHub initially built the system largely by hand in mid-2024; newer tools such as Agentic Workflows could now create similar Actions from natural-language instructions. - The workflow contains seven stages: - Intake - Copilot analysis - Submitter review - Accessibility team review - Link audits - Closing the loop - Improvement - Feedback loops allow submitters to rerun analysis, resolved issues to return for further review, and improvements to update Copilot prompts. ## Actioning Intake - Accessibility feedback can arrive through support tickets, social media, email, direct outreach, or GitHub’s accessibility discussion board. - Approximately 90% of feedback currently comes through the public discussion board. - Public discussions let other users: - Confirm reported problems - Add context - Share workarounds - Reports from the community often contain more detail than conventional support tickets. - GitHub acknowledges every report within five business days, including reports it cannot directly address. - When internal action is needed, a team member creates a tracking issue using a custom accessibility feedback template. - The template records: - The user’s original report - The feedback source - Relevant product components - This preserves important context as feedback moves from intake into triage.

Read original(opens in new tab)
github2 min readCurated summary

The era of “AI as text” is over. Execution is the new interface.

The post argues that AI is moving beyond text-based question-and-answer interactions toward embedded execution. The GitHub Copilot SDK lets applications use Copilot’s planning, tool use, file modification, command execution, and error recovery capabilities directly. This enables teams to build adaptable AI workflows without creating their own orchestration infrastructure. ## Delegating Multi-Step Work to Agents - Applications can express intent and constraints instead of hard-coding every workflow step. - For a task such as “Prepare this repository for release,” an agent can: - Explore the repository - Plan the necessary work - Modify files - Run commands - Recover and adapt when failures occur - This approach is more flexible than scripts, which become brittle when workflows depend on changing context or unexpected errors. - Teams can use agentic execution while maintaining defined boundaries and observability. ## Using Structured Runtime Context - Relying on prompts to contain system logic makes workflows difficult to test, maintain, and evolve. - The Copilot SDK supports structured, composable context through: - Domain-specific tools and agent skills - Model Context Protocol (MCP) - Runtime retrieval of relevant data - Agents can directly access systems such as: - Service ownership records - Historical decisions - Dependency graphs - Internal APIs - Permissioned tools and real-time data provide more reliable grounding than embedding organizational knowledge in prompts. ## Embedding Agents Beyond the IDE - Agentic capabilities can be integrated into: - Desktop applications - Internal operational tools - Background services - SaaS products - Event-driven systems - Applications can invoke Copilot in response to events such as file changes, deployments, or user actions. - Execution happens within the product itself rather than in a separate IDE or terminal interface. - This turns AI from an auxiliary developer tool into application infrastructure available wherever the software operates. ## Execution as a New Interface - Agentic workflows are programmable planning-and-execution loops that: - Integrate with real systems - Operate under constraints - Adapt during runtime - Use tools to complete tasks - The Copilot SDK provides this execution layer so teams can focus on defining outcomes instead of rebuilding orchestration systems. The practical recommendation is to treat AI as an executable application capability rather than merely a text interface. Teams can start by identifying multi-step workflows or event-driven tasks where structured tools, runtime context, and adaptive execution would provide more value than fixed scripts.

Read original(opens in new tab)
github3 min readCurated summary

Under the hood: Security architecture of GitHub Agentic Workflows

GitHub Agentic Workflows are designed to bring autonomous agents into CI/CD without giving them unrestricted access to repositories, secrets, or the internet. Because agents can be prompt-injected and behave unpredictably, GitHub treats them as untrusted components and compiles workflows into constrained GitHub Actions. The architecture relies on layered isolation, controlled communication, staged writes, and comprehensive auditing. ## Threat Model - Agents reason over repository state and act autonomously, so they cannot be trusted by default. - GitHub Actions normally place components in one permissive trust domain with broad access to: - Repository contents - Authentication secrets - MCP servers - Arbitrary network destinations - A malicious webpage, issue, or repository file could prompt an agent to: - Read credentials from files, environment variables, logs, or `/proc` - Upload secrets externally - Embed secrets in issues, pull requests, or comments - Make unwanted repository changes - Strict mode follows four principles: - Defense in depth - Never trust agents with secrets - Stage and vet writes - Log everything ## Layered Security Architecture GitHub Agentic Workflows use three complementary layers: - **Substrate layer** - Runs on a GitHub Actions runner VM. - Uses trusted containers, Docker isolation, network controls, and kernel-enforced boundaries. - Separates components and mediates privileged operations and system calls. - Is intended to contain damage even if an untrusted component is compromised. - **Configuration layer** - Defines which components run and how they connect. - Controls communication channels, privileges, firewall policies, Docker images, and MCP configuration. - Determines which tokens are loaded into which containers. - Converts declarative workflow configuration into a secure runtime structure. - **Planning layer** - Controls which components are active and how data moves between them over time. - Creates staged workflows with explicit data exchanges. - Uses the Safe Outputs subsystem to govern potentially dangerous operations. ## Keeping Secrets Away from Agents - In ordinary GitHub Actions, secrets may be visible through environment variables and configuration files across the shared runner trust domain. - This creates a major prompt-injection risk: an agent with shell access could discover credentials and exfiltrate them. - Agentic Workflows instead place the agent in a dedicated container with: - Firewalled internet access - MCP access through a trusted gateway - LLM communication through an API proxy - A private network connects the agent only to approved services. - The trusted MCP gateway launches MCP servers and exclusively handles MCP authentication material. - LLM authentication tokens are kept in the isolated API proxy rather than exposed directly inside the agent container. ## Controlled Execution and Writes - Open-ended workflow authoring is separated from governed execution. - Workflows are compiled into GitHub Actions with explicit constraints covering: - Permissions - Outputs - Network access - Auditability - The planning and Safe Outputs systems are intended to mediate GitHub write operations and apply controls such as call filtering, volume limits, secret removal, and moderation. GitHub’s approach is to treat agents as untrusted CI/CD components rather than granting them normal workflow privileges. Organizations adopting agentic automation should isolate agents, broker access to tools and credentials, restrict network connectivity, stage all writes for review, and maintain detailed logs.

Read original(opens in new tab)
dropbox3 min readCurated summary

With Mobius Labs' Aana models, we're bringing deeper multimodal understanding to Dropbox Dash

Dropbox is integrating Mobius Labs’ Aana multimodal models into Dropbox Dash to improve how it understands text, images, audio, and video together. Aana is designed for efficient, large-scale processing, enabling Dash to interpret scenes, actions, speech, and relationships across media rather than relying on isolated metadata. The goal is to make rich media searchable and support future AI agents that can analyze content and act on it. ## Dropbox Dash’s Multimodal Goal - Dash combines AI assistance and search to understand a team’s work, context, and tools. - Its multimodal capabilities are intended to cover formats such as: - Text - Photos - Videos - Audio - This is especially valuable for creative, technical, media, and marketing teams whose work is distributed across many applications and content repositories. ## Why Multimedia Understanding Is Difficult - A video may require analysis of: - Scene changes - Speaker shifts - On-screen text - Objects and actions - Audio cues and dialogue - Images require systems to understand people, locations, events, and visual context. - Each modality has different timelines and semantic boundaries; transcripts, shots, visuals, and audio do not automatically align. - Effective understanding requires connecting modalities—for example, linking what someone says to the diagram they are showing. - Processing this information across exabytes of content would be prohibitively expensive without highly efficient infrastructure. ## How Aana Combines Models and Context - Aana analyzes images, audio, video, and related text as a connected system instead of separate data streams. - It combines fine-tuned, open-source foundation models for: - Speech, including optimized Whisper-based models such as `faster-whisper-large-v3-turbo` - Vision - Language - Transformer and mixture-of-experts architectures support fast inference on standard GPUs. - Aana tracks how scenes evolve over time, including: - Object movement - Actions - Changing layouts - Cross-modal events - Insights are stored in a shared vector space, enabling searches such as “the part where the presenter explains the API flow” rather than requiring users to scrub through a video manually. ## Efficiency at Dropbox Scale - Aana uses low-bit 8-bit and 4-bit inference through its HQQ system to reduce memory and compute requirements. - Gemlite provides custom GPU kernels to accelerate matrix multiplication and attention operations. - The Aana SDK coordinates models, batching, GPU utilization, and production pipelines. - Teams can configure and deploy different combinations of models while experimenting with new workflows. - These optimizations are intended to make analysis of exabytes of media possible with a much smaller compute footprint than conventional approaches. ## Future Applications in Dash - Aana could help users find specific visual elements across large creative archives. - It could summarize years of client meetings into searchable highlights. - Its capabilities may support agentic workflows that: - Analyze multimedia - Surface relevant insights automatically - Generate ideas - Take actions for teams Dropbox’s integration of Aana is intended to make Dash a more context-aware search and AI assistant for rich media. The practical outcome is less time spent organizing and manually reviewing content, and more time spent creating and acting on information.

Read original(opens in new tab)
googleOriginal article

Learn Your Way: Reimagining textbooks with generative AI (opens in new tab)

Google Research has introduced Learn Your Way, an AI-driven educational experiment that reimagines traditional textbooks as personalized, multimodal learning journeys. By leveraging the LearnLM family of models integrated into Gemini 2.5 Pro, the system transforms static source material into tailored content based on a student’s specific grade level and interests. Early efficacy studies demonstrate that this approach significantly enhances retention, with students scoring 11 percentage points higher than those using standard digital readers. ### Pedagogical Foundations and Dual Coding The research is built on the "dual coding theory," which suggests that forming mental connections between different representations of information strengthens conceptual understanding. * The system moves away from a "one-size-fits-all" model toward a student-driven experience where learners can choose and intermix formats. * Personalization is used as a tool to enhance situational interest and motivation by adapting content to specific student attributes. * The framework incorporates active learning through real-time quizzing and feedback to address knowledge gaps as they arise. ### The Personalization Pipeline The technical architecture begins with a layered pipeline that processes source material, such as a textbook PDF, to create a foundational text for all other formats. * The original material is first "re-leveled" to match the learner’s reported grade level while maintaining the integrity and scope of the curriculum. * Generic examples within the text are strategically replaced with personalized examples based on user interests, such as sports, music, or food. * This personalized base text serves as the primary input for generating all subsequent multimodal representations, ensuring consistency across formats. ### Multimodal Content Generation To produce a wide variety of educational assets, the system utilizes a combination of large language models and specialized AI agents. * **Agentic Workflows:** While tools like mind maps and timelines are generated directly by Gemini, complex assets like narrated slides use multi-step agentic workflows to ensure pedagogical effectiveness. * **Custom Visuals:** Because general-purpose image models often struggle with educational accuracy, the researchers fine-tuned a dedicated model specifically for generating educational illustrations. * **Diverse Representations:** The interface provides "immersive text" with embedded questions, audio lessons for auditory learning, and interactive slides that mimic recorded classroom sessions. ### Research Outcomes and Future Application The project’s effectiveness was validated through a study comparing the GenAI approach against standard digital reading materials. * Students using the personalized AI tools showed a significant improvement in retention test scores. * Beyond retention, the system aims to transform passive reading into an active, multimodal experience that follows established learning science principles. * The "Learn Your Way" experiment is currently available on Google Labs, providing a practical look at how adaptive, learner-centric materials might replace static textbooks in future K-12 and higher education settings.

googleOriginal article

Geospatial Reasoning: Unlocking insights with generative AI and multiple foundation models (opens in new tab)

Google Research is introducing Geospatial Reasoning, a new framework that integrates generative AI with specialized foundation models to streamline complex geographical problem-solving. By combining large language models like Gemini with domain-specific data, the initiative seeks to make large-scale spatial analysis accessible to sectors like public health, urban development, and climate resilience. This research effort moves beyond traditional data silos, enabling agentic workflows that can interpret diverse data types—from satellite imagery to population dynamics—through natural language. ### Specialized Foundation Models for Human Activity * The Population Dynamics Foundation Model (PDFM) captures the complex interplay between human behaviors and their local environments. * A dedicated trajectory-based mobility foundation model has been developed to process and analyze movement patterns. * While initially tested in the US, experimental datasets are expanding to include the UK, Australia, Japan, Canada, and Malawi for selected partners. ### Remote Sensing and Vision Architectures * New models utilize advanced architectures including masked autoencoders, SigLIP, MaMMUT, and OWL-ViT, specifically adapted for the remote sensing domain. * Training involves high-resolution satellite and aerial imagery paired with text descriptions and bounding box annotations to enable precise object detection. * The models support zero-shot classification and retrieval, allowing users to locate specific features—such as "residential buildings with solar panels"—using flexible natural language queries. * Internal evaluations show state-of-the-art performance across multiple benchmarks, including image segmentation and post-disaster damage assessment. ### Agentic Workflows and Industry Collaboration * The Geospatial Reasoning framework utilizes LLMs like Gemini to manage complex datasets and orchestrate "agentic" workflows. * These workflows are grounded in geospatial data to ensure that the insights generated are both useful and contextually accurate. * Google is collaborating with inaugural industry partners, including Airbus, Maxar, Planet Labs, and WPP, to test these capabilities in real-world scenarios. Organizations interested in accelerating their geospatial analysis should consider applying for the trusted tester program to explore how these foundation models can be fine-tuned for specific proprietary data and use cases.