How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems

Bits AI uses large language models to generate first drafts of incident postmortems from Datadog Incident Management metadata and Slack discussions. The goal is to reduce documentation effort without replacing the human-led reflection that makes postmortems valuable. Datadog found that production-quality LLM features require extensive prompt experimentation, safeguards against hallucinations, careful model selection, and strong privacy and feedback mechanisms.

Why LLMs Are Difficult to Use for Postmortems

• Postmortems must remain factual, consistent, and useful, unlike more creative LLM applications.
• LLMs can ignore instructions, produce inconsistent formatting, repeat information, or hallucinate plausible but false details.
• More than 100 hours were spent refining section structures and instructions so they worked across varied incidents.
• Building these systems requires a combination of software engineering, product management, data science, and technical writing.

Balancing Structured and Unstructured Incident Data

• Structured Datadog data includes incident metadata, alerts, graphs, and manually entered fields such as customer impact.
• Slack discussions provide current, detailed, and unstructured context from the incident response process.
• Manually maintained fields are often more reliable but can become outdated, while LLMs can continually reassess live discussions.
• Combining both sources helped the model understand the full incident and prioritize information appropriately.

Cost, Speed, and Quality Trade-offs

• Datadog compared models including GPT-3.5 and GPT-4.
• Results varied by as much as 50 times in cost and speed.
• GPT-4 generally produced more accurate output but was substantially slower and more expensive.
• Model choice therefore depended on the required balance between quality, latency, and operating cost.

Preserving Human Control and Privacy

• AI-generated drafts were intended to support authors, not become final documents automatically.
• Keeping engineers involved preserves the discovery and learning process that postmortems are meant to encourage.
• Because incidents may contain sensitive information or secrets, Datadog scrubbed data before sending it to an LLM.
• The system was designed to ensure authors remained responsible for reviewing and improving the draft.

Experimentation and Reliability Safeguards

• A custom API extracted and structured incident data from Datadog, Slack, and other sources.
• The framework enabled rapid testing of different datasets, models, input configurations, and output token limits.
• Datadog refined prompts and instructions to guide the model toward useful recommendations while retaining author agency.
• Lowering model temperature and tuning other parameters reduced irrelevant or fabricated content.
• A related feature generated concise incident summaries for responders joining Slack channels, providing a faster testing environment for LLM-based incident analysis.

Evaluating Generated Drafts

• Datadog used both qualitative and quantitative evaluation methods.
• Authors of earlier postmortems compared their human-written documents with AI-generated drafts for the same incidents.
• Evaluation included criteria such as factual accuracy and conciseness, with the provided article ending before the full evaluation approach was described.
• Continuous feedback was used to refine prompts, models, and the overall drafting workflow.

Overall, reliable AI-assisted postmortems require more than selecting a capable model. The strongest approach combines carefully prepared incident data, privacy controls, model and prompt experimentation, systematic evaluation, and mandatory human review.