Building an Enterprise LLM Service Part
FAA achieves a 96.1% response rate by favoring simple, maintainable techniques over complex AI architectures. Its design choices were to use RAG instead of knowledge-focused fine-tuning, retrieve complete documents before cutting them into question-relevant sections, and rely on a basic ReAct agent loop rather than elaborate workflows or multiple agents. The article concludes that improving documentation is more valuable than adding complexity when unanswered questions mainly result from missing source material. ## RAG Instead of Fine-Tuning - Fine-tuning was rejected as the primary method for injecting enterprise knowledge. - Research cited in the article found that fine-tuning was highly effective for changing a model’s style—about 97% success—but achieved only about 11% accuracy when teaching new factual knowledge. - FAA’s experiment with approximately 40 examples showed that the model answered the exact training question correctly but failed when the wording changed slightly. - Maintaining larger fine-tuning datasets would require experts to create, verify, and continuously update training examples whenever product documentation changes. - RAG is better suited to frequently changing product information because only the source documents need to be updated. - Fine-tuning may still be useful for domain-specific terminology or reasoning patterns, but not for keeping FAA’s product knowledge current. ## Retrieving Whole Documents Instead of Pre-Chunking - Conventional RAG systems split documents into small chunks before embedding them, improving semantic search precision. - Pre-chunking can remove essential context, especially when references such as “this case” or “the following settings” are separated from the text they depend on. - FAA’s documents are generally short, well-structured, focused on one product and topic, making whole-document retrieval practical. - Instead of chunking before search, FAA embeds and retrieves complete documents, then splits them after the relevant document is known. - The post-split process has two stages: - Split the document by Markdown headers into meaningful sections. - Use a lightweight LLM to select only the sections relevant to the user’s question. - For a question about creating and deleting a VM, the main model might receive only the “VM creation” and “VM deletion” sections. - This extra filtering call remains inexpensive because the lightweight model outputs only section indexes rather than generating a full response. - The key advantage is that splitting happens after the system understands the question, preserving context while delivering only the necessary information. ## ReAct Instead of Complex Agent Workflows - FAA tested plan-and-execute workflows, in which the model first creates a multi-step plan and then carries it out. - Planning and replanning increased system complexity without producing a noticeable improvement in answer quality. - With well-designed tools and carefully filtered context, the model was able to determine tool order on its own. - FAA therefore uses ReAct: the model reasons, takes an action, observes the result, and decides what to do next. - This approach allowed the agent to handle troubleshooting questions without a separate planning layer. ## Rejecting Multi-Agent Architectures - The team also tested specialized agents, such as separate VM and Kubernetes experts. - Delegating questions and assembling the results required additional LLM calls, increasing response time from roughly 9 seconds to 14 seconds in one test. - Multi-agent routing performed poorly for cross-domain questions, such as moving data from a VM to object storage. - Specialists could miss information outside their assigned domain, whereas a single agent could maintain the complete context. - FAA therefore kept one agent with access to progressively disclosed tools and relevant documentation. ## Documentation as the Main Bottleneck - Analysis of unanswered questions showed that about 50% were caused by a documentation gap: no reference document existed. - Other failures were mostly temporary API issues or questions outside FAA’s intended scope. - This suggests the core retrieval and agent system performs well when documentation is available. - The team shares missing questions with product teams, whose updated documents are then re-embedded and incorporated into future evaluations. The practical recommendation is to start with the simplest architecture that fits the data: use RAG for changing knowledge, preserve document context during retrieval, and let a capable model operate through a ReAct loop. In enterprise systems, improving the underlying documentation may produce greater gains than adopting more sophisticated AI frameworks.
Read original(opens in new tab)
