Getting AI to Provide Investment Information (opens in new tab)
LLMs make it easy to generate financial content, but producing trustworthy investment information requires much more than fluent summaries. Toss Securities argues that AI must pass three gates before reaching users: selecting reliable evidence, controlling how responses are generated, and making outputs measurable and improvable. The central principle is to constrain AI autonomy where reproducibility and traceability matter, while preserving it for open-ended exploration. ## Why Investment Information Is Different - **Timeliness:** Market interpretations can change within hours due to earnings, geopolitical events, or policy news. - **Accuracy:** A company mentioned in an article may not be the company whose stock moved; it could be a subsidiary, a similarly named firm, or merely a promotional mention. - **Traceability:** Every generated claim needs supporting evidence, evaluation records, and reproducible processing. - **Non-stationarity:** Market behavior changes across earnings seasons, interest-rate events, elections, and geopolitical crises. Prompts and models tuned to one period may degrade later. LLMs and autonomous agents amplify these challenges: - LLMs can produce fluent but incorrect answers when evidence is incomplete or ambiguous. - Agents add more failure points through search, tool calls, planning, and state transitions. - Errors can propagate through different execution paths, increasing operational cost and making debugging difficult. ## Gate One: Selecting What the AI Should Say The first gate is a context-engineering process that filters and organizes evidence before it reaches the LLM. ### Classify Data at Ingestion - News, disclosures, and financial data are classified as they arrive using internally developed BERT-based models. - Metadata includes: - Taxonomy tags - Related companies and entities - Embeddings for vector search - Pre-classifying data avoids waiting until retrieval to determine whether it is relevant. ### Retrieve Broadly, Then Narrow the Candidates A hybrid retriever first prioritizes recall, after which candidates are reduced through: - **Deduplication:** Semantically similar articles are clustered so one event is not treated as many independent events. - **Reranking and filtering:** Evidence is evaluated for direct relevance to the company’s price movement. - **Taxonomy labels:** Items are categorized by explanation type, such as earnings, guidance, or corporate actions. - **Failure labels:** Promotional content, insufficient evidence, and other unsuitable sources are explicitly marked and filtered out. - **Rubrics:** Evidence is ranked according to predefined relevance criteria. ### Build Reasoning-Friendly Context The final context is arranged so the model checks: - What happened - How the event connects to the target company - Whether the evidence’s polarity matches the stock’s price direction - Whether the evidence is sufficient and current This ordering combines the filtered evidence with metadata such as the company, price direction, and time window. ## Gate Two: Controlling How Responses Are Generated The second gate limits the action space of LLMs and agents to satisfy product requirements such as cost, latency, reproducibility, and observability. ### Use Task Graphs for Clearly Defined Work Instead of leaving the entire process to an autonomous agent, Toss Securities separates it into explicit stages: - Candidate retrieval - Relevance assessment - Deduplication - Evidence construction - Final response generation Each stage has defined input and output schemas, making it a debugging and evaluation point while simplifying fallbacks and operational monitoring. ### Choose Autonomy Based on Requirement Clarity - **Autonomous agents** are useful for open-ended tasks such as discovering investment ideas or exploring possible market scenarios. - **Procedural orchestration** is better for fixed tasks, such as explaining why a specific stock moved. - Long ReAct loops increase tool calls, token usage, latency, and trace-management costs. - For structured products, deterministic pipelines let LLMs focus on summarization, rewriting, and evidence-based explanation rather than tool selection. Procedural graphs are not merely a replacement for agents. Once defined, they can become reusable tools or sub-agents that other agents call through structured interfaces, such as: ```text input: ticker, direction, time_window output: explanation, evidences, reasoning_type ``` ## Gate Three: Making the System Evaluatable Subjective judgments such as “the answer feels weak” do not provide a reliable improvement loop. The system therefore generates structured classifications alongside natural-language responses. ### Generate Rubric Categories with Each Answer - Outputs include event or reasoning types and failure categories. - Structured fields make it possible to measure: - Relevance false positives - Directional mismatches - Irrelevant evidence passing the filter - Precision, recall, and F1 score - The taxonomy must evolve as new market regimes and failure patterns appear. - Operational failures, evaluation sets, prompt versions, and model versions should be linked so improvements can be reproduced and quantified. ### Retrieve Context-Specific Few-Shot Examples Fixed few-shot examples are insufficient because event and failure types vary widely across market conditions. Instead: - Store operational samples with their decisions, failure labels, and embeddings. - Embed each new classification or verification task. - Retrieve similar positive and negative examples. - Include both successful and failed examples to show the model the decision boundary. This approach reuses production failures as future evaluation guidance and significantly improves precision and accuracy while preserving recall. Since false positives are especially damaging in investment services, filtering out unsupported explanations is more important than producing fluent text alone. ## Work Beyond Prompts and Model Training Building an investment-information AI service also requires substantial infrastructure outside the model itself: - Retrieval strategies and embedding models for finding relevant evidence - Separately trained classifiers for categorization - Evidence filtering, validation, and metadata management - Structured orchestration, monitoring, evaluation, and feedback loops The practical recommendation is to treat the LLM as one component in a controlled evidence pipeline—not as the sole decision-maker. Use autonomous agents for exploratory tasks, but rely on traceable procedural graphs, evolving taxonomies, and retrieval-based examples when the product must deliver repeatable, defensible financial information.
