Skip to content

Evaluation and Continuous Improvement

After establishing the conversational interface and powering it with a robust semantic foundation, Numbers Station prioritizes the accuracy and continual refinement of its outputs. Accuracy is not limited to correctness of a single answer; it must incorporate the system's ability to recall past solutions, leverage existing analytics, and continuously improve based on user feedback and real-world usage patterns. Additionally, our agentic approach and specialized Text-to-SQL techniques enable us to enhance the accuracy and performance of even the most advanced Large Language Models (LLMs) like GPT-4 and others, outperforming them in complex, messy data environments (see image below).

Refine Accuracy and Continuous Improvement

Evaluation Through a Three-Pillar Architecture

The system's evaluation emerges from a structured approach:

  • Curated Knowledge Base: The Knowledge Layer, enriched with human-validated metadata, ensures query logic aligns with established business definitions.
  • Strategic Insight Utilization: The multi-agent workflow (Planner, Search, Intent, Query, Charting, Action) continuously checks for relevance, uses historical solutions, and refines SQL to produce contextually relevant answers.
  • Agentic Framework Integration: Each agent specializes in a particular aspect of query resolution, working together under the Planner Agent's coordination to validate correctness and consistency before finalizing responses.

Benchmarks and internal evaluations show that this architectural approach yields improvements over conventional query generation and retrieval tools. Whether responding to known, repetitive questions or generating novel analyses, the layered semantics and agent collaboration help ensure that answers align with business expectations.

Balancing Accuracy With Recall and Efficiency

In practice, a large portion of enterprise queries—often 80–90%—are not new. They reference scenarios that have been answered previously in dashboards, documents, or conversation histories. Numbers Station emphasizes recall capabilities to surface these established insights quickly:

  • High Recall for Existing Answers: The Search Agent retrieves previously answered questions or relevant dashboards. This reduces time-to-insight by recycling validated knowledge, increasing user trust and productivity.
  • Reduced Redundancy: By capitalizing on existing solutions, the system avoids unnecessary recomputation, allowing users to benefit from prior work and institutional memory.

For the 10–20% of genuinely new queries, the platform leverages the Knowledge Layer, refining SQL and applying semantic context to produce accurate, actionable results. This combination of recall and novel query generation ensures that accuracy is useful and efficient in real-world conditions.

Beyond Accuracy—Measurable Business ROI

While accuracy is important, Numbers Station also focuses on turning correct answers into business value. By reducing the time spent locating known answers, empowering self-service analytics, and providing immediate, context-rich insights, the system improves decision-making efficiency. Instead of incrementally optimizing accuracy metrics alone, the platform enables tangible gains—faster analysis cycles, more informed decisions, and ultimately, a better return on your analytics investments.

Continuous Feedback Loops and Monitoring

Over time, usage patterns and direct user feedback help Numbers Station improve:

  • User Feedback Tracking: The system records user queries, monitors satisfaction signals (e.g., acceptance of results, follow-up clarifications), and identifies where answers fall short.
  • Discrepancy Flagging: If an answer is deemed incomplete or inaccurate, it can be flagged for review by domain experts. These experts can update metadata, refine relationships, or correct misunderstandings in the Knowledge Layer.
  • Iterative Improvement: Each interaction, correction, and refinement enhances the underlying semantic understanding. As corrections accumulate, the platform continually adapts, becoming more aligned with the business's evolving needs and standards.

Summary

The feedback loop ensures that Numbers Station is never static. Instead, it is a living system that learns from each query and user action, incrementally improving accuracy, recall, and relevance. By continuously optimizing the balance between correctness, speed, and context, the platform grows more aligned with the organization's goals. Over time, what begins as a strong baseline of accuracy and recall evolves into a sophisticated, continuously improving analytics environment that saves time, provides measurable ROI, and better supports complex decision-making.