AI evaluation and monitoring
Your AI's performance,measured, not assumed.
AI evaluation and monitoring means repeatable tests and continuous tracking for an AI system that is already in use. An assistant that worked well at launch can degrade without anyone noticing: documents change, the provider updates the model, users ask different questions. We build the tests and monitoring that make these changes visible in time.
“It seems to work” is not a measurement
An AI system's quality is often judged on a few test conversations and on complaints. When someone changes an instruction or the model, nobody systematically checks whether the answers that used to be good are still good.
- Changes to instructions or the model are released without a regression test
- Weak answers are discovered by users, not by the team
- Cost and response time creep up gradually, without anyone noticing
- User feedback is collected but never feeds back into improvements
Who it is for
- Companies that already have an assistant, a RAG system or an AI feature in production
- Teams that often change the instructions, sources or model and want to know whether they have broken something
- Quality, compliance or risk managers who need to document how the AI is controlled
- Managers who want to know what the AI costs and whether it is worth expanding
What we build for quality control
- 01
Test sets
Real questions and tasks with expected results, including difficult cases and questions with no answer.
- 02
Regression checks
The test set runs at every change of model, instructions or sources, before release.
- 03
Human spot-check review
A sample of real interactions is regularly assessed by people, against written criteria.
- 04
Drift detection
Flags when the type of questions, the sources or the model's behaviour moves away from the baseline.
- 05
Cost and latency
A dashboard with usage, cost per interaction and response times, with alerts at set thresholds.
- 06
The feedback loop
Answers marked as wrong become new cases in the test set and correction tasks.
How it works
The evaluation and monitoring cycle
The steps marked in yellow are approved by a person.
- Input
- Action
- AI
- Human in the loop
- Result
- 01
Input
Real interactions
Questions, answers, sources and feedback are logged, with sensitive data protected.
- 02
Action
Automatic metrics
Cost, latency, errors and quality signals are calculated continuously.
- 03
AI
Automatic pre-assessment
An evaluator model flags suspicious answers against the agreed criteria.
- 04
Human in the loop
Spot-check review
Human reviewers score a sample and confirm or reject the flags.
- 05
Action
Regression test
New cases join the test set, which runs before any change.
- 06
Result
Report and actions
The team receives the regular report with the problems found and the proposed fixes.
What information evaluation uses
- The AI system's logs: questions, answers, sources, duration, cost
- Written quality criteria: what a correct, complete and safe answer means
- Real questions and tasks with answers validated by experts
- Time set aside by reviewers on your team for the spot-check review
- The accepted thresholds for cost, latency and quality
Possible integrations
- Langfuse
- LangSmith
- Arize Phoenix
- OpenTelemetry
- Grafana
- Datadog
- Sentry
- Google Sheets
- Slack
- Langfuse
- LangSmith
- Arize Phoenix
- OpenTelemetry
- Grafana
- Datadog
- Sentry
- Google Sheets
- Slack
Each integration's compatibility is checked during the analysis phase, for the systems and versions you use.
Control and transparency
What stays a human decision
- Defining the quality criteria and the reference answers
- The final score in the spot-check review — people have the last word over the automatic evaluator
- Deciding to release or postpone a change after the regression test result
- The alert thresholds and the actions that follow when they are exceeded
Limitations and how we handle them
- A test set covers only the cases you have thought of; new types of question may slip through until they are added to the set
- The automatic evaluator — a model scoring another model — also makes mistakes, which is why it is regularly compared with human scores
- Spot-check review does not check every interaction
- Monitoring shows the problems, but fixing them takes separate work on sources, instructions or the model
- Logging conversations may involve personal data and must be limited to what is necessary
What we track in the reports
We agree the indicators together before launch and track them afterwards. The values depend on the process, the data and the volume.
- Quality on the test set
- Each version's result on the same cases, compared with the previous version before release.
- Agreement between evaluators
- How often the automatic evaluator and people give the same score, checked at every spot check.
- Test coverage
- How many of the real question types are represented in the test set, reviewed regularly.
- Cost and latency
- Cost per interaction and response time, tracked against the agreed thresholds.
- Time to fix
- How long it takes from a wrong answer being flagged to it being fixed, from the issue log.
How we implement evaluation and monitoring
- 01
Criteria and baseline
We define with your team what a good result means and build the first test set.
- 02
Instrumentation
We add logging and metrics to the existing AI system, with data protection set up from the start.
- 03
Automated testing
We link the test set to the release process, so no change goes out untested.
- 04
Review schedule
We set how often human spot checks and reports happen and who decides based on them.
Use cases
These scenarios are conceptual examples showing what can be built. They are not delivered implementations or results achieved for clients.
- Regional internet provider
Monitoring a support assistant
The assistant's answers are spot-checked weekly, and new questions about subscriptions are added to the test set.
Conceptual example - Software company
Changing the model without surprises
Before moving to a new model version, the test set runs on both, and the team compares the differences and decides.
Conceptual example - Online store
Costs under control
The cost dashboard shows usage rising after a change of instructions, and the team adjusts it before the monthly bill.
Conceptual example
Frequently asked questions
Do we need this if the AI system already works?
That is exactly when you need it. Monitoring shows whether the system keeps working after changes to documents, the model or usage, not just on launch day.
How is it different from testing at launch?
Testing at launch is a snapshot. Continuous evaluation repeats the same tests at every change and tracks real interactions over time. The pages on RAG or LLM integration describe how the system is built; this one describes how you keep it under observation.
Can you evaluate an AI system you did not build?
Usually yes, if we have access to the logs or can add instrumentation. We check the options during the initial analysis.
What happens when the automatic evaluator is not sure?
The case goes to human review. The automatic evaluator only pre-sorts; its scores are regularly compared with people's, and the criteria are adjusted when differences appear.
What data is kept in the logs?
Only what is needed for evaluation, with personal data masked where possible, access limited to designated people and a retention period set in line with the GDPR.
How big does the test set need to be?
It depends on how many types of question the system receives. We start from the most frequent real cases and the high-risk ones, and expand it based on feedback.
What is a regression test for AI?
It is running the same set of questions and tasks, with known expected results, after every change of model, instructions or sources. It shows whether the answers that were good are still good, before the change reaches users.
Can’t find the answer? Write to us and we’ll reply directly.
Want to know, not assume, how your AI performs?
Tell us which AI system you use and what changes most often. We will suggest the test set, the metrics and the review schedule.
Related solutions
- RAG systemsAssistants that search the configured sources before answering and show where each piece of information comes from.
- LLM integrationLanguage-model features added to the applications and processes you already use, with the model chosen for cost and confidentiality.
- AI agent for customer supportAnswers customers from approved information, checks order status in your systems and hands over to colleagues the cases that need a person.