Evaluations in Solace Agent Mesh
From vibes to evidence, with a framework you already have
You’ve built an agent in Solace Agent Mesh. It runs, and someone on your team dropped a “looks good” in a Slack thread. Here’s the question worth sitting with: have you actually tested it? Solace Agent Mesh now ships an evaluation framework for exactly this, and it answers a sharper question than “looks good.” Is this agent still doing the right thing? You run it against a fixed dataset of prompts, score the outputs with a configured evaluator, and compare scores across runs, so “looks good” turns into a number you can track, gate on, and trust.
If you have not reached for it yet, it is closer than you think. Evaluations are in Solace Agent Mesh platform with a Web UI for managed runs and historical comparison. They offer datasets, evaluators, experiments, and run history, so the same experiment can carry an agent from a quick local sanity check to a production gate that blocks a regression before it ships.
Offline Scoring, Not Observability
Evaluation and observability answer different questions, and a healthy deployment uses both. Observability is what happens to live traffic: per-task latency, per-tool error rates, the signals you watch while real users are in the system. Offline evaluations score against a fixed dataset and run on demand so you can ask whether a change made the agent better or worse under controlled conditions before anyone is affected.
It is also not a unit-test runner. Evaluators score with tolerance using edit distance, word overlap, or an LLM-as-a-judge (and more), rather than asserting exact equality. This is because a string match is the wrong bar for a natural-language, non-deterministic answer.
How to make use of offline evals
For everything beyond a sanity check, the Platform-managed path is what production deployments use, and it is the system of record. You define three things: a dataset (a named set of prompts with optional expected responses), an evaluator (how a response gets scored), and an experiment (a dataset bound to a target agent and a list of evaluators). Triggering an experiment produces a run — one execution of that experiment, with one run per model when you compare models side by side.. You author these as declarative config or build them in the Web UI, where the Evaluations area provides Reports and an Experiment Lab for browsing history, viewing per-example results, and comparing experiments side by side.
How Responses Get Scored
Evaluators come in two kinds, and most suites mix them. Heuristic evaluators are rule-based and ship pre-registered. LLM-as-a-Judge evaluators ask a model to score the response on a labeled scale, and Solace Agent Mesh seeds six of them, covering a general-purpose Judge plus Factuality, Summary, SQL, Translation, and Security. You can also author your own judge with a custom prompt template and scoring scale
| Evaluator | Description | Type |
|---|---|---|
| Factuality | Evaluates whether responses are factually accurate when compared to reference answers in the dataset. Helps determine whether the agent is finding correct information. | LLM as a Judge |
| Response Quality | Evaluates overall response quality using an AI model as the evaluator, with quality scored as Fail, Partial, or Pass. | LLM as a Judge |
| JSON Similarity | Measures recursive structural similarity between two JSON objects, scored from 0.0 to 1.0. | Heuristic |
| Character Similarity | Measures text similarity at the character level between the response and the expected response. Recommended for code responses rather than natural language. Scored from 0.0 to 1.0. | Heuristic |
| Text Similarity | Measures text similarity at the token level, comparing the response with the expected response. Recommended for natural language responses. | Heuristic |
| Security | Evaluates whether a response is free from security risks, such as prompt injection, harmful content, and policy violations. | LLM as a Judge |
| SQL Accuracy | Evaluates whether a SQL query in a response is logically equivalent to the reference query provided in the dataset. | LLM as a Judge |
| Summary Quality | Evaluates whether a summary accurately captures the key points of the original text. | LLM as a Judge |
| Translation Quality | Evaluates whether a translation accurately and naturally preserves the meaning of the original text. | LLM as a Judge |
| Valid JSON | Evaluates whether a response is valid JSON, scored as 1.0 for valid and 0.0 for invalid. | Heuristic |
The detail that shapes what a judge can verify is what it sees. Today, every LLM-as-a-Judge call receives the agent’s response text and any artifacts it produced. Execution trace data — tool calls, parameters, and results — is collected and saved alongside each run, but is not yet passed to evaluators. Criteria therefore work from the agent’s output: a judge can assess whether an answer is factually correct, whether a summary is faithful, or whether a translation holds meaning, but cannot directly verify which tools were invoked to produce it. Scores run from 0% to 100%; a per-example result counts as passed at or above 50%, and each run reports the average score per evaluator so you can compare runs against one another.
When to Run
Evaluation pays off well before production:
- Before any change that affects behavior, whether you edit instructions, swap a model, add a tool, or change delegation.
- When you are choosing between models, one experiment can produce a run per model and the Web UI lines the results up side-by-side.
- When a provider pushes an update, because a new model version can greet correctly yet fail at tool delegation, and you would not know until a user complained.
- On a cadence before every deploy, ideally inside CI, where the run becomes a gate rather than a one-time check.
The Upgrade Gate
The most operationally useful place for evaluation is a pre-upgrade dry run. Restore production backups into a staging deployment, roll staging to the new version, trigger your production-traffic-derived experiment against it, and compare the pass rate to the previous run on the old version. In an upcoming version of Solace Agent Mesh this will only require a few lines in a CI job: the upgrade proceeds only when the eval passes. Start the production roll only if the comparison meets your gating criterion, for example every evaluator landing within two percentage points of the prior run.
A Foundation for Shipping with Confidence
Observability tells you how your agents behave once they are live. Evaluation tells you whether they should go live at all. Together they close the loop, giving you the evidence to move an agent from “looks good in a Slack thread” to “measurably ready,” and to keep it there as models, tools, and instructions change underneath you.
So next time someone says, “looks good,” run your evals before you agree!