quality drift
Your prompt still runs. It stopped being right three weeks ago.
Eval will turn a system prompt into a test suite you can edit, then score your real production output against it every day and tell you which check started failing. None of it is built yet.
status
Eval is in development. There is nothing to sign up for yet.
There is no generator on this page. What you can see below is a worked example of what it produces. Continuous monitoring is in private beta and is not open. If you want to know when either of those changes, there is an email address at the bottom, and that is the only thing it will ever be used for.
Two things that do work today are the Cost calculator and the Watch grader. Both are ungated.
what you get
A pass rate, and the examples that failed.
Not a score out of ten and not a dashboard of green ticks. Eval will tell you which specific check dropped, and hand you the production outputs that failed it, so you can read them yourself and decide whether the check is wrong or the agent is.
the structural bit
Eval does not need the SDK.
Cost has to sit in the path of your API calls. Eval will not. It will need access to output your agent has already saved somewhere: a repository, a database, a document store. If your agent writes files, rows or documents, Eval will be able to score them.
To be precise, because this matters: that means output your agent saved. It does not mean a chat transcript inside somebody else's hosted app.
example
A prompt in, a suite out.
This is a worked example, not a live generator. It shows the shape of what Eval would produce, so you can judge whether it would be useful before anything exists to try.
the system prompt
You are a support assistant for an Irish electrical supplies wholesaler. Answer only from the product catalogue and the delivery policy provided in context. If the answer is not in context, say you do not know and offer to pass the question to a person. Never quote a price that is not in the catalogue. Never promise a delivery date. Prices are ex-VAT; say so whenever you give one. Reply in the language the customer used. Keep replies under 120 words.
the generated suite
- 01 Answers only from context
Every factual claim in the reply appears in the supplied catalogue or delivery policy.
from: Answer only from the product catalogue and the delivery policy
- 02 Abstains when it does not know
When the answer is absent from context, the reply says so and offers a handover.
from: If the answer is not in context, say you do not know
- 03 Never invents a price
Any figure presented as a price matches a catalogue entry exactly.
from: Never quote a price that is not in the catalogue
- 04 Never promises a delivery date
The reply contains no committed date or day for delivery.
from: Never promise a delivery date
- 05 States ex-VAT on every price
Any reply containing a price also states that prices are ex-VAT.
from: Prices are ex-VAT; say so whenever you give one
- 06 Replies in the customer's language
The reply is in the same language as the incoming message.
from: Reply in the language the customer used
- 07 Stays under 120 words
Word count of the reply is 120 or fewer.
from: Keep replies under 120 words
Every check will be editable. Eval will derive a first pass from your prompt; you decide what actually matters.
the daily report
Example data. Not a real run.
Yesterday, 1,284 production replies. Overall pass rate 94%.
| Check | Pass rate | Change |
|---|---|---|
| Answers only from context | 99% | no change |
| Abstains when it does not know | 97% | +1 pts |
| Never invents a price | 100% | no change |
| Never promises a delivery date | 98% | no change |
| States ex-VAT on every priceregressed | 71% | -26 pts |
| Replies in the customer's language | 99% | no change |
| Stays under 120 words | 96% | -1 pts |
States ex-VAT on every price dropped 26 points. Here are two replies that failed it.
asked: What's the price on 100m of 2.5mm twin and earth?
That's €68.40 for a 100m drum. I can add it to your next order if you like.
Price given with no ex-VAT statement.
asked: How much for a 10-way consumer unit?
The 10-way board is €54.90. Want me to check stock in Cork?
Price given with no ex-VAT statement.
This is the whole point. Not a score, but the specific check that moved and the production output that failed it, so you can decide whether the check is wrong or the agent is.
stay in touch
Tell me when the generator is live.
One email when it opens. Nothing else, ever. No newsletter, no drip sequence, and your address does not go anywhere else.
Email hello@botzone.aiEval is built by Botzone Limited, Dublin. Company number 814822.