Introducing venpor & Venny
Venny · Eval Engine
Venny is tested continuously.
The Venny Eval Engine scores hundreds of agent scenarios every 24 hours—before deployment and continuously in production—measuring response quality, regressions, token usage, and cost.
The problem it solves
AI behaviour changes. The Eval Engine detects it automatically.
Frontier models are updated every few weeks. Each update can change how Venny responds. Without systematic evaluation, quality can erode quietly. A programme running thousands of interactions needs continuous validation of behaviour, performance, and cost.
What happens when a model is updated?
Behaviour can shift without warning. A subtle change in how a model interprets an instruction can cause a previously correct response to go off-track.
✓
The Eval Engine runs the full scenario set against any new model before production. Regressions are caught in evaluation, not in a live programme.
How do you know Venny handled every interaction correctly?
In a high-volume programme, manually reviewing individual interactions is impractical. Quality issues surface after the fact, usually via an escalation.
✓
The Eval Engine evaluates Venny in real time in production, scoring individual interactions against defined criteria and flagging those that fall below threshold.
Which model performs best—and at what cost?
The same task can produce different quality, token usage, and cost across providers and models. Without side-by-side evaluation, configuration choices become guesswork.
✓
Named model configurations are run side by side against the same scenarios. The resulting ranking shows which setup handles each scenario most reliably and efficiently.
Scenarios
Hundreds of agent scenarios tested every 24 hours.
Each scenario combines a specific user input, an expected output, and its own set of judges. Scenarios span the full programme workflow—from sourcing and hiring through compliance, payments, and supplier management. New scenarios can be defined for each programme.
Judges
Three types of judgment. Applied in combination.
Each scenario has its own set of AI evaluators. The three judge types can be used individually or combined, giving a complete picture of whether Venny's output was correct, how close it came, and exactly where it went wrong.
Judge type 01
Pass / Fail
A binary check on whether the desired outcome was met. Did Venny create the position, send the message, surface the right candidate, flag the risk, generate the briefing?
Example
Did Venny offer to search the talent pool? Did it avoid fabricating candidate names before searching? Pass or fail.
Judge type 02
0-100 Score
A graded judgment of how closely Venny's output follows the instructions and matches the desired result, comparing actual output against a reference and returning a quality score.
Example
Comparing job description generated by Venny against the reference: Haiku 4.5 scores 80%, Opus scores 95%.
Judge type 03
Recommendations
A free-text analysis explaining what Venny did, where it went off-track, and what to adjust. Aimed at both users and other agents, so improvements compound run over run instead of relying on manual transcript review.
Example
Venny correctly identified the engagement risk but did not suggest a specific extension action. Recommend adding this to the instruction set.
Model Configs
Named configurations. Compared side-by-side.
A model config is a named setup specifying provider, model, temperature, and max token settings. Configs are run against the same scenario set simultaneously, and results ranked so the best-performing setup per scenario type can be selected.
This makes model upgrades and A/B comparisons structural rather than ad hoc. Every new model version is validated against the existing scenario set before going to production.
Runs
Every run is fully transparent.
Evaluation runs execute across multiple scenarios, model configurations, and repetitions in parallel. Each run produces individual task results showing the full input, actual output including tool calls and their results, and every judge's verdict.
completed
Tasks: 6 · Success: 100% · Pass rate: 58% · Avg latency: 30.1s · Best: Haiku 4.5 (temp 0.5)
Search talent pool for position fit
agent
3 judges
Haiku 4.5 (temp 0.5) · Rep 2 · anthropic/claude-haiku-4-5
2/2 pass 24.1s
Haiku 4.5 (temp 0.1) · Rep 1 · anthropic/claude-haiku-4-5
1/2 pass 27.8s
Haiku 4.5 (temp 0.5) · Rep 1 · anthropic/claude-haiku-4-5
1/2 pass 28.5s
Analytics
Config rankings
Ranked by pass rate and latency per scenario
Analytics
Scenario x Config matrix
Full cross-comparison across every run combination
Analytics
Per-judge breakdowns
Each judge scored separately across models and repetitions
Analytics
Full agent transcripts
Complete input, output, and tool call records per task
What this unlocks
The foundation for a continuously improving programme.
Three capabilities that become possible once systematic evaluation is in place.
01
Custom scenario sets per programme
As programmes are onboarded, the specific policies, escalation paths, and operating requirements that matter for each customer can be encoded as scenarios. Venny is then continuously evaluated against those exact criteria in production.
02
Earlier detection of drift and regressions
Continuous evaluation in production catches quality issues at the level of an individual interaction, not after a manager or supplier escalates. Lower operational risk and a clear audit trail for anything that did slip through.
03
Faster, safer model upgrades
Frontier models land every few weeks. The Eval Engine validates each new model against the full scenario set before production rollout. Programmes gain the upside of new capabilities without the unknowns of an unvalidated upgrade.
Compare
Fragmentation doesn't scale. Orchestration does.
Without continuous evaluation
Model changes reach production untested
Quality issues surface through escalations
Individual interactions reviewed manually
Model selection based on assumptions
Token usage and cost tracked separately
With the Venny Eval Engine
Hundreds of scenarios tested every 24 hours
Regressions detected automatically
Every interaction scored against defined criteria
Models compared side by side
Quality, token usage, and cost evaluated together
FAQ