Introducing venpor & Venny
Venny · Eval Engine
Venny is tested before it touches your programme.
The Venny Evaluation Engine runs Venny through realistic agent scenarios, scores results against defined instructions and desired outcomes, and surfaces regressions, before deployment and when live in production.
The problem it solves
AI behaviour changes. The Eval Engine catches it before you do.
Frontier models are updated every few weeks. Each update can subtly change how Venny responds to edge cases. Without a systematic way to catch regressions, quality erodes quietly. A programme running thousands of interactions needs a way to validate behaviour continuously, not anecdotally.
What happens when a model is updated?
Behaviour can shift on edge cases without warning. A subtle change in how a model interprets an instruction can cause a previously correct response to go off-track.
✓
The Eval Engine runs the full scenario set against any new model before production. Regressions are caught in evaluation, not in a live programme.
How do you know Venny handled an edge case correctly?
In a high-volume programme, manually reviewing individual interactions is impractical. Quality issues surface after the fact, usually via an escalation.
✓
The Eval Engine evaluates Venny in real time in production, scoring individual interactions against defined criteria and flagging those that fall below threshold.
Which model configuration performs best for a given scenario?
The same prompt can yield different results across providers, models, and temperature settings. Without a way to compare, the choice of configuration is guesswork.
✓
Named model configurations are run side-by-side across the same scenario set. A config ranking shows which setup handles each scenario most reliably.
Scenarios
30+ realistic agent scenarios. Each one a test Venny has to pass.
Each scenario combines a specific user input, an expected output, and its own set of judges. Scenarios span the full programme workflow — from sourcing and hiring through compliance, payments, and supplier management. Unlimited new scenarios can be defined per programme.
Judges
Three types of judgment. Applied in combination.
Each scenario has its own set of AI evaluators. The three judge types can be used individually or combined, giving a complete picture of whether Venny's output was correct, how close it came, and exactly where it went wrong.
Judge type 01
Pass / Fail
A binary check on whether the desired outcome was met. Did Venny create the position, send the message, surface the right candidate, flag the risk, generate the briefing?
Example
Did Venny offer to search the talent pool? Did it avoid fabricating candidate names before searching? Pass or fail.
Judge type 02
0-100 Score
A graded judgment of how closely Venny's output follows the instructions and matches the desired result, comparing actual output against a reference and returning a quality score.
Example
Comparing job description generated by Venny against the reference: Haiku 4.5 scores 80%, Opus scores 95%.
Judge type 03
Recommendations
A free-text analysis explaining what Venny did, where it went off-track, and what to adjust. Aimed at both users and other agents, so improvements compound run over run instead of relying on manual transcript review.
Example
Venny correctly identified the engagement risk but did not suggest a specific extension action. Recommend adding this to the instruction set.
Model Configs
Named configurations. Compared side-by-side.
A model config is a named setup specifying provider, model, temperature, and max token settings. Configs are run against the same scenario set simultaneously, and results ranked so the best-performing setup per scenario type can be selected.
This makes model upgrades and A/B comparisons structural rather than ad hoc. Every new model version is validated against the existing scenario set before going to production.
Runs
Every run is fully transparent.
Evaluation runs execute across multiple scenarios, model configurations, and repetitions in parallel. Each run produces individual task results showing the full input, actual output including tool calls and their results, and every judge's verdict.
completed
Tasks: 6 · Success: 100% · Pass rate: 58% · Avg latency: 30.1s · Best: Haiku 4.5 (temp 0.5)
Search talent pool for position fit
agent
3 judges
Haiku 4.5 (temp 0.5) · Rep 2 · anthropic/claude-haiku-4-5
2/2 pass 24.1s
Haiku 4.5 (temp 0.1) · Rep 1 · anthropic/claude-haiku-4-5
1/2 pass 27.8s
Haiku 4.5 (temp 0.5) · Rep 1 · anthropic/claude-haiku-4-5
1/2 pass 28.5s
Analytics
Config rankings
Ranked by pass rate and latency per scenario
Analytics
Scenario x Config matrix
Full cross-comparison across every run combination
Analytics
Per-judge breakdowns
Each judge scored separately across models and repetitions
Analytics
Full agent transcripts
Complete input, output, and tool call records per task
What this unlocks
The foundation for a continuously improving programme.
Three capabilities that become possible once systematic evaluation is in place.
01
Custom scenario sets per programme
As programmes are onboarded, the specific policies, escalation paths, and edge cases that matter for each customer can be encoded as scenarios. Venny is then continuously evaluated against those exact criteria in production.
02
Earlier detection of drift and regressions
Continuous evaluation in production catches quality issues at the level of an individual interaction, not after a manager or supplier escalates. Lower operational risk and a clear audit trail for anything that did slip through.
03
Faster, safer model upgrades
Frontier models land every few weeks. The Eval Engine validates each new model against the full scenario set before production rollout. Programmes gain the upside of new capabilities without the unknowns of an unvalidated upgrade.
Compare
Fragmentation doesn't scale. Orchestration does.
Manual Process
Copy/pasting metrics into reports
Chasing team members for updates
Manually tagging leads in CRM
Checking 4 tools every morning
Missed follow-ups or overdue tasks
With venpor
Agent compiles & sends daily updates to Slack
Agent collects and summarizes inputs
AI auto-tags based on behavior or inputs
Unified daily brief across all platforms
Agent sets and tracks follow-up actions
FAQ