OCOPD Build an AI Trust Harness: Model-Agnostic Evals, Tracing, and Release Gates for LLM Apps
Introduction: What Is This and Why Use It?
A few years ago, the model choice felt like the architecture.
If one frontier model was dramatically better at reasoning, coding, or instruction following, teams naturally built around that provider. Prompt formats, retry behavior, safety assumptions, latency budgets, and even product features often depended on a single model's quirks.
That world is fading. Public model comparison sites such as LMArena and Artificial Analysis increasingly show the top commercial models clustering near each other on general capability. The exact leader changes, but the practical lesson is bigger: for many production apps, the gap between strong models is now smaller than the gap between a well-instrumented AI system and an untrusted one.
This is where the AI trust harness comes in.
An AI trust harness is the layer around your model calls that answers the questions developers and business owners actually care about:
- Did the model follow the output contract?
- Did quality regress after a prompt, model, or provider change?
- Can we compare multiple models without rewriting the app?
- Do we have traces when a customer reports a bad answer?
- Can we block a release if the AI behavior gets worse?
- Can we prove that a specific model is safe enough for a specific workflow?
This tutorial shows how to build that harness using currently available tools:
- LiteLLM for provider-neutral model calls
- promptfoo for repeatable LLM evaluations
- pytest for CI release gates
- OpenTelemetry for traceable AI calls
- Plain JSON contracts for predictable application integration
This is not another generic model comparison benchmark. The goal is more useful: build a small but production-shaped harness that lets your team change models, prompts, and providers while keeping behavior measurable.
By the end, you will have a working local setup that can test multiple LLMs against the same task, enforce schema expectations, score output quality, and fail CI when trust drops below your threshold.
Prerequisites
You need:
- Python 3.10 or newer
- Node.js 18 or newer
- An API key for at least one supported LLM provider
- Basic comfort with command-line tools
- A small AI use case to test
This guide uses OpenAI in the examples because it is widely available through LiteLLM, but the harness pattern also works with providers supported by LiteLLM, including Anthropic, Google, Mistral, Cohere, Azure OpenAI, and local OpenAI-compatible endpoints.
Create a new project:
mkdir ai-trust-harness
cd ai-trust-harness
python -m venv .venv
source .venv/bin/activateInstall Python dependencies:
pip install litellm pydantic pytest python-dotenv opentelemetry-api opentelemetry-sdkInstall promptfoo:
npm install -g promptfooCreate a .env file:
OPENAI_API_KEY=your_api_key_here
PRIMARY_MODEL=openai/gpt-4.1-mini
CANDIDATE_MODEL=openai/gpt-4.1You can replace these model names with any LiteLLM-supported model available in your account. The important point is that the application code should not care which provider sits behind the model string.
Step-by-Step Guide
Step 1: Define the Workflow You Actually Need to Trust
Do not start with a broad benchmark. Start with one task your product depends on.
For this tutorial, we will build a harness for a common enterprise workflow: turning a messy customer support message into a structured triage object.
The app should return:
category: billing, technical, account, security, or otherpriority: low, medium, high, or urgentsummary: one sentenceneeds_human: booleanreasoning: short explanation for internal review
Create schemas.py:
from typing import Literal
from pydantic import BaseModel, Field
class TriageResult(BaseModel):
category: Literal["billing", "technical", "account", "security", "other"]
priority: Literal["low", "medium", "high", "urgent"]
summary: str = Field(min_length=10, max_length=220)
needs_human: bool
reasoning: str = Field(min_length=10, max_length=500)This schema is your first trust boundary. Model convergence does not remove the need for contracts. In fact, when models become interchangeable, contracts become more important because they let you swap models without rewriting downstream code.
Step 2: Build a Provider-Neutral Model Client
Create llm_client.py:
import json
import os
from dotenv import load_dotenv
from litellm import completion
from opentelemetry import trace
from schemas import TriageResult
load_dotenv()
tracer = trace.get_tracer(__name__)
SYSTEM_PROMPT = """
You are a support operations classifier.
Return only valid JSON matching this schema:
{
"category": "billing | technical | account | security | other",
"priority": "low | medium | high | urgent",
"summary": "one sentence summary",
"needs_human": true,
"reasoning": "brief internal explanation"
}
Rules:
- Mark security incidents as needs_human=true.
- Mark legal threats, account lockouts, and payment failures as at least high priority.
- Do not include markdown.
- Do not include extra keys.
"""
def triage_ticket(message: str, model: str | None = None) -> TriageResult:
selected_model = model or os.getenv("PRIMARY_MODEL", "openai/gpt-4.1-mini")
with tracer.start_as_current_span("llm.triage_ticket") as span:
span.set_attribute("llm.model", selected_model)
span.set_attribute("app.workflow", "support_triage")
response = completion(
model=selected_model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": message},
],
temperature=0,
response_format={"type": "json_object"},
)
content = response["choices"][0]["message"]["content"]
span.set_attribute("llm.response_length", len(content))
parsed = json.loads(content)
result = TriageResult.model_validate(parsed)
span.set_attribute("app.category", result.category)
span.set_attribute("app.priority", result.priority)
span.set_attribute("app.needs_human", result.needs_human)
return resultA few details matter here.
First, the model name is configuration, not application logic. That is the heart of a model-agnostic AI stack. Second, the function returns a Pydantic object, not raw prose. Third, tracing is built into the model call from day one.
Step 3: Add Local Trace Output
Create otel_setup.py:
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
def configure_tracing() -> None:
provider = TracerProvider(
resource=Resource.create({"service.name": "ai-trust-harness"})
)
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)Create run_triage.py:
from otel_setup import configure_tracing
from llm_client import triage_ticket
configure_tracing()
message = "I was charged twice this month and now my account says payment failed. I need this fixed today."
result = triage_ticket(message)
print(result.model_dump_json(indent=2))Run it:
python run_triage.pyYou should see structured JSON plus a console trace. In production, you would export traces to a backend such as Jaeger, Grafana Tempo, Honeycomb, Datadog, or another OpenTelemetry-compatible system. The local console exporter is enough to prove the wiring.
Step 4: Create a Golden Test Set
The most useful eval data usually comes from your own edge cases, not public benchmark questions.
Create eval_cases.jsonl:
{"message":"I cannot log in after enabling two-factor authentication and I no longer have access to my old phone.","expected_category":"account","expected_min_priority":"high","expected_needs_human":true}
{"message":"Your product deleted our audit logs. This is a security incident and our compliance team needs answers.","expected_category":"security","expected_min_priority":"urgent","expected_needs_human":true}
{"message":"How do I change the email address on my invoice?","expected_category":"billing","expected_min_priority":"low","expected_needs_human":false}
{"message":"The API returns 500 errors for every request from our production integration.","expected_category":"technical","expected_min_priority":"high","expected_needs_human":true}
{"message":"Thanks, the issue is fixed now.","expected_category":"other","expected_min_priority":"low","expected_needs_human":false}Keep this file small at first. Ten sharp examples beat 500 vague ones. Add cases whenever you discover a real failure mode.
Step 5: Write Deterministic Contract Tests with pytest
Create test_triage_contract.py:
import json
import os
import pytest
from llm_client import triage_ticket
PRIORITY_RANK = {
"low": 1,
"medium": 2,
"high": 3,
"urgent": 4,
}
def load_cases():
with open("eval_cases.jsonl", "r", encoding="utf-8") as file:
for line in file:
yield json.loads(line)
@pytest.mark.parametrize("case", list(load_cases()))
def test_triage_contract(case):
model = os.getenv("PRIMARY_MODEL")
result = triage_ticket(case["message"], model=model)
assert result.category == case["expected_category"]
assert result.needs_human is case["expected_needs_human"]
assert PRIORITY_RANK[result.priority] >= PRIORITY_RANK[case["expected_min_priority"]]Run it:
pytest -qThis is intentionally strict. For production, you may not want every test to require exact labels, especially for ambiguous messages. But for high-risk workflows, deterministic gates are valuable. They catch obvious regressions quickly.
Step 6: Add promptfoo for Cross-Model Evaluation
pytest is excellent for contract tests. promptfoo is better for comparing prompts and models across a matrix of cases.
Create promptfooconfig.yaml:
description: Support triage trust harness
prompts:
- file://triage_prompt.txt
providers:
- id: openai:gpt-4.1-mini
- id: openai:gpt-4.1
tests:
- vars:
ticket: "I was charged twice and support has ignored me for a week. Cancel my account if this is not fixed today."
assert:
- type: contains-json
- type: javascript
value: |
const data = JSON.parse(output);
return data.category === 'billing' && ['high', 'urgent'].includes(data.priority);
- vars:
ticket: "We believe someone accessed our admin account without permission."
assert:
- type: contains-json
- type: javascript
value: |
const data = JSON.parse(output);
return data.category === 'security' && data.needs_human === true;
- vars:
ticket: "Where can I download last month's receipt?"
assert:
- type: contains-json
- type: javascript
value: |
const data = JSON.parse(output);
return data.category === 'billing' && data.priority === 'low';Create triage_prompt.txt:
You are a support operations classifier.
Return only valid JSON with these keys:
- category: billing, technical, account, security, or other
- priority: low, medium, high, or urgent
- summary: one sentence
- needs_human: boolean
- reasoning: short internal explanation
Rules:
- Mark security incidents as needs_human=true.
- Mark legal threats, account lockouts, and payment failures as at least high priority.
- Do not include markdown.
- Do not include extra keys.
Ticket:
{{ticket}}Run the eval:
promptfoo evalThen inspect the results:
promptfoo viewThis gives you a simple but powerful comparison table. When models are close on general-purpose leaderboards, your own workflow-specific eval becomes the deciding evidence.
Step 7: Turn Evals into a Release Gate
A trust harness only matters if it can stop bad changes.
Add this script as check_promptfoo.py:
import json
import subprocess
import sys
MIN_PASS_RATE = 0.90
completed = subprocess.run(
["promptfoo", "eval", "--output", "results.json"],
text=True,
capture_output=True,
)
print(completed.stdout)
if completed.returncode != 0:
print(completed.stderr, file=sys.stderr)
sys.exit(completed.returncode)
with open("results.json", "r", encoding="utf-8") as file:
results = json.load(file)
stats = results.get("stats", {})
successes = stats.get("successes", 0)
failures = stats.get("failures", 0)
total = successes + failures
pass_rate = successes / total if total else 0
print(f"Pass rate: {pass_rate:.2%}")
if pass_rate < MIN_PASS_RATE:
print(f"Eval gate failed: expected at least {MIN_PASS_RATE:.0%}")
sys.exit(1)Run:
python check_promptfoo.pyIn CI, use both gates:
pytest -q
python check_promptfoo.pyThat is the practical difference between experimentation and engineering. You can still test new models quickly, but model swaps now pass through the same quality boundary as normal code changes.
Step 8: Compare a Candidate Model Without Rewriting Code
Now test the core claim: if models have converged, your harness should make switching boring.
Run the same pytest suite with a candidate model:
PRIMARY_MODEL=openai/gpt-4.1 pytest -qOr compare in your app directly:
from llm_client import triage_ticket
message = "Our SSO login is down for all employees and we cannot access billing or admin settings."
for model in ["openai/gpt-4.1-mini", "openai/gpt-4.1"]:
result = triage_ticket(message, model=model)
print(model, result.model_dump())If both models pass, choose based on latency, cost, availability, data policy, or regional requirements. That is exactly the point. When raw model quality is close enough, operational trust becomes the differentiator.
Step 9: Add a Human Review Queue for Low-Confidence Zones
Not every trust problem should be solved with another prompt. Some cases should leave the automation path.
A simple policy function can catch risky combinations:
from schemas import TriageResult
def requires_review(result: TriageResult) -> bool:
if result.category == "security":
return True
if result.priority in {"high", "urgent"} and result.needs_human:
return True
if len(result.reasoning.strip()) < 20:
return True
return FalseUse it after triage:
result = triage_ticket(message)
if requires_review(result):
print("Send to human review queue")
else:
print("Allow automated workflow")This is a basic example, but the principle scales. The harness should not pretend the model is always trustworthy. It should decide where automation is appropriate and where review is cheaper than risk.
Common Pitfalls
Treating Leaderboards as Production Evals
Leaderboards are useful signals, not deployment evidence. A model that performs well on broad public benchmarks may still fail your invoice parser, compliance workflow, medical intake assistant, or code review bot. Always build a task-specific eval set.
Testing Only Happy Paths
Most teams over-test clean examples. Real users write vague, emotional, contradictory, incomplete, and adversarial messages. Your eval cases should include boring normal inputs, but they must also include the messy cases that create support tickets and incident reviews.
Accepting Raw Text When the App Needs Structure
If your downstream code expects fields, ask the model for fields and validate them. Raw text is fine for chat. It is fragile for workflows. Use JSON schemas, Pydantic, Zod, or your language's equivalent validation layer.
Confusing Retry Logic with Trust
Retries can fix transient API errors or occasional malformed output. They do not prove the answer is correct. If an AI system needs trust, it needs evals, traces, policy checks, and review boundaries.
Changing Prompts Without Versioning
A prompt is production logic. Store prompts in source control. Review changes. Run evals before merging. Tie prompt versions to traces so you can debug regressions later.
Measuring Only Accuracy
Accuracy matters, but production trust also includes latency, cost, schema validity, refusal behavior, escalation rate, and incident severity. A slightly less accurate model may be better for a workflow if it is faster, cheaper, more available, and easier to monitor.
FAQ
Is This the Same as Prompt Routing?
No. Prompt routing decides which model or prompt to use for a request. A trust harness measures and controls whether any chosen model is acceptable for the workflow. You can add routing later, but the harness should exist first.
How Many Eval Cases Do I Need?
Start with 20 to 50 high-quality cases for one workflow. Add more when you find real failures. For critical systems, separate your evals into smoke tests, regression tests, adversarial tests, and larger offline evaluation sets.
Should I Use LLM-as-Judge?
Use it carefully. LLM judges are helpful for subjective dimensions such as clarity or completeness, but deterministic checks are better for contracts, labels, required fields, and safety policies. For this tutorial, the strongest checks are schema validation and explicit assertions.
Can This Work with Local Models?
Yes, if your local server exposes an OpenAI-compatible API or is supported by LiteLLM. The same harness can test local and hosted models side by side. Just remember that local deployment adds separate concerns around capacity, latency, monitoring, and model updates.
What Should I Trace?
At minimum, trace the workflow name, model name, prompt version, latency, token usage if available, output category, policy decision, and error type. Avoid logging sensitive user content unless your privacy and retention policies explicitly allow it.
How Do I Choose Between Models That Both Pass?
Use operational criteria: cost per successful task, p95 latency, provider reliability, data handling terms, regional availability, rate limits, and support for structured outputs. When quality is close, these factors often matter more than a one-point benchmark difference.
Final Thoughts
The practical AI stack is moving from model worship to harness engineering.
As strong models converge, the winning teams will not be the ones that endlessly chase the top slot on a public leaderboard. They will be the teams that can evaluate, observe, constrain, and safely replace models as the market changes.
Build the harness once. Then every future model comparison becomes less emotional and more empirical: run the evals, inspect the traces, check the policy outcomes, and ship only when the system earns trust on your workflow.