Harness is the 100x agent framework. A client agent is one agent.yaml and a markdown knowledge pack. No new code in the framework per client — and a test fails the build if a client's name appears in it.
The gap
The interesting engineering in a customer-facing agent is not getting a good reply. It is what happens on the bad path — and there are more bad paths than good ones.
| What a demo skips | What it costs in production |
|---|---|
| Nothing checks the outbound message | The model offers a discount nobody authorised, in writing, to a customer |
| The channel retries a delivery | The same person is answered twice |
| The model declines, or the API times out | Silence. Nobody is told. The conversation dies |
| No record of what was sent or why | Nothing to audit, nothing to replay, nothing to improve against |
| Volatile data at the front of the prompt | The cache never hits and every turn pays full input price |
| Per-client forks of the loop | Client ten costs what client one cost |
Architecture
A layer talks only to the layers next to it, and only through a Pydantic model. The contract chain for one turn is fixed:
InboundEvent → MemorySnapshot → AssembledContext → ModelResult
→ ToolOutcome* → Verdict* → OutboundMessage → TurnRecordBuilds the model input: system template, the client's versioned knowledge pack, policies, and memory — a rolling window plus durable per-contact facts. Deterministic: identical inputs produce identical bytes, and a test asserts it.
Capability adapters behind one interface — send, calendar, CRM, hand off to a human, speak. A client's config lists the tools it gets by name; an unregistered name fails at load. Tools never hold a channel sender.
Drives the turn end to end. Bounded retries with backoff, per-stage timeouts, a total turn budget, and an idempotency claim on the inbound message id. Owns the only path to a customer.
Structure, policy, PII, tone. Deterministic checks first and free; the model-backed tone check runs only if they pass, and only if the client turned it on.
One structured record per turn, hash-chained per conversation. Cost from real usage, drift metrics, and a replay-eval harness built to rerun recorded conversations against a changed prompt or model.
Runtime
The order of operations is the product. Read run_turn top to bottom and this is what you get.
tool_use to adapters, return every result in a single user message, loop until it stops asking — bounded by a per-client iteration cap, past which the turn escalates rather than spinning. Handle pause_turn by resuming; treat refusal as a hand-off, never as text to rewrite.OutboundMessage. A second failure notifies a human and sends nothing.The gate
The constraint is structural rather than procedural: OutboundMessage raises in its own validator if it is handed no verdicts, or any failing verdict. There is nothing to remember to call.
def model_post_init(self, _, /) -> None:
if not self.verdicts:
raise ValueError("OutboundMessage requires guardrail verdicts")
failed = [v.validator for v in self.verdicts if not v.passed]
if failed:
raise ValueError(f"OutboundMessage blocked by failing validators: {failed}")A tool that wants to put its own words in front of a person — an extra message, a spoken line — does not get a sender. It asks the loop's gate, which runs the same pipeline and raises a tool error the model can see and correct. Adding a second route to the customer means changing the loop, and an architecture test fails if clear_for_delivery is called from any module but the loop.
One honest caveat about the strength of that guarantee. Written replies are enforced by the type: they cannot exist without passing verdicts. A spoken line is enforced by the pipeline: the voice tool carries its own transport, so it calls the gate to be screened and then hands the text to the voice backend — the same checks, on the same code path, but no OutboundMessage is constructed. Two mechanisms, not one, and only one of them is a compiler-level fact.
| Check | Cost | Catches |
|---|---|---|
| Structure | free | Empty, over-length, leaked internal markup, a tool call written as prose |
| Policy | free | Banned topics from a shared library plus client-specific phrases — discounts, guarantees, legal and medical claims, competitor mentions |
| PII | free | Phone numbers other than the customer's own, any email address, IBANs, Luhn-valid card numbers. Order references, invoice numbers and opening hours are deliberately not read as phone numbers |
| Tone | 1 model call | Brand voice. Off by default, and skipped entirely when a cheaper check already failed |
Rules first is not a cost decision, it is a determinism decision. A regex returns the same verdict every time, which is what makes a logged turn replayable and an eval meaningful. Tone is the one judgement rules genuinely cannot make, so it is the one thing a model is asked.
Prompt assembly
The API caches on an exact prefix match and renders tools → system → messages. So everything stable renders first and carries the breakpoint; everything volatile renders strictly after it. One cache entry then serves every conversation that client has.
# cached prefix — identical for every turn, every contact
tools the client's tool schemas, sorted by name
system[0] system template + persona + policies
system[1] knowledge pack ← cache_control breakpoint
# after the breakpoint — changes every turn, invalidates nothing above
system[2] today's date · first contact? · long-term facts
messages conversation window
+ the inbound message ← second breakpoint (see limits)The failure mode is silent: put a timestamp in the prefix and nothing breaks, the bill just goes up. So a test renders two turns on different days, for different contacts, with different memory, and asserts the prefix bytes are identical — and separately asserts no date, contact id or fact ever appears in it.
What the test cannot assert is that the entry is still there. The breakpoint is a five-minute ephemeral cache, so "one entry serves every conversation" holds for a client with steady traffic and quietly stops holding for a client whose conversations are twenty minutes apart — each cold turn then pays a write at 1.25× input instead of a read at 0.1×. Whether caching is a saving or a small tax is a function of a client's arrival rate. It is read off cache_read_input_tokens in production, not asserted in a test.
Observability
Each record carries the previous record's hash, so editing one after the fact breaks every link from that point on. It is the EU AI Act Article 12 answer and the eval corpus at the same time — a record holds everything needed to reconstruct the turn.
turn_id · client_id · conversation_id · contact_id · message_id
started_at · finished_at · inbound_text · outbound_text
prompt_sha256 · knowledge_sha256 · config_sha256 · model
tool_calls[] · verdicts[] · regenerated · outcome
escalation_reason · error · stages[] · usage · cost_usd
retention_days · prev_hash · record_hashBecause the verdicts are deterministic, the same records drive replay evals: rerun real conversations against a changed prompt or a new model and diff the outcomes before a customer sees them. Drift alerts on four numbers per client — guardrail-failure rate, escalation rate, p95 latency, and cache hit rate. A climbing escalation rate almost always means a hole in the knowledge pack, which is a markdown edit rather than a code change — though today it still travels as a container rebuild, because packs are baked into the image and read once at startup.
The whole point
client_id: acme # must match the directory name
agent_name: Sam
channel: whatsapp
model: claude-opus-5 # unknown model = refuses to load
language: [en, ar]
persona:
voice: "warm, concise, professional"
disclosure: true # false = refuses to load
tools:
- whatsapp_send
- crm_upsert
- handoff_to_human # missing = refuses to load
guardrails:
banned_topics: [pricing_discounts, guarantees]
max_message_length: 600
escalation:
human_contact: "+9715…" # blank = refuses to load
confidence_floor: 0.7
observability:
retention_days: 365 # no default, on purposeCompliance is enforced by the loader, not by a checklist. A config that would produce an agent with no human fallback, or one that hides being an AI, does not start — it raises at load with the reason. The same is true of an unknown tool or an unknown policy key: you cannot silently misspell a guardrail into non-existence.
created clients/acme
agent: Sam on whatsapp
tools: whatsapp_send, crm_upsert, handoff_to_human
escalates to: +971500000000
Next:
1. Replace the knowledge pack in clients/acme/knowledge with the real thing.
2. Set the real escalation contact in clients/acme/agent.yaml.
3. Tune persona.voice and guardrails.banned_topics.
4. harness validate clients/acme
# the scaffolder loads what it generated and deletes it if invalid,
# so a broken client never reaches the repo. Note step 2: that default
# number loads happily.The next five questions
Architecture diagrams answer the easy half. These are the questions that decide whether a thing survives contact with a customer.
| Question | How it is handled |
|---|---|
| A tool half-succeeds — the booking lands, the call times out | The timeout comes back to the model as a failed tool result, so it reacts rather than crashes. Idempotency for the side effect itself belongs in the tool contract, not the loop — a booking tool takes an idempotency key so a retry resolves to the same booking. The loop guarantees the turn is not repeated; the tool guarantees its own effect is not |
| How is one client's state kept out of another's | By key prefix. Every document is client_id:conversation_id, every claim client_id:message_id, and the state store is reached only through a Protocol that takes the client id on every call — so there is no path that reads another client's conversation by accident. Where a deployment needs a hard boundary rather than a key, the same Protocol takes a per-client database or per-client credentials without touching the loop |
| What happens when the knowledge pack outgrows the context window | The knowledge pack is the cached prefix, so its size is a cost question before it is a capacity one: it is written once and read at a tenth of the input price by every conversation that client has. The model client exposes count_tokens, and the place to spend it is at config load — a pack over budget should fail the way any other bad config fails, loudly, before the agent serves anyone |
| How do you roll back a prompt change | By reverting the commit and redeploying — the same as any other change, because the system template is code and the knowledge packs ship in the image. Every turn record carries prompt_sha256, knowledge_sha256 and config_sha256, so you can always tell exactly which version answered a given conversation. Moving between versions without a deploy would mean lifting the pack out of the image and into the store the records already point at |
| What is the concurrency model | Async on two worker processes, scaling to zero. Turns for different conversations run concurrently; the atomic claim is what stops a duplicate delivery being answered twice. There is no per-client concurrency cap and no queue, so a burst is absorbed by the model API's rate limits and the loop's bounded retries rather than by anything the framework does |
Stack
| Concern | Choice | Why |
|---|---|---|
| Runtime | Python 3.12, FastAPI, Pydantic v2, async | Typed at every boundary; mypy strict across the framework |
| Reasoning | Claude, adaptive thinking, effort per client | One adapter file knows the API; swapping the model is a config line |
| Tool loop | Hand-written | The stock runner exposes no per-stage timeout, no idempotency claim and no gate between model output and send. One beta feature is used deliberately: the server-side refusal fallback, so a false-positive safety decline on ordinary business text is retried rather than escalated to a person |
| State | Protocol → in-memory or Firestore | The whole suite runs offline; production swaps by environment |
| Deploy | Cloud Run, scale to zero | Stateless service; a bad config crashes the worker at startup rather than serving half a fleet |
| Secrets | Secret Manager only | The config schema has nowhere to put a credential, deliberately |
And if you are choosing someone to build one for you: the questions worth asking are what happens when the model says the wrong thing, who finds out, and what you can prove about it a year later. This page is my answer. Happy to go deeper on any layer.