Zac Sheffer · 2026.9.15 · 8 min read
Kai's deal agent produces recommendations (stage moves, closed-lost verdicts, new contact candidates) for deals in the pipeline. The model's output never reaches the database without passing through deterministic, non-LLM gates. The AI package itself is stateless: it receives context, calls models, returns structured output. Everything that writes or schedules lives outside it.
Five stages: generate, gate, rank, persist, finalize. Here is what the four load-bearing ones do.
The alternative we prototyped: let the model propose changes, apply them inside a transaction, run validation, roll back on failure. Several teams we respect have shipped this pattern.
We walked away for one reason: the rollback window is a liability window. Between the write and the validation check, the data exists in the database. If validation is slow, queries external state, or any other process reads during that window, the wrong data is real. You can shrink the window. You cannot eliminate it. In a multi-tenant system with real-time sync to external CRMs, a bad recommendation could propagate to HubSpot or Salesforce before the rollback completes.
Policy-gating has a cleaner boundary. Model output never touches persistence until it passes every check. No window, no race. The model proposes; deterministic code disposes; only survivors persist. OpenAI's agent safety guidance calls this defense-in-depth: validating outputs and adding confirmation gates before consequential actions.
The cost is rigidity. Adding a new recommendation type means adding gate rules, not just a prompt change. We consider that a feature. If a new recommendation type has no policy rules, it has no guardrails, and shipping unguarded recommendations is worse than not shipping them.
Generation runs on a mid-size model at low temperature with structured output enforced by Zod schemas. The Vercel AI SDK documentation shows this pattern: pass a Zod schema to generateObject so the model produces typed, validated objects instead of freeform text.
One lesson worth highlighting: our schemas use required-plus-nullable instead of optional fields.
With constrained decoding, an optional field means the grammar must represent both the present and absent versions of the object. As OpenAI explains in their structured outputs documentation, the JSON Schema is converted into a context-free grammar that masks invalid tokens during generation. Optional across N fields means 2^N possible shapes. Required-plus-nullable keeps one shape with null slots, and the constrained decoder stays fast. If you are generating structured output from any model, this design choice is worth stealing.
Confidence bands are stated in the prompt: what counts as high, what falls below the include threshold. But stating a rule in a prompt is a request, not a guarantee. That is what the next stage is for.
The gate is a deterministic policy layer. Ordinary TypeScript, no model in the loop. It re-checks everything the prompt already asked for. Rules:
Cross-recommendation rules run here too. A NO_ACTION verdict is stripped when actionable recommendations exist. When a stage move and a closed-lost collide, the higher-confidence one survives.
The catalog runs to more than two dozen named rejection reasons: NON_FORWARD_MOVE, CONFLICTING_MOVE_STAGE, CONTACT_CANDIDATE_IS_ORG_MEMBER, and so on. Every rejection is logged with the rule that fired. When someone asks "why didn't Kai suggest X," the answer is a named policy, not a shrug.
The OWASP Top 10 for LLM Applications calls this category "excessive agency." The mitigation is exactly this: authorization and validation enforced in application code, not delegated to the model.
The gate catches the things you design for: hallucinated contacts, backward stage moves, closed-lost on active deals. The interesting rejections are the ones you discover in production logs.
Email signature contacts. The model recommends adding a contact it found in an email signature. Technically present in the thread. Technically a real person. Contextually wrong: the CFO whose assistant cc'd them on one message is not a deal participant. The source-thread rule catches this by requiring meaningful participation, not just a header mention.
Stage deja vu. The model recommends a stage move with high confidence and correct reasoning, but to a stage the deal already visited and left. It reads the thread chronologically and reaches the same conclusion the team reached three weeks ago, before new information changed the picture. The model does not know the deal already tried that stage. The gate does.
NO_ACTION hedging. The most common contextually-wrong rejection. The model produces a "nothing to do here" verdict alongside a concrete recommendation, hedging its bets. The gate strips the NO_ACTION because it contradicts the actionable recommendation in the same run. This fires on roughly 15% of runs. The model is not confused; it is cautious. The policy layer resolves that caution into a clean signal.
Survivors go to a small, fast model that scores each candidate: should this surface, and at what confidence? Surfacing requires both a positive verdict and confidence at or above 0.35, enforced again at persistence.
The ranker soft-fails by design. If it errors or times out, the pipeline falls back to the generator's own confidence, flags the run as unranked, and continues. A ranking layer that can take down the pipeline is a ranking layer you will eventually disable in a panic. Liveness beats perfection. The flag preserves honesty about which path a recommendation took.
Persistence is a single transaction. Recommendations dedupe by content hash: action type, actor, payload, and title, hashed, with the highest-confidence duplicate winning. Two runs that reach the same conclusion produce one recommendation, not a queue of clones.
One subtle rule: a healthy run that concludes "nothing to do" must not wipe the board. Existing recommendations are superseded only when the new run produced actionable ones. Otherwise a quiet week would erase the useful backlog, and users would learn not to trust the queue.
Stage failures are typed (parse error, steps exhausted, idempotency conflict), so the run record says exactly where and how a pipeline died.
Some gate rules are overengineered. The cross-recommendation conflict resolution (stage move vs. closed-lost, higher confidence wins) has fired on real data exactly twice in four months. The code is correct, tested, and maintained, but the complexity it handles is vanishingly rare. If we rebuilt from scratch, we would start without it and add it when the data justified it.
The source-thread validation for new contacts is still too loose. It checks that the contact appeared in a thread but does not weight recency or substance. A contact mentioned once in a six-month-old thread can still pass if no other rule disqualifies them. We have a ticket. It has been deprioritized twice because the false-positive rate is tolerable. Tolerable is not correct.
The ranker's 0.35 threshold was chosen empirically during development and has not been revisited against production data. It should be. Thresholds set during development tend conservative because development data is small and weird. We have the logs to run that analysis and have not yet.
Three properties fall out of the architecture:
You do not make an agent trustworthy by asking nicely in the prompt. You build the system that assumes it will sometimes be wrong and catches that before it becomes state. How we decide which deals deserve a model call at all (most never do) is the companion post.
Introzy is free to start. No card required.