Zac Sheffer · 2026.9.16 · 7 min read
Kai's nightly deal review runs against every open deal in an org. In a validated dry run against production data for a single org with about 8,600 open deals, roughly 87% resolved programmatically with zero tokens. Most of the rest needed only cheap rule evaluation. About 223 deals got an actual agent run. Those numbers are from a dry run, not live traffic. The thresholds are config values, not architecture, and we will publish how the split holds up once the job has real mileage.
At 35 to 50 cents per agent run, that filter is the difference between a nightly job that costs $78 and one that costs $3,000+.
Three layers do the filtering.
The largest class needs no intelligence at all. A deal that is 90+ days old, with no notes and no reminders in 30 days, and no email thread ever between the two parties, gets a programmatic closed-lost recommendation at confidence 1.0. No model. Batched 500 at a time.
The decisive condition is the last one. Our first design used a time window: no email in the past 60 days. Then we looked at the data. 99.4% of stale deals had zero email threads between the counterparties, ever. Not "went quiet." Never spoke. The time-window version was complexity purchased to handle a case that barely exists. We replaced it with the simpler, stricter rule: any communication history at all disqualifies a deal from the programmatic path, because a deal with history deserves the agent's read of that history.
Data first, then rules. The rule we would have designed from intuition was the worse one.
A single agent run consumes roughly 8,000 to 12,000 input tokens (deal context, email threads, activity history) and produces 1,500 to 3,000 output tokens (structured recommendations with reasoning). At current model pricing (see OpenAI's pricing page and Anthropic's pricing page), that is 35 to 50 cents per run, all-in with the ranker's second pass.
For 8,600 deals: if every deal got an agent run every night, that is roughly $3,000 to $4,300 per cycle. Per month, north of $90,000 for a single org. The three-layer filter reduces nightly agent runs to about 223. Actual cost: $78 to $112 per cycle. Two orders of magnitude.
Cost is the first-order argument. The second-order argument is signal quality. When the model sees every deal, including the 87% that are obviously dead, it spends tokens producing trivially correct recommendations ("close this dead deal") or wasted ones ("nothing to do here"). 223 pre-qualified deals means 223 runs where the model is reading actual signal, not noise. Per-recommendation quality goes up because input quality goes up.
Every model call adds 1 to 3 seconds of wall-clock time, depending on input size, model load, and structured-output enforcement. (Independent benchmarks like Artificial Analysis track variance across providers.)
For the nightly batch, per-run latency matters less than throughput: the job runs overnight. But the same filter architecture applies to the real-time intake pipeline, where a referral arrives over Slack or email and the user expects a response in seconds.
In the intake path, every unnecessary model call is 1 to 3 seconds the user waits. If intake called the model for intent classification, then entity extraction, then agent reasoning, that is 3 to 9 seconds before anything happens. We eliminated the intent classifier entirely (that story is in the intake post) and front-load deterministic checks before the model sees the input. Most intake interactions hit the model once, for the reasoning step that actually requires a model. Total latency stays under 4 seconds for the common case. OpenAI's latency optimization guide makes the same argument from the provider side.
Deals that survive layer 1 are alive. The question is which are worth a model's attention tonight. That is rule evaluation, not inference.
Strong triggers (any one qualifies a deal):
Momentum triggers (only count in pairs):
A single overdue reminder cannot justify an agent run on its own. Too weak a signal, too common. Two momentum signals together start to look like a story worth reading.
The rules also encode mutual exclusions (a deal cannot be both waiting-on-us and waiting-on-them), then sort and cap the night's candidates. The cap is a budget decision made explicit in code rather than discovered on an invoice.
The trigger rules above are one instance of a general problem: given a stream of candidates, decide which justify an expensive, slow, probabilistic step. Three patterns that apply broadly:
Pattern 1: Heuristic first. Before any model call, run the cheapest possible check. For deal triage, that is the staleness check in layer 1. For intake, deterministic phrase matching. For email classification, header parsing and sender lookup. If the cheap check produces a definitive answer, the model never sees the input. This handles the majority of volume in every pipeline we run.
Pattern 2: Confidence threshold with fallback. When a rule produces a score but not a definitive answer, compare it against a threshold. Above: act on the rule's output. Below: escalate to the model. The threshold trades cost for coverage. We default to conservative thresholds (escalate more) during the first month of any new pipeline, then tighten based on data. Key discipline: log every escalation so you can measure what the model decided differently from what the rule would have.
Pattern 3: Batch then triage. Collect candidates over a window, sort by signal strength, send only the top N to the model. Even after layers 1 and 2 filter, if 400 deals qualify but the budget supports 250 agent runs, the sort picks the 250 with the strongest triggers. The rest wait for the next cycle. This is ugly because some deals wait an extra day. It is honest: a budget is a budget, and pretending otherwise means surprise invoices or quietly dropping model calls without telling anyone. (For workloads that tolerate delay, providers offer discounted batch APIs; see OpenAI's Batch API and Anthropic's Message Batches.)
The last filter is temporal: skip any deal that already has a pending recommendation from the last five business days.
The design decision is what the gate keys on: pending age, not last-run time.
Gate on "we ran recently" and a deal goes dark for a week regardless of what the user does. Gate on "a pending recommendation exists" and the moment a user executes or dismisses everything on a deal, that deal is eligible again tonight. Human action resets the loop, not a timer. The queue stays responsive to the people who work it, and weekends do not count against the window.
By the time the agent runs, every deal in front of it has a reason to be there. The triage summary rides along in the prompt as discovery context: which triggers fired, what the rules noticed. Labeled as context, not conclusions. The agent reads the actual thread and decides, with the triage as a tip line rather than a verdict. The pipeline that gates what it produces is the other half of the story.
Deterministic rules for the knowable. Data analysis before rule design. Human actions as the reset switch. Users see the result as recommendations that are usually worth reading. If you want to run layer one's arithmetic on a deal of your own, the referral health check applies the same rules in the browser.
Introzy is free to start. No card required.