Kai Intake: LLM vs Agentic Approaches

A natural-language referral is only useful once it becomes the right CRM records in the right org. Guessing the structure without tools is worse than a slower tool loop that can check its own work.

Kai intake on email and Slack: forward to kai@, or mention @Introzy Kai / /kai, and get contacts, companies, and connections tracked in Introzy. This post covers the pipeline, where we refuse to call a model, and why we moved from structured extraction to an agentic loop.

The Pipeline From Message to Reply

After a known sender hits email or Slack:

channel ingress (email or Slack)
  -> resolve actor + org (or refuse)
  -> build transcript
  -> (Slack) deterministic opener classify
  -> enqueue intake agent job
  -> run intake agent
       -> CRM tools: search / get / create
       -> stopWhen(stepCountIs(maxSteps))
  -> natural-language reply

Ingress stays thin: verify the webhook or inbound mail, resolve the actor, build a transcript, enqueue, return. If we cannot establish who you are or which org you belong to, we never start the agent. Tenant resolution happens before any CRM tool runs.

Channel adapters own auth, transcript shape, mutation caps, and reply delivery. CRM semantics live in the same tool handlers an external MCP client calls — no private "internal create" path. Channel-specific reply formatting is injected as a system prompt suffix: email gets "no markdown, no subject line, no sign-off." Slack gets "single-asterisk bold, never double-asterisk." The agent does not know which channel it runs on except through the suffix and the transcript shape.

Structured Slack shortcuts (filled modals) never enter this pipeline. Already structured input calls CRM helpers directly.

What We Tried First

Before the agent, intake ran a structured Pass-1 extract: one schema-shaped model call, a confidence number, then either commit or clarify. Create required high confidence and the required party fields. Braintrust evals scored above 96%, with the create gate tuned around 0.95.

The remaining failures were the expensive ones. Without CRM tools, a wrong or incomplete extract meant re-prompting and hoping the next schema fill stuck. The model could not search the CRM, check for duplicates, and settle the record in one loop. Email and Slack each grew their own clarify / commit variants, so a fix in one channel did not land in the other.

Pass-1 taught us the fields and the confidence gate. It could not give us search-before-create.

The Agentic Loop

Production intake is one shared agent for email and Slack. The config is intentionally boring SDK usage, not a custom agent framework:

export const INTAKE_AGENT_CONFIG = {
  modelId: 'claude-sonnet-4-6',
  // Mixed-intent worst case: search + get + create_company +
  // create_contact + create_connection ≈ 10 tool steps.
  maxSteps: 12,
  temperature: 0.2,
  maxOutputTokens: 4096,
} as const

const modelResult = await generateText({
  model: anthropic(INTAKE_AGENT_CONFIG.modelId),
  system: systemPrompt,
  prompt: userPrompt,
  tools,
  stopWhen: stepCountIs(maxSteps),
  temperature: INTAKE_AGENT_CONFIG.temperature,
  maxOutputTokens: INTAKE_AGENT_CONFIG.maxOutputTokens,
})

The seven tools — search_contacts, get_contact, search_companies, get_company, create_contact, create_company, create_connection — are the same MCP handlers an external client calls. A wrapHandler function strips organizationId from the tool input schema and binds it from tenant context:

const boundArgs = {
  ...(args as Record<string, unknown>),
  organizationId: ctx.tenantId,
} as unknown as T
const output = await safeExec(() => handler(boundArgs, ctx))

The agent never asks a user for an org UUID. The org was resolved at ingress.

De-duplication lives inside the create handlers. create_contact matches on email and returns isNew: false when the record exists. create_company matches on name. create_connection returns dealIsNew: false when the seeker/provider pair already has an open deal. The system prompt tells the model to read those flags:

Check isNew on create_contact / create_company results
(isNew: false means duplicate, not an error).
Check dealIsNew on create_connection.
Shape your final reply from those results.

Under Pass-1, a duplicate was a post-hoc fixup after the schema confidently created a second Jane Doe. Now it is a tool result the model reads before writing its reply.

After the loop, deriveOutcomeCategory classifies the run from tool results: mutation_success, duplicate_handled, clarification_needed, search_only, step_exhausted. The category is never used for routing — it feeds dashboards and Braintrust evals to spot step-budget exhaustion or rising clarification rates.

Don't LLM What You Can Classify

Slack openers hit a deterministic phrase matcher first. The handoff type is two values:

export type KaiSlackIntentHandoff = 'slack_intake_agent' | 'none'

Clear create-contact / create-company / create-opportunity phrases hand off to the agent. Empty or unsupported openers stay none.

The classifier can return unknown, and the handoff helper maps that to none. But the web adapter still sends substantive unknown text to the agent:

if (
  openerText.length > 0 &&
  (rawHandoff === 'slack_intake_agent' || input.result.intent === 'unknown')
) {
  return { handoff: 'slack_intake_agent', openerText }
}

Ambiguous natural language is the agent's job. We briefly put an intent-classifier LLM in front of this path for "hard" openers. The agent, with tools and the full thread, re-derived intent on its own. The classifier label never changed the write path. Pure cost and latency. We deleted it.

Don't LLM what you can classify, and don't classify with an LLM what the intake agent will re-derive with tools.

Where We Refuse the Model

Same philosophy as failing closed on ambiguous Zoom calendar matches: wrong writes are worse than waiting.

Empty /kai gets an ephemeral hint. Unknown email senders get a deterministic reply. Over the daily mutation budget, the worker skips the model entirely:

const mutationCount = await countKaiSlackAgentMutationsForOrganizationSince(
  prisma,
  organizationId,
  capWindowStart
)
const capCheck = evaluateKaiSlackAgentMutationCap({ count: mutationCount })

if (capCheck.exceeded) {
  await postAgentReply({ /* fixed over-cap copy */ })
  return
}

No tool loop. Fixed copy, then stop.

The cap is a rolling 24-hour window over events tagged with the agent's mutation source — default 20 writes per org per day, env-configurable. The query counts USER_CREATED, COMPANY_CREATED, and PIPELINE_DEAL_CREATED events where metadata.source matches the agent tag. Human CRM writes do not count against the agent's budget.

What Surprised Us

The best change was a deletion. The intent-classifier LLM felt like a responsible middle ground between brittle phrases and a full agent. Once the intake agent existed, the classifier was redundant.

The second surprise was stale docs. After Pass-1 was removed, QA playbooks still described pass1 gates and confidence clarify paths for weeks. Ops followed playbooks that could not happen on main. Update the playbooks in the same PR as the cutover.

When to Use Which

1. Deterministic code or a structured modal when the path is known or must fail closed: empty openers, unknown senders, mutation caps, clear phrase routing, filled forms.
2. An agentic tool loop when the user describes parties in natural language and you need search, de-dup, and create against CRM with the thread carrying context.
3. A one-shot structured LLM without tools — usually avoid for intake. It scores well on evals. The expensive tail is where you collect examples instead of resolving records.

The iteration comes from tools plus multi-turn transcript, not from a second classifier labeling the first model.

Where This Goes Next

• Field-level provenance for what the agent searched versus what it created
• The full story of how email and Slack share one intake agent end to end

Try Introzy: Forward an intro to Kai, or mention @Introzy Kai in Slack, and let intake land as CRM records you can actually trust. Get started free →

We ship in public. Come build with us.

Introzy is free to start. No card required.