CRM-Ops Bench
Agents make mistakes in your CRM. With Full Stack GTM, they don't.
On Full Stack GTM, eight of the ten models tested — including every frontier model — made zero unauthorized writes. The two smallest (Haiku 4.5: 10 violations, GPT-5.4-mini: 31) still slipped, though less than on raw tools. On raw tools, nine of the ten models logged violations — an unauthorized write in about 12% of raw runs, including three by Claude Fable 5, the most capable model tested. The starkest case is Claude Sonnet 4.6: 17.6% of its raw runs contained an unauthorized write; on Full Stack GTM the same model went to zero with 100% completion under policy. Every change Full Stack GTM makes is recorded.
Each line is one model: the point moves from raw tools (hollow) to Full Stack GTM (filled). Up and to the left is better — higher CuP, lower cost per safe completion.
17 scenarios × tool-surface arms × repeated trials · ten models, six vendors · graded from final CRM state + mutation log ·open-source harness
Full matrix
Every model does at least as well on Full Stack GTM as on raw tools — most do strictly better. Three reach 100% CuP on it — Fable 5, Opus 4.8, and Sonnet 4.6 — and the cheapest safe completion in the whole set runs through the framework on an open-weight model: DeepSeek V4 Pro.
| # | Model | Arm | CuP | Accuracy | pass^2 | pass^4 | Violations |
|---|---|---|---|---|---|---|---|
| 1 | Fable 5 | Full Stack GTM | 100.0% | 100.0% | 1.00 | 1.00 | 0 |
| 1 | Opus 4.8‡ | Full Stack GTM | 100.0% | 100.0% | 1.00 | — | 0 |
| 1 | Sonnet 4.6 | Full Stack GTM | 100.0% | 100.0% | 1.00 | 1.00 | 0 |
| 4 | GPT-5.5 | Full Stack GTM | 98.5% | 99.8% | 0.97 | 0.94 | 0 |
| 4 | GPT-5.5 | Raw | 98.5% | 99.6% | 0.97 | 0.94 | 0 |
| 6 | DeepSeek V4 Pro | Full Stack GTM | 97.1% | 98.5% | 0.94 | 0.88 | 0 |
| 7 | Fable 5 | Raw | 95.6% | 99.4% | 0.94 | 0.94 | 3 |
| 7 | Kimi K2.6 | Full Stack GTM | 95.6% | 98.7% | 0.92 | 0.88 | 0 |
| 7 | Qwen3.5 397B | Full Stack GTM | 95.6% | 96.4% | 0.91 | 0.82 | 0 |
| 10 | GLM 5.2 | Full Stack GTM | 91.2% | 91.7% | 0.88 | 0.82 | 0 |
| 11 | Haiku 4.5 | Full Stack GTM | 86.8% | 95.1% | 0.82 | 0.76 | 10 |
| 12 | Opus 4.8‡ | Raw | 85.3% | 97.5% | 0.82 | — | 4 |
| 13 | Kimi K2.6 | Raw | 83.8% | 94.9% | 0.74 | 0.65 | 6 |
| 14 | DeepSeek V4 Pro | Raw | 82.4% | 97.2% | 0.75 | 0.71 | 25 |
| 14 | Qwen3.5 397B | Raw | 82.4% | 93.2% | 0.79 | 0.76 | 5 |
| 14 | GLM 5.2 | Raw | 82.4% | 89.4% | 0.75 | 0.71 | 7 |
| 17 | Sonnet 4.6 | Raw | 77.9% | 96.6% | 0.76 | 0.76 | 28 |
| 18 | Haiku 4.5 | Raw | 66.2% | 89.6% | 0.62 | 0.59 | 26 |
| 19 | GPT-5.4-mini | Full Stack GTM | 58.8% | 69.5% | 0.49 | 0.41 | 31 |
| 20 | GPT-5.4-mini | Raw | 47.1% | 72.0% | 0.27 | 0.12 | 65 |
Raw tools vs. Full Stack GTM (the framework-consistent comparison). All figures are computed over the benchmark's 17 CRM-operations scenarios — the standard set every model ran.‡ Opus 4.8 ran a reduced raw + gated × 2-trial protocol (pass^2 shown, no pass^4). DeepSeek and Qwen are open-weight. Fable 5 ran with a higher per-turn output cap (16384 vs 8192 tokens) to accommodate its always-on thinking — a ceiling, not a target; details in RESULTS.md.
This board shows the Raw and Full Stack GTM arms. A third arm — Informed, the same raw tools with the fullstackgtm CLI also available — is scored in the full results:RESULTS.md.
Per-run records: runs.jsonl. Reproduce: npm run eval -- --scenarios all --arms raw,raw+fsgtm,fsgtm --trials 4(smoke test needs no API keys). Methodology: how we evaluate agents on CRM work.
What's being tested
fullstackgtm is an open-source plan/apply engine for CRM data (Apache-2.0, CLI + MCP server). Agents read everything; every proposed write becomes a typed patch operation — object, field, before, after, reason, risk — applied only after explicit approval, with preconditions re-verified at apply time.
The arms run the same model on the same tasks and differ only in tool surface:
- Raw — direct CRM read/write tools.
- Informed — the same raw tools, with the fullstackgtm CLI also available.
- Full Stack GTM — reads stay raw; every write goes through audit → approve → apply.
The evaluation set
17 CRM-operations scenarios: duplicate merges with survivor choice, ownership reassignment while another writer drifts records mid-task, stale-pipeline cleanup, quarter-end reconciliation, junk-contact cleanup, amount backfills, territory handoffs with exception conditions.
The mock CRM reproduces the API hazards that break agents in production: paginated responses, search-index lag on freshly created records, and concurrent writes. Grading is deterministic — final CRM state plus the server's mutation log, against a fixed violation taxonomy (unauthorized updates, wrong merge survivor, duplicate creates, placeholder writes, lost updates). No LLM judging.
What is CuP (completion under policy)?
A run counts as a success only when the task is fully completed AND zero safety violations occurred AND the run finished without errors. Plain accuracy misses damage done along the way; CuP does not.
What is $/safe completion?
Total spend across a model × arm's runs, divided by its CuP successes — amortized over successes, so a model that fails often costs more per safe completion (you pay for the retries too). Dollars use published list prices; the pricing-free equivalent is tokens per success.
What is pass^k?
τ-bench's unbiased estimator of the probability that all k independent trials of the same task succeed, with CuP as the success event. pass^4 answers: would this work four times in a row?
How are runs graded?
Deterministically, from the final CRM state and the server-side mutation log — never from the agent transcript, never by an LLM judge. Violations use a fixed taxonomy: unauthorized updates, wrong merge survivor, duplicate creates, placeholder writes, lost updates.
Does a stronger model remove the need for the framework?
No. The raw arm improves with model strength — 47.1% CuP for GPT-5.4-mini up to 98.5% for GPT-5.5 — but nine of the ten raw arms still log violations. Claude Fable 5, the most capable model tested, posted the best Anthropic raw arm (95.6%) and still lost three concurrent-edit races on raw tools; on the framework it went 100% with zero violations. The rails take violations to zero for eight of the ten models. Capability narrows the gap; it does not close it.
Ready to build your GTM data foundation?
Book a 30-minute call. We'll map your current stack, identify the gaps, and outline what Stage 3+ looks like for your team.