Agent Experience (AX) Benchmark

How well does an AI agent complete real fund-ops tasks when FundOS documentation is the knowledge source?

Last run: 2026-07-29  ·  Model: gemini-2.5-flash  ·  20 tasks  ·  Harness on GitHub ↗
20/20
Tasks correct
0
Hallucinations
100.0%
Score (40/40)
100.0%
Approval gate coverage (4/4)
32884t
Avg tokens loaded

FundOS AX vs Raw API Reference (same model, same tasks)

Both conditions ran the identical task set on the identical model. The only variable is the knowledge source: curated AX docs (llms.txt + llms-full.txt + CLAUDE.md) vs the raw /api/docs HTML. Neither side is truncated.

MetricFundOS AXBaseline
Tasks correct 20/20 13/20
Hallucinations 0 0
Score 40/40 (100.0%) 26/40 (65.0%)
Approval-gate coverage 4/4 4/4
Avg tokens loaded 32884 35957
Avg response time 1355 ms 1559 ms

Category Breakdown

Category Tasks Score Correct Hallucinations
Agents 2 4/4 2/2 —
CFO / Finance 2 4/4 2/2 —
Deal CRM 3 6/6 3/3 —
HF Ops 1 2/2 1/1 —
LP / Investors 3 6/6 3/3 —
MCP / Auth 3 6/6 3/3 —
Risk 2 4/4 2/2 —
VDR / Documents 3 6/6 3/3 —
Webhooks 1 2/2 1/1 —

Per-Task Results (FundOS AX Condition)

# Category Task Score Hallucination
1 MCP / Auth How do I connect the FundOS MCP server to Claude Code? Give the exact CLI command. ✓ 2/2 —
2 MCP / Auth What transport protocols does the FundOS MCP server support? ✓ 2/2 —
3 MCP / Auth How do I authenticate with FundOS MCP using a bearer API key? Show the exact header. ✓ 2/2 —
4 Deal CRM What MCP tool should I call to get the full deal pipeline as a kanban view? ✓ 2/2 —
5 Deal CRM I want to add a new deal to the pipeline from an inbound pitch. Which tool do I use and do… ✓ 2/2 —
6 Deal CRM How do I score a deal's IRR, MOIC, and WAL in FundOS? ✓ 2/2 —
7 LP / Investors What tool lists all LP investors in the system? ✓ 2/2 —
8 LP / Investors I need to issue a capital call to an LP. What tool do I call and what human-approval step … ✓ 2/2 —
9 LP / Investors Compute the LP/GP waterfall splits at exit for a European waterfall fund. ✓ 2/2 —
10 VDR / Documents I want to answer an LP's diligence question in a data room Q&A. Which tool do I use and do… ✓ 2/2 —
11 VDR / Documents How do I search across all documents in a deal room with natural-language queries? ✓ 2/2 —
12 VDR / Documents I want to grant a new investor access to a data room. Which MCP tool do I call and what is… ✓ 2/2 —
13 Risk What MCP tool checks whether a covenant is in breach given a new financial value? ✓ 2/2 —
14 Risk How do I get a list of currently open risk alerts for my portfolio? ✓ 2/2 —
15 CFO / Finance Compute P&L and NAV for a fund account over a date range. Which tool and what parameters d… ✓ 2/2 —
16 CFO / Finance How do I generate a CIM (Confidential Information Memorandum) for a deal in FundOS? ✓ 2/2 —
17 Agents What is the first MCP tool I should call at the start of any multi-step FundOS workflow to… ✓ 2/2 —
18 Agents How do agents propose write actions without auto-executing them in FundOS? ✓ 2/2 —
19 HF Ops Which MCP tool retrieves the T+0 affirmation scorecard for hedge fund ops? ✓ 2/2 —
20 Webhooks What event fires when an agent proposes a write action that needs human approval in FundOS… ✓ 2/2 —
Reproducibility note: This is FundOS's own benchmark of its own documentation. We do not claim third-party or independent validation. Every task, result, and methodology file is open source at github.com/8vdx1/fundos-mcp. Run it yourself: pip install anthropic pyyaml && python benchmarks/ax/runner.py. Scoring: 2pts = correct answer + human-approval gate met; 1pt = correct but missed gate warning; 0pts = wrong or hallucinated.