DeepFinance - Trajectory-Graded Finance Benchmark

# DeepFinance - Trajectory-Graded Finance Benchmark DeepFinance is a long-horizon benchmark for PE and IB work. It grades the path to the answer, not only the answer.^[[Julius Research, "DeepFinance: Can AI diligence a $100MM deal?", 5 Nov 2026](https://www.juliusresearch.ai/blog/deep-finance)] - Built by Julius Research (the lab behind julius.ai). - Thesis: in real deal work, every number needs a source and every judgment call must be auditable. Final-output evals miss this. - It tests existing agent harnesses. It is not an agent or a product. ## Headline numbers | Metric | Value | |---|---| | Tasks | 21 | | Human time per task | 27.1 h (avg) | | Rubric criteria per task | 84 (avg) | | Files per data room | 19 (avg) | | Prompt length | 240 words (~29% shorter than GDPval) | - Four pillars: valuation, due diligence, underwriting, financial modelling. - Experts from J.P. Morgan, UBS, Evercore, HBS, Wharton and CBS wrote and reviewed the tasks. ## Diagram 1 — How a run works ```mermaid flowchart LR T["Task request objective · constraints · deliverables"] D["Data room ~19 files: xlsx, docx, eml, PDF, CSV CIMs · SPAs · term sheets · IC minutes conflicting sources on purpose"] H["Native harness claude-code · codex · opencode … web search on · ask-user off own sandbox"] O["Deliverable"] TR["Trajectory sources · tool calls · calcs · revisions"] G["Isolated grader 84 expert rubric criteria partial credit"] S["Score % criteria met avg over 10 attempts, then tasks"] T --> H D --> H H --> O --> G H --> TR --> G G --> S style H fill:#fff4d6,stroke:#c99a2e,color:#000 style G fill:#d4f1d4,stroke:#2e8a3a,color:#000 ``` ## Diagram 2 — What the rubric grades ```mermaid flowchart TB R["Rubric criterion"] R --> O["Output Are calcs, conclusions and deliverables correct?"] R --> W["Workflow Did the agent find and use the right evidence?"] R --> J["Judgment Does the evidence support each assumption and decision?"] style O fill:#e6f0ff,stroke:#3b6fc9,color:#000 style W fill:#fff4d6,stroke:#c99a2e,color:#000 style J fill:#ffe4e1,stroke:#c94f4f,color:#000 ``` - Prior evals mostly stop at Output. The post's comparison: - FinanceBench, BizBench: simple questions or one final number. - FinanceQA: hands the model the relevant 10-K section. - GDPval: preference ranking on final outputs. - Halluminate Westworld: studies the trajectory but does not score it. - Average criteria per task: BigFinanceBench 16.9, PRBench Finance 17.1, BlueFin 35.1, DeepFinance 84. ## Diagram 3 — Leaderboard (top 10, mean rubric score %) ```mermaid xychart-beta title "DeepFinance mean rubric score (%)" x-axis ["Opus 5.5", "Sonnet 5.5", "GPT-6 Astra", "GPT-6.1 Sol", "Fable 5.1", "GPT-6.1 Terra", "Gemini 4 Argon", "Muse Spark 1.3", "Kimi K3.1 Pro", "Grok 4.7"] y-axis "Score" 0 --> 60 bar [54.8, 54.6, 54.2, 52.9, 45.6, 43.5, 41.3, 37.0, 34.6, 34.1] ``` - Best score is ~55%. No model clears half the work reliably on the hardest tasks. - Top four (Opus 5.5, Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol) are statistically tied (paired bootstrap, 95%). - Anthropic and OpenAI take the top six. Best other provider: Gemini 4 Argon at 41.3%. - Bottom group (Nemotron 3 Ultra, Inkling, Inkling Small) scores 10.9–14.5%, ~40 points below the top. - Cost: GPT-6.1 Sol costs $0.81 per task at 52.9%. GPT-6.1 Terra costs $0.38 at 43.5%, the cheapest model above 30%. - Fable 5.1 is the most expensive model tested that scores under 50%. ## Diagram 4 — Why agents fail: the cascade ```mermaid flowchart LR E["Wrong upstream input earnings subtotal · source choice · timing"] E --> DC["Debt capacity"] E --> CF["Cash flows"] DC --> F["Fees"] CF --> V["Valuation"] V --> RT["Returns"] F --> RT RT --> X["Many failed checks ≠ many independent errors"] style E fill:#ffe4e1,stroke:#c94f4f,color:#000 style X fill:#fff4d6,stroke:#c99a2e,color:#000 ``` ## Failure modes - **Evidence without judgment.** Models find the data but rarely challenge given assumptions. They do not reconcile a CFO's number against an analyst's unless told to. - **Correct maths on wrong inputs.** - Lending task: right debt-capacity maths on a wrong earnings subtotal. - Pharma royalty task: right caps and milestones on a forecast that omits a sales channel. - **Evidence found but not used.** In an acquisition task, a later closing date cuts the required offer by $1.25M. Several attempts spotted the buyer's right to pick the date. No frontier attempt used it. > [!question] So what for an AI-native fund > The gap is judgment, not retrieval. A fund agent stack needs a step that forces source reconciliation and assumption challenge. See [[AI native hedge fund moc#22. Reflection agents]] and [[LLM-as-Judge]]. ## Caveats - 21 tasks only. All finance-specific. - Grading depends on expert rubrics and layered human plus agentic review. - No limitations or future-work section in the post. - Model names and scores are as reported by Julius. Not independently checked. ---- tags: #boat #agents #investing links: [[AI native hedge fund moc]] [[Evals]] [[Domain Experts as Eval Builders]] [[AI Verification]] [[Evaluation and Verification Infrastructure]] [[Agent Harnesses and Agentic Search]] [[LLM-as-Judge]]