Finance Eval Meta-Analysis

Frontier finance benchmarks

The arc of LLM progress now runs through an enormous, unprecedented physical buildout. AI capex this year is on the order of $1T across hyperscalers, labs, and the broader supply chain. Though there is debate on the timing, it is consensus that this investment must eventually earn a return, and that it would come from deploying models across the economy, with white-collar verticals first.

Finance in particular presents a core focus, given its outsized influence on economic function, business creation, and capital allocation. It is also an unbounded white-collar field. Beyond the work financial firms staff today, there is nearly limitless demand in doing more, from deeper research coverage to broader economic modeling to government policy budgeting and planning.

As such, we sought to evaluate the state of model progress within finance, to see both how far we have come and where we are heading. One of AGI's end goals is to automate critical portions of finance, not only for frontier-lab ROI and enterprise efficiency, but also to further one of America's greatest strengths, its robust and liquid capital markets. Greater visibility into that development is then critical.

We propose, to our knowledge, the first meta-analysis of this kind.

Model card chronology

In particular, we first look toward the set of benchmarks that frontier labs themselves cite on their model cards. Figure 1 presents this timeline on a per-lab basis. Though not every model release includes a finance score, it is clear that finance is an increasingly important priority for the frontier labs. Of the 9 flagship launches from July through December 2025, only Anthropic's cite a third-party finance benchmark. From February 2026 onward, finance evaluations appear in launches from most major labs.

Saturation since release

In addition to the finance benchmarks directly cited by frontier lab model cards, we believe there to be a broader set of evals equally representative of model performance and interest, albeit not publicly (Figure 2). We also measured their best published model score on a per-benchmark basis from initial release to today.

Two observations follow from Figure 2. First, every benchmark with a published starting point has climbed since release, in some cases by double digits within months. Second, and we believe more importantly, each benchmark remains relatively unsaturated. Even with the best frontier models, the best published scores still hover around 50 to 60%, and no leaderboard has crossed 70% on their headline metric. Models have improved quickly on these evals without coming close to completing them.

Benchmark landscape

We found that, despite these finance evals being nominally finance-specific, they all still highly vary in their construction.

The underlying reason is that finance itself is a broad, load-bearing term, with many individual sub-industries comprising many different jobs, each involving many various tasks. One way to begin defining finance is top-down, delineating between “buy-side” and “sell-side,” then across investment banking, equity research, private equity, and hedge funds, and then across asset classes within each. Another way is bottom-up, as tasks themselves can be stratified across the nature of their expected input-output relationship. Certain tasks involve just retrieval-based Q&A, for instance, whereas other longer-horizon tasks involve completed deliverables in applications like Excel or PowerPoint. Thus, finance benchmarks by construction have to be opinionated on what exact types of workflows they seek to approximate and measure model progress by.

We believe that there is little open discussion comparing such nuances of finance benchmarks, despite their active use in hillclimbing frontier models and thus deployment across financial firms. As such, we hope to provide this write-up as a valuable source of said context.

Primary differences

Of the many criteria that separate workflows within finance, two broad axes dominate. The first axis separates deterministic answers from open-ended answers. The second axis separates Q&A and retrieval-based questions from Excel, PowerPoint, and Word-based deliverables.

With this framing in mind, we plotted the set of finance benchmarks below, illustrating the primary design differences (Figure 3). It's clear that these evals, despite being finance-focused, are all quite different. The implication is that eval score comparability is only useful conditioned on further grouping or clustering based on sub-characteristics.

What's more is that one might expect Q&A and retrieval-based evals to eventually give way toward longer-horizon, deliverable-based evals, with the latter being more complex. However, this has yet to be the case. Vals' v2.0 Finance Agent, Rogo's BigFinanceBench, and Thoughtful Lab's DiligenceBench (Q&A and retrieval-based benchmarks) all released after Mercor's APEX-Agents and Handshake's BankerToolBench (deliverable-based benchmarks), to similarly low levels of model saturation. At the moment, the output axis of whether a deliverable is involved or not does not seem to proportionally affect task difficulty.

Detailed overview

Importantly, the characteristics distinguishing finance benchmarks are many, far beyond just the two axes presented above. Below, in Table 1, we detail each benchmark in further context. Variables include: the information provided to the model at test-time, the tools and harness available, the required output, the grading method and verifier, and the released task-level artifacts.

Note that we encourage readers to read the original source documents provided by each benchmark constructor to get the clearest, task-level context of what models are being tested on.

Table 1. 7 benchmark entries with public materials. Vals v1.1, a February 2026 refresh, uses the same 537-question benchmark with updated data, harness, and evaluation. The released-artifacts row records what each publisher had posted on July 24, 2026.

Task distribution

Each benchmark report also includes a published task distribution by their categorization of choice (Figure 4). For example, Rogo's largest workflow is KPIs and unit economics, Mercor's is sensitivity analysis, and Handshake's is financial modeling and scenario work. It is important to note that the classification schemes themselves differ, with Vals classifying by question type, Mercor and Rogo by workflow, Handshake by product, and DiligenceBench by sector. Since we do not have access to the private datasets for each benchmark at this time, we rely on each constructor's own definitions rather than an independently derived, shared taxonomy.

Case studies

This collection of finance benchmarks was released with varying degrees of public transparency, with Rogo's BigFinanceBench and Vals' Finance Agent being the most auditable. Since these two evals do not require any private workspaces, documents, or harnesses to conduct tests on, below, we provide preliminary case studies.

More specifically, since Rogo's benchmark releases per-model trajectories and LLM-judge grades, we audit those directly. And for Vals' benchmark, we generate the trajectories ourselves, running its 27 public v2.0 tasks.

Limitations

This article is v1.0 of a versioned reference, published in July 2026 from publisher materials we accessed on July 24, 2026. Later readings, Rogo's July 28 leaderboard figure, Hebbia's materials accessed July 29, and the Figure 1 launch-artifact checks completed July 29, are dated where they appear. Our landscape map and comparison table will refresh in later versions under the same definitions. Both case studies stay frozen, one on Rogo's May 2026 public-50 release and one on our July 2026 Vals runs, and later versions will note changes rather than edit them silently. Cite this article as Finance Eval Meta-Analysis, v1.0, July 2026.

Rogo produced and owns the released model runs and grades. We did not run or grade any model on Rogo's tasks, and no part of our Rogo analysis independently replicates Rogo's results. Figure 1 quotes each lab's launch materials as written, with sources recorded per entry in the model-card record, so its scores are the labs' self-reported results rather than the benchmark publishers' gradings. Figure 2's at-release values likewise come from each publisher's launch materials. We placed each benchmark in Figure 3 by editorial judgment from public task and output descriptions, without measuring anything. The task-distribution panels in Figure 4 restate each publisher's own published category counts. Hebbia's landscape entry rests on its September 2025 release post and methodology paper, accessed July 29, 2026, the only materials it publishes. We assigned the failure modes by reading trajectories, and we established no cause experimentally.

Our Vals case study rests on runs we executed ourselves in July 2026, with MiniMax M3 as the only solver, GLM 5.2 as the only judge, and 3 trials per task. We kept the run manifests in the repository. Those runs give us model-failure evidence, and only that. They do not support comparison with Vals' published leaderboard, which uses Vals' agent, judges, and configuration, none of which we reproduce.

Retained means we can use a task for analysis, not that the task is free of every issue. Our retained Rogo set contains 26 valid tasks, 9 with convention or clarity notes, and 7 with rubric errors we judge nondecisive. Some export rows share a model, task, and trial key, and where a matching grade existed we kept the completed retry over the infrastructure-error row. That choice changes no majority-vote count in Figure 5.