BigFinance
Bench

Models

Questions

928

Updated

Sep 17, 2026

This page tracks frontier model performance on BigFinanceBench: a leaderboard, cost-versus-accuracy view, and a worked example showing how models are graded step by step.

Finance research is graded on the audit trail, not just the bottom line. 928 SME-authored questions, each with an expert-authored grading rubric. Every model runs the same minimal open-source harness, so scores measure the model, not the scaffolding. Scores are a two-judge mean (Gemini 3.1 Pro and Claude Opus 4.7) averaged across model families to limit self-preference. New models are added as they're released.

Leaderboard

1.	Muse Spark 1.1	53.4%
2.	GPT-6 Astra	49.5%
3.	Claude Opus 5	46.1%
4.	Claude Fable 5	45.2%
5.	Gemini 3.6 Flash	44.3%
6.	GPT-5.5	44.3%
7.	GPT-5.6 Sol	44.2%
8.	Gemini 3.5 Flash	42.2%
9.	Grok 4.5	41.7%
10.	Claude Opus 4.7	41.5%
11.	Kimi K3	41.0%
12.	Claude Opus 4.8	40.6%
13.	GPT-5.6 Terra	39.6%
14.	Claude Sonnet 5	39.4%
15.	Claude Sonnet 4.6	38.4%
16.	Qwen 3.6 Max	36.8%
17.	GPT-5.6 Luna	36.1%
18.	GLM 5.1	35.7%
19.	DeepSeek V4 Pro	35.6%
20.	Gemini 3.1 Pro	34.9%
21.	MiniMax M3	31.6%
22.	Qwen 3.6 27B	30.0%
23.	Gemini 3.5 Flash-Lite	29.0%
24.	Kimi K2.6	26.6%
25.	Gemini 3 Flash	26.2%
26.	Nemotron 3 Ultra	21.9%
27.	Gemma 4 31B-IT	20.8%
28.	GPT-5.4 Mini	6.6%
29.	Mistral Large 3	6.5%

Final Answer measures whether the model's response matches the expert answer.

GPT-6 Astra ran at OpenAI reasoning effort “high”. Other OpenAI rows use the provider's default effort.

Cost vs. accuracy

A selection of models are labeled; every dot reveals its model on hover.

Sample prompt

What is the YoY change in BPS for Core Net Take Rate (payments-only yield) for Toast from FY 2022 - FY 2024? Please round the final answer to the nearest hundredth of a bp

Reference answer

-1.09 bps, +1.14 bps

Compare
Outputs

GPT-5.5

Claude Opus 4.7

Rubric94.9%

Points37/39

Steps22

Rubric7.7%

Points3/39

Steps6

Final Answer

“FY23 vs FY22: -1.09 bps; FY24 vs FY23: +1.14 bps.”

Final Answer

“Using Toast's Financial Technology Solutions revenue net of related costs of revenue divided by GPV as the Core Net Take Rate (payments-only yield): • FY2022: ($2,268M − $1,792M) / $91.7B ≈ 51.91 bps • FY2023: ($3,189M − $2,503M) / $126.1B ≈ 54.40 bps • FY2024: ($4,053M − $3,175M) / $159.1B ≈ 55.19 bps YoY change in BPS: • FY2022 → FY2023: +2.49 bps • FY2023 → FY2024: +0.78 bps”

Trajectory

Search
Toast Core Net Take Rate payments-only yield FY 2024 2023 2022
Edgar
TOST 10-K
Fetch
www.sec.gov/tost-20241231.htm query: "Core Net Take Rate payments-only yield net take rate GPV"
Fetch
www.sec.gov/tost-20231231.htm query: "Core Net Take Rate payments-only yield net take rate GPV"
Fetch
www.sec.gov/tost-20221231.htm query: "Core Net Take Rate payments-only yield net take rate GPV"
Search
site:investors.toasttab.com Core Net Take Rate Toast payments-only yield 2024
Search
Toast Q4 2024 net take rate basis points payments Toast Capital gross profit
Fetch
s28.q4cdn.com/Toast-4Q-and-FY-2024-Earnings-Call-Prepared-Remarks.pdf query: "full year 2024 core net take rate payments-only yield basis points 2023 2022"
13 steps elided
Fetch
www.sec.gov/tost-20241231xexhibit991.htm query: "Payments financial technology solutions gross profit Non-GAAP Non-Payments Fina…"
Python
gp={2022:418,2023:561,2024:726} gpv={2022:91.7,2023:126.1,2024:159.1} for y in [2022,2023,2024]: tr=gp[y]/gpv[y]*10 print(y,tr) print('YoY 23-22',gp[2023]/gpv[2023]*10 - gp[2022]/gpv[2022]*10) print(…

Trajectory

Edgar
TOST 10-K
Fetch
www.sec.gov/tost-20241231.htm query: "gross payment volume GPV payments revenue transaction-based take rate"
Fetch
www.sec.gov/tost-20241231.htm query: "financial technology solutions revenue transaction-based costs Toast Capital"
Fetch
www.sec.gov/tost-20221231.htm query: "Gross Payment Volume GPV 2022 2021 key business metrics"
Python
# Data gpv = {2022: 91.7e9, 2023: 126.1e9, 2024: 159.1e9} fintech_rev = {2022: 2268e6, 2023: 3189e6, 2024: 4053e6} fintech_cost = {2022: 1792e6, 2023: 2503e6, 2024: 3175e6} # Net take rate (payments-…
Answer

Rubric

1 point

Records 2022 Gross Payment Volume (GPV) = $91,700,000,000

1 point

Records 2022 Gross Payment Volume (GPV) = $91,700,000,000

1 point

Records 2023 Gross Payment Volume (GPV) = $126,100,000,000

1 point

Records 2023 Gross Payment Volume (GPV) = $126,100,000,000

1 point

Records 2024 Gross Payment Volume (GPV) = $159,100,000,000

1 point

Records 2024 Gross Payment Volume (GPV) = $159,100,000,000

4 points

Defines Core Net Take Rate as Non-GAAP Payments Gross Profit as a % of GPV

4 points

Defines Core Net Take Rate as Non-GAAP Payments Gross Profit as a % of GPV

2 points

Records 2022 Non-GAAP Payments Gross Profit = $418,000,000

2 points

Records 2022 Non-GAAP Payments Gross Profit = $418,000,000

2 points

Records 2023 Non-GAAP Payments Gross Profit = $561,000,000

2 points

Records 2023 Non-GAAP Payments Gross Profit = $561,000,000

2 points

Records 2024 Non-GAAP Payments Gross Profit = $726,000,000

2 points

Records 2024 Non-GAAP Payments Gross Profit = $726,000,000

2 points

Calculates 2022 take rate = 0.4558% (+/- 0.0001%)

2 points

Calculates 2022 take rate = 0.4558% (+/- 0.0001%)

2 points

Calculates 2023 take rate = 0.4449% (+/- 0.0001%)

2 points

Calculates 2023 take rate = 0.4449% (+/- 0.0001%)

2 points

Calculates 2024 take rate = 0.4563% (+/- 0.0001%)

2 points

Calculates 2024 take rate = 0.4563% (+/- 0.0001%)

10 points

Calculate bps change between 2023 and 2022 = -1.09 bps, when rounded to the nearest hundredth of a basis point

10 points

Calculate bps change between 2023 and 2022 = -1.09 bps, when rounded to the nearest hundredth of a basis point

10 points

Calculate bps change between 2024 and 2023 = +1.14 bps, when rounded to the nearest hundredth of a basis point

10 points

Calculate bps change between 2024 and 2023 = +1.14 bps, when rounded to the nearest hundredth of a basis point

BigFinanceBench

Leaderboard

CompareOutputs

BigFinance
Bench

Compare
Outputs