← DataBard
Public accounting — model benchmarkMeasured, never crafted
Benchmark · Beam 1.0 501B
Can an open 501B model do a data analyst's job?
We ran Reflection Beam 1.0 501B-A23B through DataBard's actual synthesis workload — the exact production prompts that turn a schema into a two-host analysis script — and graded every output the same way our pipeline does. Built for the Beam × Hugging Face build contest. Results below are frozen from the harness run; nothing was curated or re-rolled.
Beam 1.0 501B-A23B (Reflection API)
65%
mean check score
- Runs passed
- 12/12
- Mean latency
- 7.7s
- Output tokens
- 26,206
- …of which reasoning
- 19,166
Production synthesis model (llama-3.3-70b)
98%
mean check score
- Runs passed
- 12/12
- Mean latency
- 52.2s
- Output tokens
- 6,307
Methodology
- Workload:DataBard's production synthesis prompts — schema + computed health insights in, two-host podcast script or executive summary out. 6 tasks across two labelled demo schemas (a Uniswap V3 subgraph and an e-commerce warehouse), podcast and executive formats, with and without a research question. 2 run(s) per task per model, temperature 0.8.
- Checks per output:valid JSON array (the pipeline rejects anything else), segment count in the format's range, speakers limited to the two hosts, sane narration length, the computed health score actually appears in the script, and — for question tasks — the question gets answered.
- Access:Beam via Reflection's own API under its free daily allowance (no spend). Beam is a thinking model — reasoning tokens count against the output budget, so the harness inflates
max_tokens4× to keep reasoning from starving content. - Run: · harness
scripts/beam-bench.mjs
Per-task scores
| Task | beam | prod |
|---|---|---|
| web3-podcast | 90% | 100% |
| ecom-podcast | 100% | 90% |
| web3-exec | 0% | 100% |
| ecom-exec | 0% | 100% |
| web3-question“Which tables carry the biggest cascading risk if they fail?” | 100% | 100% |
| ecom-question“Where are the governance gaps in this warehouse?” | 100% | 100% |
Findings
- Shape failures, not gibberish: 4 of 24 runs produced parseable JSON with the wrong shape (4× object-not-array). In production the pipeline would have rejected these and fallen back to templates — the honest, boring failure mode, which is exactly what this bench measures.
- The reasoning tax is real: 19,166of the output tokens were reasoning. Beam's pitch is token-efficient thinking — here reasoning dominated the budget, which is why small
max_tokensvalues returned empty content until the harness inflated the budget. - Latency: see the headline cards. The bench measures the synthesis call only — no audio generation, no data fetch.
What next
The harness lives in the repo (scripts/beam-bench.mjs) — rerun it any day against Reflection's free daily allowance and the numbers on this page update. That is the point: a benchmark as a live accounting artifact, not a one-off blog post.