Suyeol Yun, Head of AI
Every AI benchmark scores a list of queries somebody wrote. But real queries almost never repeat. The unit that repeats is the task. We turned 33,453 anonymized customer queries into task signatures and clustered them into 501 task islands.
I. The unit is wrong
Every benchmark is making a bet. You test a system on a few hundred questions, and you expect the score to tell you something about the thousands you didn’t test.
Nobody picks a question because it is unique. You pick it because it is representative, because a system that gets this one right will probably get the similar ones right too. Machine learning calls that family a task, and its definition always carries two things at once: which questions belong to the task, and how an answer to them gets scored.1
Here is the problem. Recent benchmarks from the major vendors in this market write a separate rubric for every question, rather than defining rubrics over tasks. That settles what counts as the same kind of work: two questions belong to the same task when they get the same rubric. So the rubrics already carry a taxonomy, and it says every question is its own kind of work.
II. Deleting everything that doesn't change the work
The obstacle to drawing those groups is that queries never repeat: across those queries, 88% are one-of-a-kind even after normalization. You cannot tabulate the probability of a query you will never see twice. Prediction requires recurrence, and the unit that recurs is not the query. It is the task.
So we compute, for every query, its task signature: what remains when you delete everything that doesn't change the work the system must do. Ticker, language, market, dates, phrasing: deleted. The verb, the artifact requested, the evidence source, the qualifiers that change the job: kept. Two queries get the same signature exactly when they ask for the same work: same routing, same tools, same grading rubric. Cluster the signatures and the whole history of usage collapses into task islands: recurring families of work.
III. The map
Clustering the signatures yields 501 named task islands covering 89% of the 33,453 queries. These are the groups drawn solid: a measured, monitorable distribution over the kinds of work that exist. Some of what it shows, nobody would have put in an editorial benchmark: explaining sudden price drops and explaining rallies are distinct tasks customers ask differently; extracting interest-rate sensitivity tables from bank filings is a 227-query profession of its own; and 327 queries are customers probing what the product can do: a product-gap signal masquerading as usage.
IV. Reading the field through the map
With the map in hand, a question that has never been answerable becomes a routine computation: take any published benchmark's task list and place each item on the map. Coverage stops being a claim and becomes a number: which islands a benchmark touches, and how much real demand those islands carry.
The same map runs our own shop. With the tasks written down, coverage is something we can check, so nothing customers ask slips through unnoticed. We score the system island by island instead of on whichever queries happened to be handy, and improvements work the same way. A fix made at the task level reaches everyone who asks that kind of question.
never touches anything time-sensitive: Equity Price-Movement Attribution, Market Wraps, Meeting Prep.
never touches the high-traffic head: Revenue & Metric Retrieval, Segment & Revenue Attribution, Company News Updates.
never touches the live-desk islands: Market Wraps, Investor Relations & Meeting Prep, Equity Price-Movement Attribution.
V. What the map does not give you
The map hands a benchmark two things it never had: which tasks to test, and how much each one actually comes up. Scores can then be reported island by island, so a low number says where a system fails.
What it does not give you is the rubric. An island tells you which questions belong together. It does not tell you how to grade them.
And a rubric written once goes stale. One results release turns “not yet computable” into an exact reported figure overnight, and the same facts have to support different conclusions for an equity desk and a credit desk. A hand-picked query list grades a desk that does not exist. A hand-frozen rubric grades a market that no longer exists.
So we make the same move again, one level up. This post replaced hand-authored query lists with the unit of demand that recurs, the task. FinAutoRubric, our companion work, replaces hand-authored per-item rubrics with the unit of grading that recurs: the expert principle. Ten reusable principles, validated once by experts, are instantiated into a context-specific rubric for each task, bound to its entities, its information cutoff and its domain lens, then checked by an automatic refinement loop before acceptance.
Notes
- The pairing is standard across three decades and three subfields. Mitchell, T. M. (1997), Machine Learning, McGraw-Hill, §1.1, defines learning with respect to “a class of tasks T and performance measure P”: the class and its measure named together. Finn, C., Abbeel, P., & Levine, S. (2017), Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML (arXiv:1703.03400), §2.1, writes a task as 𝒯 = {ℒ, q(x₁), q(xt+1|xt,at), H}, with the loss ℒ and the distribution q as entries of the same object. Zhai, X., Puigcerver, J., Kolesnikov, A., et al. (2019), A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (arXiv:1910.04867), §2, describes a task as “a tuple containing a task-specific dataset distribution DT and evaluation procedure.” ↩ back
- The machinery, for readers who want it. Write S for the set of possible signatures (the shapes a question can take once the ticker, language, date and phrasing are gone), σ : Q → S for the map taking a query to its signature, and ρ̄ : S → R for the map taking a signature to the rubric template that grades that shape of question; the ticker and date re-enter as parameters when the template is instantiated. Two conditions: (i) ρq = ρ̄(σ(q)): which rubric a query gets depends on the query only through its signature; and (ii) s ≠ s′ ⇒ ρ̄(s) ≠ ρ̄(s′): distinct signatures receive distinct rubrics. Together they give q ∼ q′ ⇔ σ(q) = σ(q′): same signature exactly when same rubric. In partition terms, the rubric assignment partitions Q (FIG 1 called the cells tasks), and the signature partitions Q too. Condition (i) says the signature partition refines the rubric partition; condition (ii) says the refinement also runs the other way. Two partitions that refine each other are the same partition, so grouping by signature is grouping by rubric. Signatures are cheap to compute for all 33,453 queries; rubrics are not. ↩ back