Benchmarks - Measurement

GPT-6.2 Astra is a great fact-checker. It is not yet an analyst.

GPT-6.2 Astra is a great fact-checker. It is not yet an analyst.

GPT-6.2 Astra is a great fact-checker. It is not yet an analyst.

Benchmarks - Measurement

Jacob Choi, Guyoung Jung & Suyeol Yun

Jacob Choi, Guyoung Jung & Suyeol Yun

We ran GPT-6.2 Astra on 18 real analyst tasks: it was the fastest model and the best at citing sources, but its shorter answers left it about 5 to 8 points behind GPT-6 Astra run the same day.

5 to 8 pts
How far GPT-6.2 Astra trailed GPT-6 Astra on the same 18 tasks, depending on which GPT-6 Astra run you compare with.
131 s
Median time per task, the fastest of the five models, against 219 seconds for GPT-6 Astra run the same day and 217 for Claude Opus 5.5.
94.5 vs 59.9
Best of five models on evidence and citation, last on coverage: it cites what it says, but it says less.

Ask GPT-6.2 Astra to check a number in a 10-K and you get a fast, clean answer with the source attached. Ask it to prepare you for a management meeting and you get something shorter than you need: the right drivers named, but not worked through.

That is the short version of what we found. OpenAI released GPT-6.2 Astra on November 17, and we ran it through the same 18 real analyst tasks we use for every model in this series. It was the fastest model we tested and the best at citing its sources. It also scored about 5 to 8 points below GPT-6 Astra, which we ran on the same day.

I. Should you switch?

Every new release brings the same question from research teams. Early results from other benchmarks of short, precise questions put GPT-6.2 Astra near the top. We wanted to see how it handles the requests analysts actually send: prep me for this meeting, test this thesis, write the credit view.

We kept everything else fixed. Same tasks, same data and tools, same instructions, same three AI judges. Only the model changed. Scores can move from one day to the next, so we also reran GPT-6 Astra on the same day to get one clean comparison.

II. Faster, but a step behind

At its high effort setting, GPT-6.2 Astra scored 71.1 out of 100. GPT-6 Astra, run the same day, scored 79.0.

That 79.0 was at the high end of what we have seen from GPT-6 Astra, so we also compared against all our earlier GPT-6 Astra runs. Depending on which run you pick, the gap is about 5 to 8 points. GPT-6.2 Astra is behind in every case.

We tested the Claude models the week before, on different days, so we do not quote an exact gap. The direction is clear: on full research requests, the Claude models remain ahead.

FIG 1Score against cost per task
GPT-6.2 Astra (solid) and GPT-6 Astra (dashed)Claude Opus 5.5Claude Fable 5.1 (solid) and 5.2 (dashed)
Each line is one model at medium, high and top effort; GPT-6.2 Astra's cost is provisional and Claude results are from runs the week before.

The effort setting controls how long the model thinks before it answers. Turning it up helps, but only so far. Going from medium to high added about 10 points. Going from high to the top setting added 1.3, small enough to be noise between runs.

III. Where it wins: speed

A typical task took GPT-6.2 Astra 131 seconds. GPT-6 Astra, run the same day, took 219 seconds, and Claude Opus 5.5 took 217. If your day is full of quick checks, that difference adds up.

FIG 2Score against time per task
GPT-6.2 Astra is the fastest of the five models at every effort setting.

IV. Precise, but brief

The judges grade each answer on eight separate qualities, from factual accuracy to structure.

On evidence and citation, GPT-6.2 Astra scored 94.5, the best of the five models. When it makes a claim, it shows where the claim comes from, and it stays on the question.

On coverage, it scored 59.9, the lowest of the five. Its answers were roughly half the length of Claude Opus 5.5's. It named the drivers but did not explain how they work, and it skipped steps an analyst would expect.

FIG 3GPT-6.2 Astra against Claude Opus 5.5, quality by quality
GPT-6.2 AstraClaude Opus 5.5
Scores out of 100, effort high for both models. GPT-6.2 Astra leads on evidence and citation and trails most on factual accuracy, structure and coverage. Claude Opus 5.5 was run the week before.

Two examples show the pattern. On a filing fact-check, where there is one right answer, it gave the exact figures directly and did well. Asked to run a stock screen, it pointed to a published list instead of running the screen. On credit work, it missed checks an experienced analyst would make without being asked, such as stripping out one-off gains.

Our working explanation for why other benchmarks rank it higher: short questions with one right answer reward precision, while full analyst requests reward the extra steps. A model that is precise but brief would do well on the first kind and lose points on the second. We have not tested this directly.

V. The cost number is not final

We measured $3.62 per task at high effort, but the real cost is likely closer to $1.91, roughly the same as Claude Opus 5.5. The difference comes from caching, where repeated text is billed at a discount. Each request carried a fixed block of about 18,200 prompt tokens that was never cached, and that block made up nearly half the bill. This looks like a launch-week issue, and we will update the number once we can confirm it.

VI. What this means for you

  • Use it for quick, checkable lookups. Filing facts, a figure from a call, a definition. It is fast and it shows its sources.
  • Do not switch for in-depth research yet. For meeting prep, thesis checks and credit views, our results do not support moving to it from GPT-6 Astra or the Claude models.
  • Read its answers for what is missing. What it says is well sourced. The risk is what it leaves out.
  • Know which benchmark matches your work. A test of short questions and a test of real analyst requests can rank the same model differently. Both are useful if you know which kind of work you are buying the model for.

How we tested, and the limits

  • 18 real analyst tasks, the same set used earlier in this series. Each answer was graded out of 100 against a checklist.
  • Three AI judges: Claude Fable 5.1, GPT-6 Astra and Gemini 3.1 Pro. Because GPT-6 Astra is from the same family as the model tested, we recalculated the results without its scores. Every conclusion held.
  • GPT-6.2 Astra ran each task once at three effort settings: medium, high and the top setting. GPT-6 Astra ran once on the same day as a control.
  • One run per task means some noise, which is why we give the gap as a range.
  • This post uses a new round of runs at fixed effort settings, so its scores differ from those in Parts 2 and 3.
  • Claude results come from runs the week before, so we do not state exact gaps to them.
  • A few third-party data tools were left out of this round, for both GPT-6.2 Astra and the same-day control.
  • Cost is tokens used times each provider's list price. The GPT-6.2 Astra cost is provisional, as explained in section V.
  • These results describe our tasks and our setup. They do not show that GPT-6.2 Astra is weaker in general.
  • Contributors: Suyeol Yun and Guyoung Jung.