# When AI Writes Quantum Code, Who Checks the Work?

By Christian Perez, published 2026-11-04

Canonical: https://quantumdev.io/blog/when-ai-writes-quantum-code-who-checks-the-work/

What an ORNL audit of quantum code benchmarks teaches builders about AI-generated circuits.

> _Disclaimer: While this article covers a paper focused on Qiskit, AWS builders should know that the Qiskit-Braket provider lets you run Qiskit circuits on Amazon Braket simulators and quantum hardware._
>
> _These findings apply to your work as well._
>
> __
>
> _The final section provides more detail._

On one set of quantum computing tasks, a frontier large language model scored 0%. Then the researchers fixed the grader, left every line of the model's code untouched, and the same output scored 94.1%.

Here is what "fixed" means. The task was T1, 17 Quantum Approximate Optimization Algorithm (QAOA) problems in a new benchmark the researchers built, and the model was GPT-6 Astra. The grader was checking exact resource quantities that the prompts never asked for. The researchers removed only those checks, kept every correctness check, and rescored the saved outputs. A second topic, T6, had the same kind of problem, and the same fix moved Astra from 1.2% to 92.9%.

That result comes from a new preprint out of the Quantum Science Center at Oak Ridge National Laboratory, "Benchmarking the Natural Quantum Leap: Auditing LLM Evaluation for Quantum Code Generation" (Seefried et al., arXiv 2610.28678). The team ran 15 models across 5 established quantum code benchmarks with 530 tasks, and 10 models on QuantumBenchEval, a new 96-task benchmark of their own. Then they did something that a lot of leaderboards actually tend to skip. They audited the graders, their own included.

What I hope you get from all of this is that if you're picking an AI coding assistant for Qiskit or Braket work, or you're about to paste a model-generated circuit into your Amazon Braket project, this paper is a useful guide to what a score does and does not actually tell you about the LLM and how it behaves in the quantum ecosystem.

## Why is quantum code harder to grade?

When you grade normal code, a unit test asks a simple question: Did the function return the right value? Quantum code makes that question a little bit more complicated. A correct circuit can be written many ways. Two circuits can look nothing alike and still be the same operation up to a global phase. Outputs are often measurement distributions, so "right" can mean "statistically close enough."

The paper opens with two examples of the work involved: preparing a four-qubit W state, and decomposing a CNOT into iSWAP gates. Tasks like these actually require reasoning about circuit semantics, measurement statistics, and backend constraints on top of ordinary API usage.

Benchmark authors grade this in different ways. Some run unit tests, some ask another LLM to judge the answer, and some write custom scoring functions. The ORNL team ran every sweep through one harness and scored each with its own native evaluator.

![Table of six quantum code benchmarks with their task counts and how they grade: QiskitHumanEval Standard, 151 tasks, unit tests; QiskitHumanEval Hard, 151, unit tests on the same tasks without the supplied code skeleton; QuanBench-44, 44, unit tests; QuanBench-117, 117, unit tests; QCoder, 67, an LLM judge with no executable tests; QuantumBenchEval (new), 96, task assertions plus an LLM rubric for the VQE topic, T3.](https://quantumdev.io/blog/media/image-03f31244d56a5ef419c0b7bfc7fe522863f24608-1260x668-webp-960.webp)

Each model generated five completions per task from a shared prompt template, with no re-ranking, voting, or post-processing (Table 1 in Section 3 of the paper). Decoding settings were not identical across model providers: most models ran at a temperature of 0.8, and Claude 5.1 and Muse-Glimmer-30B ran at 1.0.

## The Standard suite is pretty much saturated

![Bar chart of QiskitHumanEval pass@1 for five models, from Table 2 of the paper. Every model scores far higher on Standard than on Hard: GPT-6 Astra about 98% against about 63%, Muse-Glimmer-30B about 95% against about 28%, and Claude Fable 5.1 about 70% against about 56%.](https://quantumdev.io/blog/media/image-0a991174a105e2d8f31cbcfede6d1c91ac2cddd5-1284x798-webp-960.webp)

On QiskitHumanEval Standard, GPT-6 Astra hit 98.7% pass@1, and Gemini 3.1 Pro Preview reached 97.6%.

> _pass@k means the model passed the benchmark assessment at "k" number of attempts, therefore pass@1 means the model passed on the first attempt._

Across five attempts, only one of the 151 Standard tasks went unsolved by every model. Standard is 151 short programming problems on core primitives such as state preparation, circuit construction, and measurement. The prompt supplies the import and a function skeleton. An answer counts as correct if and only if it passes the task's unit tests. The strongest models now solve nearly all of it (Astra reaches 100% pass@1 in 8 of 10 task categories), so the authors describe the suite as near saturation and say the remaining headroom lies in harder tasks such as the Hard variant and their new benchmark. In a task-by-task comparison with Holm correction, Astra's 1.1-point lead over Gemini 3.1 Pro has an adjusted p of 0.90, which means standard cannot tell the two apart, not that they are proven equal.

> _Holm correction is a way to avoid fooling yourself when you run several statistical tests at once. Each test has a small chance of showing a "significant" difference by pure luck. Run five tests, and the odds that at least one comes up by luck go up._
>
> _Holm Fixes that by raising the bar for each test. You sort your results from strongest to weakest. The strongest has to clear the strictest cutoff, and each one after it gets a slightly easier cutoff. Once one test fails, you stop and call the rest not significant too._
>
> _In the paper, the authors compared the top two models on five different test suites. After this correction, none of the gaps held up. That's why the article says the benchmarks can't really tell those top models apart._

For a builder, a high Standard score is fair evidence that a model handles short scaffolded Qiskit tasks, but it is not a reason to pick one top model over another, and it does not tell you how a model will do on harder work without the skeleton.

QiskitHumanEval Hard looks very different. The best pass@1 drops to 63.8%, and 31 of 151 tasks went unresolved by every model. All 15 models scored lower on Hard than on Standard. Hard uses the same task IDs but takes away the import and function skeleton that Standard supplies, so the authors say the drop reflects sensitivity to prompt format and scaffolding, not only difficulty, and they treat it as a benchmark-level constraint rather than a measured difficulty effect. Within Hard, Astra ranges from 20.0% on quantum information to 85.7% on state preparation and analysis, though the authors caution that small categories have wide intervals.

## First place is not really a clear win

On each of the five primary suites, the authors compared first and second place task by task. After Holm correction for all the model pairs compared within each suite, none of the five top-two gaps was statistically significant. That does not show the top two models are equal, instead it means these task sets cannot actually be reliably separated. On QiskitHumanEval Hard and QCoder, the undirected evidence did lean towards the leader (p = 0.016 and 0.003) before the correction removed it.

The reality is that these gaps are actually quite small. On QiskitHumanEval Standard, Astra led Gemini 3.1 Pro by 1.1 percentage points. On QuanBench-117, the recorded top two tied at 84.6%. On QuanBench-44, the leader's 95% interval ran from 45.9% to 73.6%. With a few dozen or perhaps a few hundred tasks, scores move that much from sampling alone.

> _"An observed first place, therefore, should not be read as evidence that the leading model is reliably better than the runner-up."_

Rankings also shift between suites. Most execution-scored suites agreed  fairly well, but QiskitHumanEval Standard and QCoder rankings agreed at only 0.54. Pick a different benchmark, and you can get a totally different order.

## When the grader is the bug

![Bar chart of GPT-6 Astra's pass@1 on QuantumBenchEval topics T1 (QAOA, 17 tasks) and T6, rescoring the same saved outputs: near 0% with the original grader and above 90% with the corrected grader on both.](https://quantumdev.io/blog/media/image-3e43e165ec194caf50eac5c3207801a6187cb384-1254x826-webp-960.webp)

Here, the researchers did something that not a lot of researchers tend to do. They turned the audit on their own benchmark first. In T1, the evaluator checked exact resource quantities the prompts never specified. In T6, it required a reproducibility value in a form the prompts never stated. Removing only those checks moved Astra to first place among the frontier models on T6, where it had been last. All 32 reference solutions passed under both versions of the grader, and no recorded sample flipped from pass to fail. One caveat from the authors: the rescoring was replayed on a different platform, and 5 of 1,600 samples came out differently. A third topic, T2, also checked undisclosed quantities, but removing those checks changed little, so they report it as recorded.

Then they audited the evaluator of QuanBench, an established benchmark. For QuanBench bench-117, they found 3 defects.

1. Test scripts ran without executing their unittest test cases, so a deliberately wrong answer could pass.
2. The reference solution ran in the same namespace as the model's code, so an answer could pass by inheriting the referenced function.
3. A missing dependency imported by the reference solution could make a valid answer fail before it ran.

The authors showed these mechanisms with controlled probes, and found one false pass in the recorded results. Claude Opus 4.6 was scored as passing QuanBench-117 Task 110 (build a CNOT gate from iSWAP gates), but the circuit it returns is not equivalent to CNOT up to global phase. They are, however, careful about scope. The original evaluator source was not captured at the time, so the probes do not prove that every saved QuanBench outcome came from the inspected version. Their position is that all historical QuanBench scores, passes and failures alike, are recorded outcomes that need to be replayed under an isolated evaluator before anyone reads them as capability estimates. That applies to every QuanBench number on this page.

They also looked at the 17 QuanBench-117 tasks that no model passed. For each one, the dominant recorded failure traces to an import in the benchmark's own reference or test code, from packages such as `qiskit_algorithms`, `qiskit_optimization`, `qiskit.primitives`, and `qiskit_machine_learning`. That does not mean every model answer on those tasks was correct, since some failed earlier for other reasons. It does mean the apparent shared ceiling was at least partly the benchmark's environment.

LLM judges have their own problem. On QCoder task, QPC004_A2, Two responses that were character-for-character identical and correct (three alternating CNOT gates implementing a swap) were judged as "PASS" in one call and "FAIL" in another by the same judge model. QCode results carry other caveats too. Its protocol typically shows each model its own earlier attempts, so the five samples are not independent. Its judge changed partway through the study, and the judges are Anthropic models, the same provider as Claude 5.1, which leads QCoder at 43.0% pass@1.

## Empty answers count as wrong answers

GPT-5 returned empty code for 316 of its 755 QiskitHumanEval Standard samples, or 41.9%, and each was scored as a failure. Of those 316, 315 stopped at exactly 2,048 output tokens, which the authors say is consistent with the output budget running out before any code was written. GPT-5's empty rate rose on suites with longer tasks, reaching 63.0% on QCoder.

So part of GPT-5's score reflects its token limit. On QCoder, GPT-6 Astra, Claude Fable 5.1, and Muse-Glimmer-30B produced no usable code at 2,048 tokens, so their runs used 32,768 tokens instead, a 16x difference. The authors say gaps between those earlier and later QCoder runs mix model capability with output budget and judge version. If you're reading a leaderboard, the output budget is absolutely part of the result.

Syntax is a separate issue. Gemini 3 Flash produced syntactically valid code for only 53% of its QuanBench-117 samples. All 2,650 audited non-empty GPT-6 Astra outputs parsed as Python, and a substantial share still failed the evaluation. Valid Python and correct quantum code are different checks.

## What should you check before relying on a quantum leaderboard?

The paper ends with six recommendations for benchmark builders. Here they are as questions you can ask any leaderboard or vendor claim.

![Table of six questions to ask of a quantum code leaderboard and why each matters. Do the reference solutions pass, and do the tests run? QuanBench-117's evaluator ran scripts without running their unittest cases. Is the model's code isolated from the reference code? On QuanBench-117 a shared namespace let an empty answer pass a probe. Does the grader check only what the prompt asked for? Undisclosed checks held GPT-6 Astra to 0% on QBE topic T1; removing them gave 94.1%. Are empty or invalid outputs reported? GPT-5's empty-output rate reached 63.0% on QCoder. Are confidence intervals and corrected comparisons shown? No top-two gap on the five primary suites was significant after Holm correction. Were the samples drawn independently? QCoder's sequential protocol means its pass@k is not the standard estimator.](https://quantumdev.io/blog/media/image-51956849fcf2369f82d441c6539ffcb1ddd12188-1232x826-webp-960.webp)

I think that there are two more questions that are worth adding. Which judge model and version graded it? QCoder's judge changed between runs, and no run was rejudged.

Is the suite saturated? A top score on QiskitHumanEval Standard no longer separates the strongest models.

## How do you check LLM-generated Qiskit yourself?

1. Check what the code does.

Code similarity is a weak signal: CodeBLEU correlated with pass/fail at r = 0.280 on QiskitHumanEval Standard. Compare unitaries with Qiskit's `operator.equiv` (equivalence up to global phase), statevectors for state prep, and distributions with a tolerance for sampled outputs.

1. Run it in a clean environment.

Some failures in the paper were imports, such as `UnitaryGate`from `qiskit.quantum_info`, which does not export it. Pin Qiskit and let import errors surface in CI.

1. Make sure your tests run

Feed your harness one deliberately wrong candidate and one empty one. Both should fail.

1. Spell out your contract

If a test checks quantity, return type, or format, put it in the prompt. T1 and T6 show what happens otherwise.

1. Sample more than once

Quantum-specialized and small open-weight models gained up to 38 points from pass@1 to pass@5 (Granite-Qiskit on recorded Quanbench-117); the strongest models gained about 6 at most. Choosing among samples still needs a real verifier.

1. Watch for truncation

If a response cuts off mid-block, raise the output budget before you blame the model.

## Where this leaves us

On short Qiskit API task, the strongest models are near the ceiling of QiskitHumanEval Standard. On QuantumBenchEval, results depend heavily on topic: GPT-6 Astra and Claude Fable 5.1 both reached 94.1% on corrected T1, yet no model exceeded 32% on T4. The authors also list their own limits: QuanBench outcomes await replay, decoding settings and output budgets differ across models, and the roster reflects one point in time.

The ORNL team says it is releasing its harness, per-sample outcomes, and audit artifacts, along with a public leaderboard called Quantum Arena, so new models can be evaluated under the same protocol. The paper does not give a URL for it.

What does your team use to verify AI-generated quantum code today?

## Why should you care?

A leaderboard score is a rough hint about a model, not a final answer on whether it's good, and sometimes the test itself is the thing that's broken. Tests that hand the model a starter template tend to go a lot better than ones where it starts from scratch. Before you pick a tool, try a few models on real tasks from your own project and see which one holds up. Whatever model you use, still read what it writes and run it against tests you trust, because code that looks right and code that works are not always the same thing.

As you continue to build on AWS, this lands pretty directly on Amazon Braket. Braket has its own Python SDK with its own way of writing circuits, and there's also a Qiskit-Braket provider that lets you run Qiskit circuits on Braket simulators and quantum hardware. Every benchmark in this paper is Qiskit-focused, and the paper did not test Braket at all. My read is that a model's Qiskit leaderboard score tells you very little about how well it writes Braket SDK code. If you're asking an assistant for Braket code, test it on Braket code.

The nice part is that Braket gives you inexpensive places to do that testing. The Braket SDK comes with a free local simulator that runs on your own machine, and there are managed simulators too, like SV1 for general circuits and DM1 for simulating noise. The managed simulators bill by how long each simulation runs. Real quantum hardware is built differently, with a fee per task plus a fee per shot, and the per-shot price depends on the device. Run AI-written circuits on the local simulator first, move to a managed simulator if you need more qubits or noise, and only spend on hardware once the code has passed your own checks.

Those checks are where the paper's lesson really pays off. "It ran without errors" is not the same as "it's right," and the paper found graders that passed wrong answers and failed correct ones. Write a free unit test that checks results you already know should come out, like a Bell pair giving you mostly 00 and 11, and run them on the simulator every time the model changes something. Feed your tests a wrong answer and an empty one once, just to make sure they actually fail.

If you use Braket's managed Jupyter notebooks or Hybrid Jobs for bigger experiments, keep the same habit: simulator first, real tests, then hardware.

I get it, the paper is all Qiskit. But I'd really encourage you to build on AWS with Amazon Braket and try a small version of these. Test yourself with the Braket SDK. Write a handful of Braket tasks, each with unit tests that check the result on the local simulator. Then give a model each task in two ways, once with a starter template and once without, the same idea as the paper's standard and hard prompts. Run each one several times instead of once, try a couple of different models, and before you trust any of the scores, feed your grader a known-wrong answer and an empty one to make sure both fail. This would be your own small experiment, not a copy of the paper's benchmark, and a handful of tasks won't crown a winner, but it will tell you a lot more about which assistant actually writes good Braket code for you than any Qiskit leaderboard will.

Article Link: [https://arxiv.org/html/2610.28678#abstract1](https://arxiv.org/html/2610.28678#abstract1)

This article was first published on [AWS Builder Center](https://builder.aws.com/content/3LcD8jriMyeRJ6qPDqsjp9eF3rT/when-ai-writes-quantum-code-who-checks-the-work) on 4 November 2026.
