AI Watch

Gemini 4 Hydrogen vs 3.8 Flash on a Real Coding Task: Similar Score, Half the Quota

By Lesia Kapliuk · Published · Updated

Updated Nov 15, 2026: we added a run of Hydrogen at Extra. It scored the same as High in almost twice the time and quota, and made the same title mistake, details.

TL;DR: We gave the same real coding task to Gemini 4 Hydrogen and Gemini 3.8 Flash in the Antigravity CLI: eight runs across all thinking levels, graded with one checklist. The two models scored about the same. They made different mistakes, though, and Hydrogen finished in about half the time (16 vs 32–33 minutes on Medium and High) with about half the quota.

  • That reverses our earlier short-puzzle test, where Flash stretched the shared quota 1.6 times further.
  • (Update, Nov 15, 2026) Hydrogen at Extra scored the same as High (34 of 46) but took 28 minutes and 11.6 pp of quota, against 16.5 minutes and 6.5 pp.
  • Claude Sonnet 5.5 in Antigravity used up the whole 5-hour Claude allowance of our Google AI Pro account in 11.5 minutes and stopped before finishing (our weekly Claude allowance was already mostly used).
  • The same model in Claude Code on a Claude Pro plan finished in 9 minutes, using 19% of its 5-hour session, though our settings there kept it from running the tests.

Results: Hydrogen vs 3.8 Flash scores, time and quota

The same real coding task, nine runs in the Antigravity CLI on Google AI Pro, graded out of 46 (how we tested).

RunScore (of 46)Hard checks: pass / partial / failTimeQuota used
Flash Low #1356 / 4 / 021 min~7 pp (estimated)
Flash High346 / 3 / 133 min9.3 pp
Hydrogen High346 / 3 / 116.5 min6.5 pp
Hydrogen Extra346 / 3 / 128 min11.6 pp
Hydrogen Medium33.56 / 3 / 116 min4.2 pp
Flash Low #2336 / 3 / 130 min10.6 pp
Hydrogen Low #2307 / 1 / 24.5 min1.8 pp
Flash Medium295 / 4 / 132 min8.3 pp
Hydrogen Low #1255 / 3 / 29 min~3 pp (estimated)

Google AI Pro, Antigravity CLI 1.5.8–1.5.9, November 13–15, 2026, one task. Hydrogen Extra was added on November 15. The two "#1" Low runs ran at the same time, so their quota is split by an estimate. Quota is the drop in the remaining 5-hour Gemini allowance, in percentage points.

Bar chart of nine runs on one coding task: Gemini 4 Hydrogen and 3.8 Flash scored 25–35 of 46; Hydrogen took 4.5–28 minutes vs 21–33 for Flash and used less quota except at Extra

On average Flash scored 32.75 and Hydrogen 30.6 (31.3 with the Extra run added (Update, Nov 15, 2026)). That is less than the spread inside one model: Flash ranged from 29 to 35, Hydrogen from 25 to 34. On this task we can't call either model the better coder.

Why we ran a real coding test

Our first test used 24 short tasks with one right answer, and every configuration got all 24 right. Readers of our Reddit post pushed back, and they had a point:

  • "try some harder task": short puzzles don't separate the models;
  • Hydrogen is faster "but dumber at coding", so Flash is still the better coder;
  • comparing one level against the same level on another model isn't fair, so try different levels.

The post also said we were staying on Flash until January because it used less of the shared quota. That was true for short single answers. Here is what happened on real agent work.

The task

A real feature in a production web app, from our own backlog, with a written brief of about two pages. The stack is Laravel on PHP 8.5, MySQL, Redis and ClickHouse, with a validation engine written in Rust. We are not naming the project; the shape of the task is what matters:

  • the admin panel has to accept a new version of a product-feed specification (a large JSON file) and stop accepting the old one;
  • for each field it must work out whether the field is required, required only under conditions (shown as readable text, not raw JSON), or optional, from several kinds of rules scattered across the file;
  • field names written as patterns (like "custom label 0 to 4") have to become real field names, and a pattern that can't be expanded must not crash anything or invent names;
  • an "Apply" button copies these levels into the site's own catalog and must never quietly weaken an existing requirement;
  • tests, docs and commits as the project's rules require.

It takes reading a long spec, finding rules that are easy to miss, and designing how to store and show conditions. Short puzzles don't need any of that.

How we tested

  • Same start for everyone: the same commit of the same repository, one fresh copy (git worktree) per run, the same brief, no hints during the run. Each agent wrote a report at the end.
  • Where: the Antigravity CLI (agy) in print mode on our paid Google AI Pro account, versions 1.5.8 and 1.5.9, November 13–14, 2026. One run at a time, except the first two Low runs, which ran in parallel.
  • Models and levels: Hydrogen at Low, Medium and High, 3.8 Flash at Low, Medium and High. We ran both Low levels twice: in the first round Flash Low came first in the table and Hydrogen Low last, which looked like outliers. The second Low runs landed much closer to the rest. (Update, Nov 15, 2026) On November 15 we added Hydrogen at Extra (xhigh), after a reader asked about it.
  • Grading: a checklist of 10 hard checks the agents never saw, each worth 4 points for a pass, 2 for a partial pass and 0 for a fail. Up to 6 bonus points went to work the brief asked for outside the checks (docs, updated existing tests, commits in logical steps) and to good practice it didn't ask for (a server-side guard, an honest report). Harm cost 2 points each: unrelated refactoring, or a test that passed before and fails after. That makes 46 points at most. The checks include running the app's admin pages against the real spec files, so a run can't pass by writing tests that agree with its own mistakes.
  • Who graded: Claude Opus 5.5 graded every run with the same checklist and the same scripted probes. A second, independent grading by Claude Opus 5.5 at High effort, on another machine and without our scores, mostly agreed. The one real disagreement, how to score the title mistake described below, we settled with the author of the task.
  • Quota: before and after each run we read the remaining Gemini allowance our account reported and waited until it stopped changing (the figure updates with a delay). The numbers are percentage points of the 5-hour Gemini allowance on Pro, not the weekly one, so they can't be compared directly with the weekly figures in our first test. The ratios can.

Coding mistakes: where Hydrogen and Flash went wrong

The scores are close, but the mistakes are different, and they repeat.

Neither model reliably read the rule levels. In the spec, title and description each have an "either this field or its structured twin" rule. The one for title is an error, so a title is always needed. The one for description is only a warning, so a description is recommended, not required. The two models handled this in opposite ways:

  • Flash treated both rules as hard requirements. It kept title required, which is right, but marked description as required too in three of its four runs. One Flash run even wrote in its own hint text that Google only flags a missing description with a warning, and still marked it required. Only the first Flash Low run noticed the difference and got both fields right.
  • Hydrogen never marked description as plainly required. Both Low runs made it optional. Medium and High showed it as "required if structured_description is missing", the same way they showed title.

Hydrogen weakened title. In the spec, either title or structured_title must be present for every product. Every Hydrogen run, Extra included (Update, Nov 15, 2026), showed title as "required if structured_title is missing". That is logically the same, but it hides that one of the two is always needed and puts title next to fields like color that depend on the product category. Worse, "Apply" then turned the catalog's required title into a conditional one, which the brief explicitly forbids. Flash kept title required in all four runs. Claude Sonnet 5.5 in Claude Code made the same title mistake as Hydrogen (Update, Nov 15, 2026).

Hydrogen High designed the trickiest part best. Only Hydrogen High built an "Apply" that gives the same result when clicked twice, and stored conditions in a shape the existing field editor can open and save without breaking them. Every other run, Flash included, stored conditions the editor can't handle.

Flash finished the job more carefully. It updated the docs and the existing tests in every run, and committed in logical steps in three of four. Hydrogen High did too, while Hydrogen Medium added only one line of docs, in the wrong file. Hydrogen Low did none of it in either run and broke one existing test without noticing; its second report said honestly that it had run only its own test file.

Our reading: the two models don't differ much in how well they code. They differ in their default when a rule is unclear: Flash makes the field required, Hydrogen makes it conditional. Each default happens to be right for one of the two fields. The most careful design came from Hydrogen High, and the most consistent work around the code from Flash. That comes from eight runs of one task, so treat it as an observation, not a rule.

Antigravity quota: Hydrogen used about half as much as Flash

Here the first test flips. On short puzzles Hydrogen took 1.6 times as much quota as Flash for the same work. On this agent task:

  • Medium: Hydrogen 4.2 pp, Flash 8.3 pp;
  • High: Hydrogen 6.5 pp, Flash 9.3 pp;
  • Low (second runs, measured one at a time): Hydrogen 1.8 pp, Flash 10.6 pp.
  • Extra (Hydrogen only) (Update, Nov 15, 2026): 11.6 pp, more than Flash at any level, for the same score as Hydrogen High.

Hydrogen was also about twice as fast at every level: 16 minutes on Medium and High, where Flash took 30 to 33.

What we think explains it: Flash read far more. Across a run, Flash's requests re-sent 17 to 26 million tokens of context (counted as cached input), against 2 to 8 million for Hydrogen, and Flash produced more output too. Over dozens of agent steps that outweighs Flash's lower price per token. On a single short answer there are no steps to add up, which is why the first test came out the other way.

So we take back our "staying on Flash until January" for agent work. It still holds for short single answers.

Do thinking levels matter on a coding task?

  • Hydrogen: a higher level cost more and gave more complete work. Low was the fastest and cheapest (1.8 pp, 4.5 minutes) but skipped the full test suite and the docs. Medium and High scored about the same; High cost about 50% more quota and was the only run with the best design for "Apply".
  • Hydrogen Extra (Update, Nov 15, 2026): we hadn't planned to run it. In our short-puzzle test Extra thought about 2.7 times as much as High and got nothing more right, so we expected more cost for about the same code. A reader asked, so we ran it once. It scored 34, the same as High, in 28 minutes instead of 16.5 and with 11.6 pp of quota instead of 6.5. The code was better in places and worse in others: its stored conditions are readable and survive the existing field editor, and it merged brand's own rule with the identifier rule correctly; but it pre-selected every "required → required if" change on the Apply screen with an "alternative available" badge, even for fields like color that have no alternative, and it made the same title mistake as every other Hydrogen run. More thinking did not fix it.
  • Flash: the level made no clear difference to either score or quota. Its two Low runs differed more from each other (35 and 33) than Low did from High.

Hydrogen and the agy sandbox on Windows

On Windows, Hydrogen stopped within two minutes in five attempts: inside the agy sandbox, PHP reported the project's files as unreadable, the agent asked to leave the sandbox, and agy's own policy refused. Flash never ran into this under the same settings. To get Hydrogen High and the second Hydrogen Low through, we ran them with agy's sandbox checks off (Hydrogen Extra too, from the start (Update, Nov 15, 2026)), with pushing to the remote disabled for the run. The time and quota figures in the table cover only the completed runs. The failed Hydrogen attempts we measured took about 2 pp more in total; Flash High also had one failed first attempt, for a different reason (a command it needed was refused in non-interactive mode), which took 1.6 pp.

Claude Sonnet 5.5 in Antigravity on Google AI Pro: out of quota in 11.5 minutes

For comparison we gave the same task to Claude Sonnet 5.5 at Medium effort.

Claude Sonnet 5.5 in AntigravityClaude Sonnet 5.5 in Claude Code
PlanGoogle AI ProClaude Pro
Resultstopped after 11.5 min, "Individual quota reached", no reportfinished in 9 min, 35 points (tests not run)
Quotawhole 5-hour Claude allowance: 100% → 0%19 pp of the 5-hour session (15% → 34%), 2 pp of the week

November 14, 2026. Antigravity CLI 1.5.9 on our account; Claude Code in headless mode on a second laptop with the same repository copy, brief and folder layout.

  • In Antigravity the Claude allowance is a separate pool from Gemini. On Pro, a single real task used all of the 5-hour part of it in 11.5 minutes and didn't finish. Our weekly Claude allowance in Antigravity was already mostly used that week, so the weekly limit may have played a part; the 5-hour figure went from full to zero during the run.
  • In Claude Code the same model finished the job and used under a fifth of the 5-hour session. Its 35 points tie the best Gemini runs. One caveat in its favor: our permission settings there didn't let it run the test suite, so it finished without testing, and we graded its code without penalizing its two failing tests.
  • These are two different agents around the same model, so this is not a model comparison. It does speak to a question that comes up often on Reddit: in our one run on Google AI Pro, Claude in Antigravity used up its 5-hour allowance before finishing a single real task.

What this means for you

  • For agent work in Antigravity, start with Hydrogen. In our runs it scored like Flash, finished in half the time and used about half the quota.
  • Check what Hydrogen does to "always required" rules. It turned "one of these two is required" into "required if the other is missing", in all five runs, Extra included (Update, Nov 15, 2026).
  • Check Flash on warning-level rules. It tended to make recommended fields required.
  • Don't expect Extra to fix it (Update, Nov 15, 2026). In our run it cost almost twice the time and quota of High for the same score.
  • On Low, tell Hydrogen to run the full test suite. Left alone, it didn't, and it broke a test it never ran.
  • For short single answers, our first test still holds: Flash used less of the shared pool, at least while its promotional price lasts (until December 31).
  • Don't plan real work around Claude in Antigravity on Pro. In our run, one task used all of its 5-hour allowance.

Limits of this test

  • One task in one codebase. A different kind of work may rank the models differently.
  • One run per configuration, two for the Low levels; one for Hydrogen Extra. The two Flash Low runs differed by 2 points and the two Hydrogen Low runs by 5, so small differences in the table are noise.
  • Graded by Claude Opus 5.5, with a second independent Opus grading as a cross-check, against a checklist the agents never saw.
  • The grader is a Claude model, and one graded run was Claude Sonnet 5.5; the checklist and scripted probes were the same for every run.
  • Three Hydrogen runs (High, the second Low and Extra) ran with agy's sandbox checks off, after five attempts stopped early inside the sandbox on Windows; all Flash runs used the default settings.
  • Quota figures are what our account reported; we can't see how Google counts them.

Update history

  • Nov 15, 2026: added a run of Hydrogen at Extra (34 points, 28 minutes, 11.6 pp) to the table, chart, thinking-level and quota sections; noted that Claude Sonnet 5.5 in Claude Code made the same title mistake as Hydrogen; explained why we repeated the Low runs.

Sources: our own runs on a paid Google AI Pro account, November 13–15, 2026: the Antigravity CLI 1.5.8 and 1.5.9 in print mode with JSON output (time and token counts as it reports them), the quota figures our account received from Antigravity before and after each run, and one Claude Code run on a Claude Pro plan with its /usage screens before and after. How we work: editorial policy.