How Should You Compare DeepSeek V4 Pro 0813 and Qwen3.8 Max 0902 by Cost per Solved Coding Task, Including Failed Attempts and Retries?

Cost per solved coding task is the inference spend on every attempt a model made, counting first tries, retries, and attempts on tasks it never solved, divided by the number of tasks whose hidden tests pass. Measured that way, DeepSeek V4 Pro 0813 and Qwen3.8 Max 0902 are compared on what a working fix costs rather than on what one call costs, and the two figures can rank the models differently. On GMI Cloud MaaS, both dated releases, deepseek-ai/DeepSeek-V4-Pro-0813 and Qwen/Qwen3.8-Max-0902, are served on the same OpenAI-compatible endpoint, and serverless usage can be read per API key and per model name. A per-attempt ledger built from your own logs can therefore be checked against billing for each model, the condition any platform has to meet before a cost comparison of these two releases is worth trusting.

Whether a cheaper call leads to a cheaper solution depends on attempts spent and tasks finally solved. Under feedback repair, each retry prompt carries the task plus the failed patch and its test output, so it sends more input than the first attempt, and whatever an abandoned task spent buys nothing. A sound comparison records each task's full trajectory, from the first attempt to a pass or to abandonment.

Tasks and the definition of solved

Cut tasks from repositories your team works in, pinned to a fixed commit so every attempt starts from the same tree. Each task is an issue description plus a failing test, and each has a second set of hidden tests that the model never sees. A task counts as solved when all four hold:

1. The patch applies cleanly to the pinned commit.
2. The project builds, and lint passes at the level your CI enforces.
3. The task's hidden tests pass.
4. Tests that passed before the patch still pass.

Hidden tests run after any attempt that passes the visible checks. A hidden pass ends the task. A hidden fail counts as a failed attempt, and the next retry gets only a fixed line saying the patch was not accepted, so the hidden tests stay hidden. Log the first visible pass as well, since a production loop without hidden tests would stop there and spend differently.

Aim for 100 tasks or more across the work you plan to hand to a model, such as bug fixes, small features, test writing, and behavior-preserving refactors, and report results by task kind as well as in total. Pin both dated IDs, keep sampling settings and starting output budgets equal across arms, and raise a budget after a cap hit by the same rule on both.

First attempt, retry, and abandonment rules

Write the loop down before the first run and apply it unchanged to both models:

• Maximum attempts per task (K). Record it, because the cap changes the answer, as the worked ledger shows.
• What a retry receives. Feedback repair sends the original task, the previous patch, and the visible test or build output, trimmed to the failing assertions and a bounded log tail. Blind resampling resends the original prompt.
• Context growth. Carry only the latest failed patch and its output into each retry, not the whole history. A repair prompt can outgrow the context window as the history builds up; with context_length_exceeded_behavior set to error, that attempt stops with an error rather than a silently shortened prompt, and the ledger records it as a failure.
• Abandonment. A task is abandoned after K failed attempts, or earlier if a per-task token budget runs out or a retry repeats the previous patch. Its spend stays in the numerator.
• Infrastructure errors. Timeouts, 429 responses, and server errors are logged as their own type and retried with backoff outside the K attempts, under a separate cap per task. A task that reaches it is reported as an infrastructure stop. Any tokens billed for these calls stay in the numerator.

The ledger behind the number

Every attempt gets its own ledger row: task ID, attempt number, model ID, the prompt_tokens and completion_tokens the response reports in usage, elapsed wall-clock time, the outcome, and the failure type. At list input and output rates, an attempt costs its prompt tokens times the input rate plus its completion tokens times the output rate. The pricing documentation places current rates on each model card in the console, so store the day's rates in the evaluation config. Both models also price cache reads, so the headline numerator is each model's billed total for the evaluation key, and per-attempt and per-task figures remain ledger estimates at list rates.

cost per solved task = sum of attempt costs across all tasks / number of tasks solved

The denominator counts tasks, not attempts, and only solved ones. Report alongside it the solve rate, the first-attempt solve rate, the cost per attempt, and the share of spend that went to abandoned tasks. A model with no solves in a task kind has its spend reported and its ratio marked undefined.

Run the evaluation under its own API key. The Serverless Usage view filters by API key and model name, daily or hourly, so each model's token usage for the evaluation window can be matched against the sum of your ledger rows. Match the view's time zone and window to the ledger first. A gap that remains is a reconciliation item, such as attempts the harness missed.

A worked ledger with assumed numbers

Every figure in this section is an assumed input chosen to show the arithmetic. None of it is a measurement of either model, and none of it is a GMI price. Model A and Model B each get 100 tasks with K = 3 and feedback repair. Prompts are assumed at 20K tokens on the first attempt, 26K on the second, and 32K on the third. Model A writes 3K output tokens per attempt and Model B writes 2.5K. Costs are in abstract units, with Model A at 1 unit per 1K input tokens and 4 units per 1K output tokens, and Model B at 1.6 times both rates, so every call to Model B costs more.

Model A is cheaper on every call. At K = 1 it is also cheaper per solved task, and at K = 2 and K = 3 it is not. Most of its spend goes to the 38 tasks it never solves, each of which uses all three attempts at the largest prompt sizes. The third attempt costs Model A 1,892 units for 5 extra solves, about 378 per solve, and Model B 1,613 units for 4, about 403 per solve, several times the average cost of a solved task. What carries over to your own data is that the ranking depends on solve rates per attempt and on K, and the rate card alone cannot settle it.

Quality and time next to cost

Cost per solved task treats every passing patch as equal. Review a sample of solved patches from each model for size, readability, and edits outside the task's scope, and record how many a reviewer would merge as written. Passing patches that need rework carry a cost the ledger does not show.

Sum wall-clock time across attempts and test runs for each solved task, and report the median and the slow tail, since a model that usually solves on the first attempt can finish sooner even when each call takes longer.

Failure types and the retry budget

Tag each failed attempt with one type:

The like-for-like comparison runs both models at the shared K. A second, production view sets a cap per model from the marginal numbers: stop adding attempts once the cost of the next attempt divided by the extra solves it buys exceeds what a solved task is worth to you. When failures cluster in one type, fix the prompt or the task spec before raising K, since a larger K alone repeats the conditions behind it.

FAQ

How many tasks does a cost-per-solved gap need before it means something?

Bootstrap by task. Resample task IDs with replacement a few thousand times, keeping each task's results for both models together, and compute the difference in ledger-estimated cost per solved task on each sample, reporting how many samples leave either model with no solves. When the 95% interval for that difference spans zero, add tasks in a batch sized in advance before choosing.

How is a cascade that tries one model first and escalates to the other scored?

A cascade is a third arm with its own ledger. Attempts on the first model and escalated attempts on the second both go in the numerator, and each solved task is counted once in the denominator. That ratio decides the comparison, while the escalation rate and the second model's solve rate on escalated tasks explain where the cascade lands.

Next step: run the ledger on your own repositories

Create an evaluation key in the GMI Cloud console, run the first 20 tasks with K = 3 to confirm the ledger reconciles with the usage view, then run the full set. When the winning model heads toward production volume, talk to our team about volume pricing and custom commitments for that model.