How Can You Verify an API Platform Is Consistently Serving DeepSeek V4 Pro 0813 and Qwen3.8 Max 0902 in a Coding Comparison?

The date suffix is the version anchor each request in this comparison carries. deepseek-ai/DeepSeek-V4-Pro-0813 and Qwen/Qwen3.8-Max-0902 each name one dated release, and sending those full IDs in every request keeps a suffix-less ID, such as the deepseek-ai/DeepSeek-V4-Pro in our quickstart examples, out of the results. On GMI Cloud MaaS, both dated releases have their own cards in the model library and answer on the same OpenAI-compatible chat completions endpoint, so the two arms of a coding A/B share a client, a request format, and a usage report. Run a small fixed probe set against both models each day and log what each response reports, so a shift in results is traced to its cause before it mixes into the comparison.

A platform suits a controlled coding comparison of these two releases when it serves both exact dated IDs through one API, takes both arms through the same client, and reports usage by model name, and GMI MaaS meets all three conditions. When a probe flags a change, the request IDs, timestamps, and exact settings in your log give our team concrete material for tracing that request window on the serving side.

What drift changes in a coding A/B that runs for weeks

A comparison that accumulates results over weeks depends on the model behind each ID staying the same. If something moves partway through, results from before and after the change are no longer measurements of the same pair, and an average hides that. The cause is not always the model.

A daily probe set pinned to both dated IDs

Probes are a small, fixed set of coding prompts kept apart from the main comparison suite and sent to both IDs at the same hour every day. Twenty to forty items is a practical starting size, or 120 to 240 calls a day across both arms at three repeats each. Pick items a script can grade:

• A function to write against unit tests the harness runs locally.
• A seeded bug to fix, returned as a unified diff that must apply cleanly.
• A refactor with a strict output format, such as JSON listing the changed files.
• One or two long prompts built from real repository files, sized near the largest context your comparison uses.
• A short exact-answer task, such as a regex or a SQL query against a fixed schema.

Freeze the probe file and store its hash. Nobody tunes prompts against it, and a new probe set starts a new baseline.

The Serverless Endpoint documentation describes a temperature of 0 as yielding deterministic, repeatable outputs for testing and debugging, so probes run at temperature 0 even when the main comparison samples higher. Each arm gets its own settings map that sends the same numeric output cap under that model's parameter name. Probes leave any thinking or reasoning mode at each model's default instead of switching it between days. Set context_length_exceeded_behavior to error so a probe that outgrows the context fails visibly instead of being truncated.

import hashlib, json, time
from openai import OpenAI

# SDK retries off: each probe call sends exactly one request
client = OpenAI(base_url="https://api.gmi-serving.com/v1", api_key="<GMI_API_KEY>",
                max_retries=0, timeout=180)
ARMS = {  # same values, each model's own name for the output cap;
          # thinking/reasoning modes stay at each model's default
    "deepseek": {"model": "deepseek-ai/DeepSeek-V4-Pro-0813",
                 "temperature": 0, "top_p": 1, "max_tokens": 2048},
    "qwen": {"model": "Qwen/Qwen3.8-Max-0902",
             "temperature": 0, "top_p": 1, "max_completion_tokens": 2048},
}
EXTRA = {"context_length_exceeded_behavior": "error"}

def utc_now():
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())

def probe(arm, item, repeat):
    body = {**ARMS[arm], **EXTRA, "messages": item["messages"]}
    row = {"utc_day": time.strftime("%Y-%m-%d", time.gmtime()), "arm": arm,
           "probe": item["id"], "repeat": repeat, "started_utc": utc_now(),
           "request_body": json.dumps(body, sort_keys=True)}
    t0 = time.monotonic()
    try:
        r = client.chat.completions.create(
            messages=item["messages"], **ARMS[arm], extra_body=EXTRA)
    except Exception as e:
        return {**row, "status": "api_error", "ended_utc": utc_now(),
                "api_latency_s": round(time.monotonic() - t0, 2),
                "http_status": getattr(e, "status_code", None),
                "request_id": getattr(e, "request_id", None),
                "error": repr(e)}
    # API time stops here, before any grading runs
    row.update({"ended_utc": utc_now(),
                "api_latency_s": round(time.monotonic() - t0, 2)})
    text = r.choices[0].message.content or ""
    u = r.usage
    row.update({"returned_model": r.model, "response_id": r.id,
                "created": r.created, "prompt_tokens": u.prompt_tokens,
                "completion_tokens": u.completion_tokens,
                "total_tokens": u.total_tokens, "output": text,
                "output_sha": hashlib.sha256(text.encode()).hexdigest()})
    try:  # each check extracts the code block from the reply, then grades it
        return {**row, "status": "ok", "passed": bool(item["check"](text))}
    except Exception as e:  # a broken checker is not a wrong answer
        return {**row, "status": "check_error", "error": repr(e)}

Run each probe three times per arm and append every row, errors included, to a JSONL file. With SDK retries off, each probe call sends exactly one request. API errors and checker failures carry their own status, so pass counts come only from graded rows and error counts are tracked separately.

Fields to log from every response

Every chat completion returns model, id, and created, plus a usage object with prompt_tokens, completion_tokens, and total_tokens. The runner stores them with the exact request body and the output text. Add the harness commit, client SDK version, prompt template hash, and the repository commit the coding tasks came from.

Two fields carry most of the signal. The returned model value for each dated ID should read the same on every row, and a row where it differs is a hard flag. prompt_tokens for an unchanged probe should hold steady, because the same text through the same tokenizer counts the same. If it moves while the probe and template hashes have not, something on the input path changed, which is firmer evidence than any shift in output quality.

Give the probes their own API key. The Serverless Usage view filters by API key and model name, daily or hourly, so each dated ID's probe totals can be matched against the token sums in your log. With the view set to UTC to match the runner, a remaining gap is a reconciliation item, starting with calls made under the probe key outside the runner.

Keep A/B traffic on the chat completions route with explicit IDs. GMI Router is a separate endpoint that picks a best-fit model per request and reports its choice in routing metadata, which suits production cost control rather than a fixed two-model comparison.

Separating a behavior change from sampling noise

A change in outputs suggests drift without proving that the model changed. Its job is to stop results from being merged until the cause is known. Build the baseline from the first five to seven probe days and record, for each arm, the range of daily pass counts, the share of probes whose repeats return identical output hashes, the median completion length, and the median API latency.

As a worked threshold with assumed numbers, 30 probes run three times give 90 calls per arm per day. Suppose one arm's baseline days passed between 78 and 84. Counts are compared only once all 90 calls have been graded, so items lost to errors are rerun first. A day at 69 triggers a same-day rerun. A low day that the rerun does not reproduce is logged and left alone, while two days outside the range, or a rerun that confirms the drop, moves the comparison into the protocol below.

The re-run protocol after a flag

1. Freeze the comparison. Tag every comparison result after the last clean probe day as provisional, for both arms, without deleting anything.
2. Rule out your own side. Compare the harness commit, SDK version, template hash, and repository commit with the baseline rows.
3. Replay stored requests. Resend a set of stored baseline request bodies unchanged to both dated IDs and compare token counts and output hashes with the stored rows.
4. Localize it. A shift on one ID puts that model's serving path and its arm settings first in line. A shift on both puts shared settings, the harness, and the common serving path first.
5. Bring the evidence to us. Response IDs, timestamps, requested and returned model fields, and exact settings let our team look at a specific request window.
6. Rerun or split the window. Once the cause is known, rerun the provisional window on both arms. Results from before and after a confirmed change are reported as two comparisons and never averaged into one.
7. Re-baseline. Collect fresh baseline days before treating the probe ranges as valid again.

For a benchmark that must run unchanged over a long period, a Dedicated Endpoint is another option for models that support dedicated deployment: a private inference endpoint for a chosen model on reserved GPU capacity, listed in the console with its model, region, and replica count.

FAQ

Should the probes keep running after the comparison ends?

Keeping the same probe file running on the winning model turns it into a regression monitor for production. The baseline ranges already exist, so a flag on a production ID follows the same protocol.

What happens to the comparison when a newer dated release appears?

A new dated release has its own ID, so it enters as a third arm with its own baseline instead of replacing an existing arm mid-run. The original pair keeps its own probe history and stays under the same flag rules, and the new arm is scored on the same probe file and task suite from its first day.

Pin both dated IDs and run day one of the probes

Both dated cards are in the model library in the GMI Cloud console, where you can create a separate key for the probes. Run the first probe day, store the rows, and start the baseline before the main comparison collects results. For a long-running benchmark on reserved capacity, or to go through a flagged request window with us, contact our team.