How Can You Run a Fair Trial of DeepSeek V4.1 Flash on a Token API and a Dedicated Endpoint Using the Same Traffic Trace?

A token API and a dedicated endpoint can be compared fairly only when three things match on both sides: the same DeepSeek V4.1 Flash revision, the same numeric precision, and the same request arrival times. On top of those, the trial pins reasoning, sampling, and output-length settings and replays one recorded trace against both paths, so gaps in time to first token, tail latency, errors, and cost per successful request can be traced to the serving path. GMI Cloud offers both serving paths the trial compares. DeepSeek V4.1 Flash is served per token through the GMI serverless API as deepseek-ai/DeepSeek-V4.1-Flash, and Prime Inference deploys open weights like these on reserved single-tenant GPUs with a runtime our engineers tune for the model, so the dedicated arm loads the checkpoint you pin for the trial.

GMI lets teams move between serverless and dedicated deployments without changing APIs, so one replay client drives both arms, with the base URL, key, and model field set per arm. Serverless and dedicated usage appear as separate reports in the same billing dashboard, so both arms' costs come from one source.

Pin the revision, precision, and request parameters

Write down every setting that can make the two arms serve different models in practice.

Set max_tokens and the sampling options the serverless endpoint exposes on every request, so neither arm falls back to a default. Prime Inference quantization is configurable, and our engineers set the dedicated arm's precision with you during setup.

Before any timing run, pass an equivalence gate that tests whether the arms behave alike on your tasks. Send 200 to 500 fixed prompts at temperature 0 to both arms and score the outputs with your own evaluation, exact match for structured tasks and rubric scoring for open text. Batched GPU inference is not always bit-for-bit deterministic, so set the tolerance in advance from repeated runs of one arm plus the margin your task can absorb, and compare output lengths as well as scores. Until the gap falls inside that tolerance, equivalence is not established and timing runs wait.

Capture and scrub the traffic trace

The trace needs enough detail to reproduce load without carrying sensitive content. Keep these fields per request:

• Arrival offset in milliseconds from the start of the window
• Input token count, max_tokens, and the output tokens actually produced
• Stream flag and request class, such as chat, extraction, or agent step
• A prefix-group ID marking requests that share a system prompt or document

Pick a window of two to four hours that includes both a real peak and a quiet stretch. When you scrub the content, replace text with length-matched filler measured in tokens with the model's own tokenizer, not in characters. Requests that shared a prefix in production should share a new synthetic prefix of the same length, since unique random text erases prefix-cache behavior on both arms. The replay keeps arrival times, input lengths, and prefix structure, but output length follows the new answers, so give each request the same max_tokens on both arms, log each response's finish reason and output tokens, reasoning included, and compare the arms' length distributions after each run. Save the scrubbed trace as a fixed file with a recorded hash.

Replay with the original arrival times

Use an open-loop replayer that sends each request at its recorded offset whether or not earlier requests have finished. A closed-loop client with fixed concurrency slows down with the server and hides the queueing a peak creates.

1. Run the replayer from one fixed client host near your users, and region-pin the dedicated arm close to it. Both arms are then measured as that client sees them, network path included.
2. Apply one cache rule to every run on both arms: the same short warm-up set, sharing no prefix with the trace, sent just before the run and discarded, after the same idle gap since the previous run.
3. Alternate the arms in an A-B-B-A order across the day so time-of-day effects fall evenly on both.
4. Hold the dedicated replica count fixed for the first runs. A later run with autoscaling enabled shows how burst capacity follows the trace; tag requests that arrive while a replica is being added and report them apart from steady-state latency.
5. Replay at 1x speed, then at 1.5x or 2x, to see how much headroom each arm has before the tail breaks.
6. Use identical client timeouts and turn off client retries, so every failure is counted once.
7. Log scheduled and actual send time per request. If send lag at the peak grows past a threshold set before the run, the replay did not hold the trace's timing and the run does not count; client connections and CPU are the first places to look. After the last arrival, let in-flight requests finish or time out and keep them in the results.

Measure TTFT and tail latency where the load is

One overall p95 hides what happens in the peak minutes. Report each metric by input-length bucket and by time slice so the peak shows up on its own.

Count tokens, not stream chunks, since one chunk can carry several tokens. Under 2K, 2K to 16K, and over 16K input tokens is a workable first split for chat traffic. Cut TTFT by prefix group too: a gap concentrated in shared-prefix groups suggests caching, but compare it across length buckets and time slices before attributing it. Shared serverless capacity varies with platform load, while other tenants' traffic does not reach a single-tenant endpoint, so compare the peak slices side by side rather than the daily aggregate.

Cost per successful request on each arm

Compute cost the same way on both arms: what the window cost, divided by the requests that succeeded. Whatever failed requests were billed stays in the numerator, and they leave the denominator. Price both arms at the rates you would pay in production, before any credits. On both arms, the window runs from warm-up to the last in-flight request.

• Token API arm. From Serverless Usage, take the trial key's usage for the window in each billed token category, price each category at its own rate, sum them, and divide by successful requests.
• Dedicated arm. Run the trial on its own deployment and sum its GPU-hours over the window, GPUs per replica times the replicas running in each interval, at your GPU-hour rate. A dedicated deployment is billed only while it is in Running status, so warm-up time counts and Queued or Deploying time does not.

Give each token-arm run its own API key, so the usage dashboards can be filtered to that run by key, model, and hourly range, and give each dedicated run its own deployment, so its GPU-hours come from that deployment alone. The dedicated figure reflects this trace's utilization on the replicas you held, so keep monthly projections out of the trial result.

Reproduce the result and size the difference

Run each arm at least three times at different times of day and report the median and range for every metric. Because A-B-B-A pairs runs taken close together, compute the difference within each pair and judge it against a margin your product cares about, such as a TTFT gap users would notice. Three pairs is a minimum. Record the configuration next to the numbers so anyone can rerun it: GPU type, GPUs per replica, replica count, region, engine, quantization, trace hash, and replayer version. Run the first dedicated arm on H200, which is available now on GMI Cloud.

FAQ

How many requests does the trace need for a stable p99?

Each reporting cell, meaning one length bucket in one time slice, needs enough requests that the slowest 1% is more than a handful of samples. As a planning figure, a few thousand requests per cell puts a few dozen samples above p99; with only a few hundred, report p95 and treat p99 as indicative. Bursty traffic is correlated in time, so report p99 alongside how much it moves between repeated runs.

Should the dedicated arm wait for B300?

Run it on H200 now. B300 is a Prime Inference option in pre-order, and the scrubbed trace, equivalence set, and replayer all carry over, so the same protocol yields a comparable B300 result once that capacity is scheduled.

Set up both arms of the trial

Create the dedicated arm from the console by following the Dedicated Endpoint guide, and point the token arm at deepseek-ai/DeepSeek-V4.1-Flash on the serverless API. For checkpoint selection, precision settings, and runtime tuning on the dedicated arm, contact GMI sales and an inference engineer will work through the setup with you.