Which Providers Can Host Kimi K3 for Long Documents and Handle KV-Cache and Runtime Tuning?

For long-document Kimi K3 workloads, we recommend a specialist inference provider that runs K3 on a dedicated, single-tenant endpoint and handles KV-cache and runtime tuning for you. GMI Cloud Prime Inference is built for that job: it lists Kimi K3 with 1M+ context, keeps weights loaded on reserved GPUs, applies per-model kernel, scheduling, and routing optimization, and optimizes the KV cache for long-context workloads, with our engineers working through the configuration with you. Moonshot's own API is a fast way to evaluate K3, Amazon Bedrock suits teams standardized on AWS, and self-hosting fits only teams ready to own KDA-aware caching, multi-node parallelism, and engine upgrades themselves.

The reason to lean on a managed specialist is that K3 is demanding to serve on long inputs. Its hybrid attention means a cached prefix has to restore both recurrent state and per-token KV, and roughly 1.56 TB of weights needs a large multi-GPU footprint. Whichever provider you shortlist, get written answers to five questions: how much context the endpoint serves and what happens past it, whether prefix caching works with K3's hybrid attention, how cache hits are billed and reported, which engine and hardware run the model, and how reasoning tokens are budgeted. These answers decide whether contracts, codebases, and research archives get predictable latency and cost.

At GMI Cloud, the path has two steps. Test Kimi K3 on our OpenAI-compatible API with your own document set; our developer page lists it as moonshotai/Kimi-K3. Then move steady traffic to a dedicated Prime Inference endpoint, where the runtime and KV cache are tuned to K3 and your prompt shape.

Five questions to ask any Kimi K3 host

Two traits of Kimi K3 sit behind every question below. Its attention is hybrid: 69 Kimi Delta Attention (KDA) layers keep a fixed-size recurrent state, while 24 Gated MLA full-attention layers still store per-token KV, so a cached prefix has to restore both, and generic prefix caching built for standard Transformers doesn't cover that. And it's large: a 2.8T-parameter Mixture-of-Experts model with about 104B active parameters per token, shipped as roughly 1.56 TB of MXFP4 weights. vLLM's launch guide names a single 8× B300 node as the smallest footprint, with 16× B200 also supported, and SGLang's recipes need at least two 8-GPU nodes on Hopper-class hardware. On GMI Cloud, NVIDIA Blackwell GPUs, including B200 and B300, are pre-order and coming soon, while H200 is available now.

1. What context length is served, and what happens past it?

The model's 1,048,576-token window is a ceiling, not an endpoint promise. Ask what the endpoint serves and how many long sessions it holds at once. SGLang's K3 guide separates the two: the maximum context setting only caps the longest request, while KV precision, parallelism, and GPU memory decide concurrency. Also ask whether over-length input is truncated or rejected; silent truncation drops the end of a contract or codebase.

2. Is prefix caching on, and how is state retained?

In vLLM, prefix caching for K3 is currently off by default while the hybrid-cache design evolves, so it has to be enabled explicitly. Even when it's on, KDA state is kept only at checkpoints: at the end of a prompt, at optional fixed intervals, and for shared prefixes that don't end on a boundary, only after a second hit. Where those checkpoints sit decides whether the tenth question about the same 400-page filing reuses cached state or pays for a full prefill, so retention should be tuned to your prompt shape.

3. How are cache hits billed and reported?

As of September 2026, Moonshot's own API applies context caching automatically and prices cache writes (with short and long TTL options) separately from cache hits. Whatever the structure, ask for cache-hit tokens in the usage data of every response. Without it, you can't confirm reuse or model cost per question.

4. Which engine, hardware, and recipe status?

Moonshot lists vLLM, SGLang, and TokenSpeed as recommended engines. vLLM has added hybrid prefix caching, prefill/decode disaggregation, KV offload, and speculative decoding for K3. SGLang's K3 recipes cover KDA-state prefix caching, prefill/decode disaggregation, and speculative decoding. The recipes are still maturing: as of late September 2026, SGLang marks only two B300 1×8 configurations as verified and hasn't re-run accuracy checks on any of them. B300 is a Blackwell GPU, which GMI Cloud lists as pre-order and coming soon. H200 is available now on GMI Cloud, and SGLang still marks its H200 recipes as in final verification. Ask for the engine version and topology, pin them for each evaluation so runtime changes don't distort the comparison, and re-check accuracy on your own documents.

5. How are reasoning tokens budgeted?

K3 always reasons and returns reasoning_content alongside its answer. Thinking effort is set with reasoning_effort (low, high, or max, with max as the default). On Moonshot's API, output defaults to 131,072 tokens and can be raised to 1,048,576. Because K3 generates its reasoning before every answer, output volume, and with it cost, depends on effort level as much as input length. Ask which effort levels and output limits your endpoint supports.

Structure long-document requests so the cache can help

Caching pays off only when requests reuse a prefix:

1. Put the stable content first. Place the document, retrieved files, and fixed instructions at the start of the prompt, and the question at the end. An early change invalidates everything after it.
2. Keep the prefix byte-identical. Don't insert timestamps, request IDs, or reordered file lists ahead of the document.
3. Batch questions per document. Send the questions about one document close together, so they land while its cached state is still retained.
4. Return full assistant messages. In multi-turn or tool-calling flows, K3 needs the complete prior assistant message, including reasoning_content and tool_calls, passed back unchanged.
5. Reject overflow instead of truncating. GMI's chat completions API truncates over-length prompts by default. Where the model supports it, set context_length_exceeded_behavior to error, and check each prompt's token count against the endpoint limit before sending.

Hosting options for Kimi K3 compared

Self-hosting puts KDA-aware caching, checkpoint policy, multi-node parallelism, and engine upgrades on your team; a dedicated managed endpoint gives single-tenant isolation without that operating load.

How GMI Cloud hosts Kimi K3

Test on the OpenAI-compatible API

Kimi K3 is listed in GMI Cloud's API model catalog as moonshotai/Kimi-K3. The OpenAI-compatible API uses the base URL https://api.gmi-serving.com/v1, and GET /v1/models returns the exact model identifiers available to your API key. Run your real documents here: your longest file, a typical file, and a fixed set of questions with the criteria you'll grade answers against.

Move steady long-document traffic to Prime Inference

When traffic is steady, Prime Inference runs Kimi K3 on dedicated single-tenant GPUs. K3 is listed on Prime with 1M+ context, among the open models Prime offers for one-click deployment.

• Weights stay loaded. Reserved GPUs stay warm with weights pre-loaded, so requests to a 1.56 TB model don't wait on a cold start.
• A runtime tuned to the model. Prime applies per-model kernel, scheduling, and routing optimization. vLLM, TensorRT-LLM, and SGLang are pre-tuned per GPU class, quantization is configurable, and the platform handles multi-GPU orchestration.
• Built for long context. The Prime page lists an optimized KV cache, bounded P95/P99 latency, and no shared-pool contention for long-context workloads. Write your own p95/p99 targets into the acceptance test and tune KV-cache use and concurrency against them.
• Memory-heavy GPU options. Prime hardware options include H200 and B300. H200 is available now. B300, with 288 GB of memory per GPU, is the GPU vLLM names for K3's smallest single-node footprint; it is part of NVIDIA Blackwell, which GMI Cloud lists as pre-order and coming soon, so treat a B300 deployment as a pre-order rather than capacity you can provision today. Whichever GPU you plan on, memory has to hold session state for your concurrent requests as well as the weights.
• Placement control. Endpoints can be region-pinned for first-token latency or region-locked for data residency.
• GPU-hour billing. Prime bills hourly per GPU with no minimum contract and no per-token markup. Sustained workloads can reserve capacity on a seasonal basis or annually at lower per-hour rates.

Turn the evaluation into a deployment specification

Record the engine version, checkpoint revision, maximum request length, peak concurrent sessions, and measured latency for the configuration you tested, and keep the prompt set and scoring criteria with those results. Prime lists K3 with 1M+ context, and GMI's inference platform includes KV-cache reuse, scheduling, and load planning; the specification pins down how context limits, overflow handling, prefix-cache retention, and reasoning effort apply to your prompt shape. H200 is available now; B300 is NVIDIA Blackwell, pre-order and coming soon. On Prime's GPU-hour billing, cache reuse improves economics by increasing useful work per GPU hour; it is not a separate per-token discount. Our engineers work through the specification with you before deployment.

What to bring to a Kimi K3 sizing conversation

• Your longest and median documents, with token counts
• Your questions and the pass/fail criteria you'll grade answers against
• Concurrent sessions and latency targets
• How much of each prompt is shared across requests
• Region and data-retention requirements

FAQ

Can we call Kimi K3 through the OpenAI SDK?

Yes. Use the GMI base URL above and the K3 identifier that GET /v1/models returns for your API key. K3's model guidance requires the complete prior assistant message, including reasoning_content and tool_calls, passed back unchanged in multi-turn and tool-calling flows.

Can we run a paid production trial before committing?

Yes. With no minimum contract, you can run production traffic on Prime Inference before reserving capacity, and qualified prospects can receive GPU-hour trial credits to validate K3 on their own documents. SLA terms depend on GPU series and deployment configuration and are set in your contract.

Run Kimi K3 on your own documents

Bring your longest documents and the questions you need answered. Our inference engineers will recommend a Kimi K3 endpoint configuration on Prime Inference, covering context, caching, hardware, and replicas. Talk to an engineer, or create an account to test K3 on the API today.