OCOPD Kimi K3 on 16x GB10: A Prompt Collection for Serious Cluster Benchmarking
Running a full Kimi K3 model on a 16x GB10 cluster at 20+ tokens per second average is the kind of result that gets infrastructure engineers to stop scrolling. The reported setup mentions dspark, a llama-benchy coherent corpus run, 38 tokens per second peak, and around 750 tokens per second prefill. More importantly, the author says they plan to test further, improve tensor-parallel speed, and publish a vLLM image with instructions when ready.
That is exactly the moment where good prompts matter.
Not fluffy prompts. Not generic "optimize my AI system" prompts. You need prompts that turn a messy first run into a reproducible engineering workflow: benchmark design, cluster diagnosis, vLLM deployment documentation, and performance reporting that other developers can actually trust.
This collection is built for developers working with large model inference, NVIDIA GB10-style clustered hardware, dspark-based experiments, llama-bench-style benchmarking, and vLLM serving images. It does not assume magic. It does not invent model variants. It helps you reason from the real measurements you have.
Concept & Theory
A first successful full-model run is not the same as a production-ready inference stack. It proves the path exists. It does not prove the path is efficient, stable, reproducible, or cost-effective.
For a large model like Kimi K3 running across 16 nodes or accelerators, the main performance questions usually fall into a few buckets:
- Prefill throughput: how fast the system processes input tokens before generation begins.
- Decode throughput: how fast it produces output tokens after prefill.
- Average versus peak tokens per second: whether the system is consistently fast or only fast in short bursts.
- Tensor parallel communication overhead: whether distributed execution is losing time to synchronization and interconnect latency.
- Memory pressure: whether KV cache, weights, and runtime buffers are forcing inefficient scheduling.
- Benchmark realism: whether the corpus and request shape reflect actual developer workloads.
The forum context gives useful early numbers: 20+ tps average on a llama-benchy coherent corpus, 38 tps peak, and 750 tps prefill. That is enough to build an investigation plan, but not enough to declare victory. A developer-friendly benchmark should include exact software versions, model identifiers, quantization or precision settings, batch sizes, context lengths, tensor parallel configuration, interconnect details, and the command lines used.
The prompts below are designed to help you create that discipline quickly.
The Prompts
Prompt 1: Benchmark Plan for Full Kimi K3 on a 16x GB10 Cluster
Use this when you have an initial run and need a rigorous test matrix before publishing numbers.
You are a senior LLM inference performance engineer. Help me design a reproducible benchmark plan for running the full Kimi K3 model on a 16x NVIDIA GB10 cluster.
Real observed context:
- Model: Kimi K3 full model
- Cluster: 16x GB10
- Runtime path mentioned: dspark
- Benchmark style: llama-benchy coherent corpus
- Current observed decode throughput: 20+ tokens/sec average
- Current observed peak decode throughput: 38 tokens/sec
- Current observed prefill throughput: about 750 tokens/sec
- Goal: test more, improve tensor-parallel speed, and later publish a vLLM image and instructions
Constraints:
- Do not invent hardware specs, model variants, or unpublished benchmark claims.
- Treat the current numbers as preliminary.
- Focus on practical tests developers can actually run.
- Separate prefill, decode, end-to-end latency, and stability metrics.
Create a benchmark plan with these sections:
1. Measurement goals
2. Minimum environment metadata to record
3. Benchmark matrix covering context length, output length, concurrency, and batch size
4. Metrics to collect for each run
5. Commands or command templates I should adapt for llama-bench-style testing and vLLM testing
6. How to compare dspark results with a future vLLM image fairly
7. Common sources of misleading tokens/sec numbers
8. A final reporting template suitable for a GitHub README or forum post
Be specific. Use tables where helpful. Explain why each metric matters. Keep the tone practical and skeptical, as if preparing results for other infrastructure engineers.Why this works
This prompt forces the model to separate preliminary excitement from benchmark reality. It also pushes for metadata capture, which is where many public LLM performance posts become impossible to reproduce.
Prompt 2: Tensor Parallel Bottleneck Diagnosis
Use this when average throughput is far below peak throughput, or when scaling across 16 devices looks weaker than expected.
You are an expert in distributed large language model inference. Analyze possible bottlenecks in a full Kimi K3 run on a 16x GB10 cluster.
Known observations:
- Full Kimi K3 is running across 16x GB10
- First run used dspark
- llama-benchy coherent corpus showed 20+ tokens/sec average decode throughput
- Peak decode throughput reached 38 tokens/sec
- Prefill throughput reached about 750 tokens/sec
- The next goal is to improve tensor-parallel speed and eventually publish a vLLM image with instructions
I need a diagnostic workflow, not a generic explanation.
Please provide:
1. A ranked list of likely bottleneck categories for distributed LLM inference at this scale
2. For each bottleneck, the symptoms I would expect in logs, metrics, or benchmark output
3. Specific measurements to collect, including GPU utilization, memory utilization, interconnect behavior, CPU load, token latency distribution, and scheduler behavior
4. How to distinguish compute-bound decode from communication-bound decode
5. How to interpret a large gap between prefill throughput and decode throughput
6. Safe configuration experiments to try before changing code
7. Risky experiments that should be isolated and documented carefully
8. A concise decision tree I can follow after each benchmark run
Important rules:
- Do not claim a specific cause without evidence.
- Do not invent GB10 interconnect details or unpublished Kimi K3 internals.
- Use cautious language and explain what each test would prove or disprove.
- Assume I want reproducible engineering notes that can later be shared publicly.Why this works
The phrase "try tp speed this up" points directly at tensor parallel performance. This prompt keeps the investigation evidence-based. It avoids the classic trap of blaming one layer, such as networking, without measurements.
Prompt 3: vLLM Image and Instructions Draft
Use this before publishing a container image or deployment guide. It helps turn a working local setup into something other developers can run.
You are a developer relations engineer and production inference specialist. Draft a clear, reproducible README for a future vLLM container image intended to run the full Kimi K3 model on a 16x GB10 cluster.
Current context:
- A first full Kimi K3 run has completed on a 16x GB10 cluster
- The initial run used dspark
- Observed preliminary performance: 20+ tokens/sec average decode, 38 tokens/sec peak decode, about 750 tokens/sec prefill on llama-benchy coherent corpus
- The vLLM image is not yet published, but instructions will be released when ready
Your job is to create a README draft with placeholders where exact values are still unknown.
Include these sections:
1. Project status and benchmark disclaimer
2. Supported hardware assumptions and what users must verify
3. Required software versions to document, including NVIDIA driver, CUDA, container runtime, Python, PyTorch, vLLM, and any distributed runtime dependencies
4. Model acquisition and licensing checklist
5. Container image usage template
6. Cluster launch command template with placeholders for tensor parallel size, hostfile, networking, ports, cache paths, and model path
7. Benchmark command template
8. Troubleshooting section for startup failure, NCCL or distributed communication issues, out-of-memory errors, slow prefill, slow decode, and inconsistent tokens/sec
9. Performance reporting template users can paste into GitHub issues
10. Security and operational notes for exposing an OpenAI-compatible vLLM endpoint
Rules:
- Do not fabricate an actual image name, registry URL, or final command if not provided.
- Use placeholders like <IMAGE_NAME>, <MODEL_PATH>, and <TP_SIZE>.
- Keep instructions concrete enough that I can replace placeholders later.
- Include warnings where benchmark claims are preliminary.
- Write in a style suitable for a public GitHub repository.Why this works
A container image without clear environment details becomes a support burden. This prompt creates a release-quality skeleton while avoiding fake certainty about unpublished image names or exact commands.
Prompt 4: Public Benchmark Report and Forum Update
Use this when you want to post follow-up results without overstating what the run proves.
You are an AI infrastructure writer with hands-on benchmarking experience. Help me write a public technical update about running the full Kimi K3 model on a 16x GB10 cluster.
Facts I can state:
- Full Kimi K3 has completed a first run on my 16x GB10 cluster
- The initial run used dspark
- On llama-benchy coherent corpus, I observed 20+ tokens/sec average decode throughput
- Peak decode throughput reached 38 tokens/sec
- Prefill throughput reached about 750 tokens/sec
- I am running more tests and trying to improve tensor-parallel speed
- I plan to publish a vLLM image and instructions when the setup looks ready
Write a technical post with the following structure:
1. Short summary of what was achieved
2. Exact benchmark numbers clearly labeled as preliminary
3. What is known about the setup
4. What is not yet known or not yet validated
5. Next tests planned
6. What I will include before publishing the vLLM image
7. Request for feedback from other developers running large model inference clusters
Tone requirements:
- Clear, humble, and technical
- No hype language
- No claims about production readiness
- No invented details
- Explain why average throughput, peak throughput, and prefill throughput should be interpreted separately
Output two versions:
- A concise forum post
- A longer GitHub discussion or README updateWhy this works
Public benchmark posts age well when they are precise. This prompt helps communicate progress while protecting credibility.
Prompt 5: Run Log Summarizer for Kimi K3 Cluster Experiments
Use this after each test run to turn raw notes into an engineering record.
You are a meticulous performance engineer. Convert my raw benchmark notes into a structured run log for Kimi K3 full-model inference on a 16x GB10 cluster.
Context to preserve:
- Model: Kimi K3 full model
- Hardware: 16x GB10 cluster
- Runtime under test: <RUNTIME_OR_IMAGE>
- Benchmark corpus or workload: <WORKLOAD_NAME>
- Current focus: improving tensor-parallel speed and preparing reproducible vLLM instructions
Raw notes:
<PASTE_RAW_NOTES_HERE>
Create a run log with:
1. Run ID and timestamp placeholder
2. Environment summary
3. Launch configuration
4. Workload configuration
5. Results table with prefill throughput, average decode throughput, peak decode throughput, latency, errors, and stability notes
6. Observed anomalies
7. Hypotheses supported by this run
8. Hypotheses rejected or not proven
9. Next experiment recommendation
10. Short changelog entry comparing this run to the previous run
Rules:
- If a value is missing, mark it as "not recorded" instead of guessing.
- Do not infer causality from correlation.
- Keep the output suitable for a benchmark-results.md file.Variables
Use these variables across the prompts to adapt them to your own Kimi K3 cluster tests:
<MODEL_PATH>: Local or mounted path to the Kimi K3 model files.<RUNTIME_OR_IMAGE>: dspark, a local runtime, or a future vLLM image name.<TP_SIZE>: Tensor parallel size used for the run, likely tied to the 16-device layout but still worth recording explicitly.<WORKLOAD_NAME>: llama-benchy coherent corpus or another benchmark corpus.<CONTEXT_LENGTH>: Input token length used during prefill measurement.<OUTPUT_LENGTH>: Number of generated tokens used during decode measurement.<CONCURRENCY>: Number of simultaneous requests.<BATCH_SIZE>: Batch size or scheduler-level batching configuration.<CUDA_VERSION>: CUDA version used in the test environment.<VLLM_VERSION>: Exact vLLM version or commit hash once testing moves to vLLM.<NVIDIA_DRIVER_VERSION>: Driver version on every cluster node.<RAW_NOTES>: Logs, command output, failures, and manual observations from a run.
A small but important habit: record variables even when they feel obvious. In distributed inference, the "obvious" setting is often what breaks reproducibility.
Expected Output
After using these prompts, you should have practical artifacts, not just advice:
- A benchmark matrix for full Kimi K3 cluster testing
- A tensor-parallel bottleneck diagnosis workflow
- A vLLM image README draft with placeholders
- A public benchmark update that avoids inflated claims
- A structured run log format for repeated experiments
For the specific 16x GB10 Kimi K3 context, the most useful result is a clean comparison trail. You want to see whether changes improve average decode throughput, peak decode throughput, prefill throughput, latency distribution, startup reliability, and operational simplicity. A single tokens-per-second number is never enough.
Here is a compact expected benchmark record format you can ask an AI assistant to maintain:
# Kimi K3 16x GB10 Benchmark Record
## Run
- Run ID: <RUN_ID>
- Date: <DATE>
- Runtime: <RUNTIME_OR_IMAGE>
- Model path: <MODEL_PATH>
- Tensor parallel size: <TP_SIZE>
- Workload: <WORKLOAD_NAME>
## Environment
- NVIDIA driver: <NVIDIA_DRIVER_VERSION>
- CUDA: <CUDA_VERSION>
- vLLM: <VLLM_VERSION_OR_NOT_APPLICABLE>
- Container image: <IMAGE_NAME_OR_NOT_APPLICABLE>
## Results
| Metric | Value | Notes |
|---|---:|---|
| Prefill throughput | <PREFILL_TPS> tokens/sec | <NOTES> |
| Average decode throughput | <AVG_DECODE_TPS> tokens/sec | <NOTES> |
| Peak decode throughput | <PEAK_DECODE_TPS> tokens/sec | <NOTES> |
| End-to-end latency | <LATENCY> | <NOTES> |
| Stability | <PASS_FAIL> | <NOTES> |
## Interpretation
- What improved:
- What regressed:
- What is still unknown:
- Next test:The deeper lesson is simple: early full-model success is valuable, but reproducibility is what makes it useful to the wider developer community. If you are preparing to publish a vLLM image and instructions for Kimi K3 on a 16x GB10 cluster, these prompts give you the scaffolding to make the release clearer, easier to validate, and much easier to improve.