Benchmarking OpenJet Engine on an RTX 3090

The test setup

The engine README reports back-to-back runs on an otherwise idle NVIDIA RTX 3090 with Qwen3.8-27B Q4_K_M. These are single-conversation measurements. The mixed-quant UD-Q4_K_XL engine is outside this comparison.

llama-server runs with --spec-type draft-mtp --spec-draft-n-max 2 --flash-attn on -ctk q8_0 -ctv q8_0 -ngl 99 --parallel 1. OpenJet uses the bundled draft_vocab_32k.bin. Draft depth and KV format differ between the tested configurations; this is a comparison of the project's runtime settings. The README reports that using q8_0 KV instead of fp16 slows OpenJet decode by roughly 1–2% and prefill by 2–7%.

Decode throughput measures output token generation. Prompt throughput, or prefill, measures processing the input. Neither is total request latency: loading the model, processing the prompt and transporting the response are separate costs. The scripts read the server's reported timings or the engine CLI's timing output, rather than dividing output tokens by the entire HTTP request duration.

Reasoning: the everyday coding preset

The thinking benchmark uses thinking enabled with xhigh effort, temperature 1.0, top_p 0.95, top_k 20 and min_p 0. It runs five coding prompts with two seeds, 1000 and 1001, requesting up to 1,024 generated tokens per run.

The tasks cover merging intervals with pytest tests, finding the largest files with Bash, implementing a C ring buffer, explaining mutexes and semaphores, and finding a fragile spot in a pasted CUDA file. The last prompt includes the first 8,000 characters of prefill.cuh. The script renders the same system and user prompt for both engines, including the thinking prefix.

The script averages each prompt's throughput over its seeds, then calculates the overall rate as total generated tokens / total decode time. This avoids giving a short response the same weight as a long response. It also prints draft acceptance and tokens per speculative cycle.

The HTTP headline comes from the README's /v1/chat/completions measurements. The checked-in think_bench.py instead calls llama-server's /completion endpoint and compares it with the OpenJet CLI. It reproduces the second row's setup, rather than providing an automated reproduction of the two-server HTTP test.

Greedy decoding: speed depends on the prompt

The code benchmark uses the four prompts in prompts.txt, with thinking disabled and greedy sampling. Each short prompt requests 512 generated tokens. The default is two repetitions, retaining the fastest decode rate for each prompt and engine, then taking the arithmetic mean of those four rates.

Four drafts deliver 1.93× the baseline on the Python prompt and 1.69× on the mean. They do not win on every prompt: the hash map explanation is slightly faster with three. Predictable continuations allow more drafts to be accepted, so a code-generation maximum should not be presented as a universal rate.

Without speculation, the README reports 40.4 tok/s for llama.cpp and 42.9 tok/s for OpenJet with -d 0. The main benefit comes from speculative verification: OpenJet verifies the current token plus drafts in a persistent CUDA cycle while reusing weight reads. Its 32,768-token draft vocabulary reduces drafting work; verification still uses the full vocabulary.

Longer prompts and conversation reuse

For the context sweep, code_bench.py concatenates repository CUDA source files, truncates them using an approximate characters-per-token ratio, and appends a C++ unit-test task. The measured token counts therefore differ from the requested sizes. It requests 256 generated tokens and records prompt processing and decode separately. The script retains the best prompt and decode rates independently, so a row need not describe one particular run.

These are the README's published context results. It notes that they predate sequence-token additions to the draft vocabulary and double-buffered attention, which subsequently added 1–8% to OpenJet decode in this workload. The table preserves the measured figures without applying a projected uplift. The condensed README table does not label a draft depth for this sweep; the script prints both three- and four-draft results.

The second-turn test first asks for a C string-reversal function, then appends a source file and asks what its first kernel does. llama.cpp reuses its prompt cache; OpenJet appends with batched prefill. The default script selects the lowest prompt processing time over two repetitions.

These times cover processing the new prompt, not the complete second response. They show why prefill and conversation reuse matter when a coding agent pastes a file or receives a large tool result.

Correctness alongside speed

Different arithmetic ordering and rounding can change the winning token when probabilities are close. A generated answer can then diverge even when the next-token distributions are very similar. The project therefore also compares teacher-forced distributions against llama.cpp generating one token at a time with a q8_0 KV cache, using the same GGUF.

The short-context evaluation scores 13,299 positions of repository code and prose after 1,024-token prompts. The reference perplexity is 2.4263. Lower KL divergence indicates distributions closer to the reference.

At 30K context, OpenJet with q8_0 KV reports 99.65% agreement and KL 0.0007; at 96K, 99.60% and 0.0008. Each long-context test scores 1,999 positions from a single repository-code sequence. These checks support numerical similarity for the tested inputs; they are not a coding-quality evaluation or proof of identical output on every task.

The reference tool stores the 256 most likely tokens per scored position. The README also reports a 2,000-sample test per position for speculative sampling. Greedy verification follows the main model's choice, and sampled verification preserves its distribution; changing the verification batch size can still change floating-point rounding on near-ties.

Run the benchmarks yourself

Follow the engine installation guide for Linux, CUDA and the llama.cpp build. The documented setup uses llama.cpp tag b10246; the published results do not supply a separate exact llama.cpp commit for every run. Place the supported original GGUF at models/Qwen3.8-27B-Q4_K_M.gguf in the repository checkout. From the repository root:

cd megakernel
make

# Reasoning: five prompts, two seeds, up to 1024 tokens.
# Runs llama.cpp, then OpenJet with three and four drafts.
python3 bench/think_bench.py --seeds 2 -n 1024

# Greedy prompts, context sweep and second-turn processing.
python3 bench/code_bench.py --reps 2 -n 256 \
  --ctx-sizes 0,2000,4000,8000,16000,30000

# Separate decode baseline with speculation disabled.
python3 bench/bench.py llama --spec 0 -n 512
python3 bench/bench.py mk --draft 0 -n 512

At the linked revision, think_bench.py hardcodes the original model path and executable; it has no --model or --mk-bin options. code_bench.py and bench.py do accept those options. Keep the working directory and model path consistent with the commands above. The scripts launch llama-server themselves; run them sequentially with the GPU otherwise idle.

For the distribution comparison, supply local code and prose text files and run:

make tools/llama_ref
tools/llama_ref -m ../models/Qwen3.8-27B-Q4_K_M.gguf \
  -f /absolute/path/code.txt -f /absolute/path/prose.txt \
  --score-from 1024 -b 1 --ref-out llama.ref
tools/llama_ref -m ../models/Qwen3.8-27B-Q4_K_M.gguf --kld llama.ref
./qwen38_mk -m ../models/Qwen3.8-27B-Q4_K_M.gguf \
  --kld llama.ref --score-path mixed --kv q8

mixed uses batched prefill for the unscored prompt and the decode kernel for the scored continuation. The corpus and its exact contents affect the result; using your own files reproduces the procedure, not necessarily the published numbers.

What the results establish

For this RTX 3090, model and configuration, the published results show higher decode and prefill throughput with OpenJet Engine, including a 1.53× reasoning improvement with speculative decoding enabled on both sides. The greedy mean, the best individual prompt and the reasoning aggregate measure different workloads and use different aggregation rules.

The repository does not provide a complete run manifest covering GPU clocks, power limits, driver version, model checksum, exact benchmark dates and every engine revision. It also does not publish confidence intervals for the speed tables. Best-of-two greedy results describe favorable observed throughput. Other GPUs, quantizations, concurrent requests and end-to-end agent tasks need their own measurements.

Read the pinned results and correctness notes, inspect the benchmark scripts, or continue to the engine documentation and OpenJet vs llama.cpp comparison.