Benchmarking OpenJet on an RTX 3090

How OpenJet and llama.cpp were compared on Qwen3.8-27B Q4_K_M on an RTX 3090: 2.0x decode with reasoning, 2.6x at 98K context, prompt reading and correctness.

OpenJet decodes Qwen3.8-27B at 121 tokens per second with thinking on and 96 tokens per second with 98K tokens in context on an RTX 3090. Against llama.cpp with MTP speculative decoding enabled, that is 2.0× and 2.6×. It reads a 98K-token prompt in 76 seconds, against 112 for llama.cpp.

This post explains what was measured, how the harness calculates its results, and how to run the comparison yourself. Sources are pinned to OpenJet revision 8b5aa29.

The test setup

For an explanation of the implementation behind these measurements, read Inside OpenJet's engine: making speculative decoding pay: weight reuse, on-device verification, recurrent-state replay and tensor-core prefill.

Both engines served the same unsloth Qwen3.8-27B Q4_K_M GGUF on one RTX 3090, one conversation at a time, through the same benchmark harness, `engine_compare.py`. It starts each server, sends identical OpenAI chat requests, and runs the engines in interleaved rounds so that background load and clock drift fall on both.

Settingllama.cppOpenJet
Versionb11982 llama-serverEngine built from this source by the project's CI
SpeculationMTP, at most 2 draftsMTP, up to 7 drafts, verified on the GPU
KV cacheq8_0, flash attention onq8_0
Context capacity106,496 tokens106,496 tokens
ConcurrencyOne server slotOne conversation
Measured29 October 2026, 4 samples per prompt17 November 2026, 8 samples per prompt

llama-server runs with --spec-type draft-mtp --spec-draft-n-max 2 --flash-attn on -ctk q8_0 -ctv q8_0 -ngl 99 --parallel 1 --jinja. OpenJet's engine runs as OpenJet starts it: --kv q8 --exact-reuse -d 7.

Every request uses thinking at xhigh effort, temperature 1.0, top_p 0.95, top_k 20, and generates 1,024 tokens. Decode throughput is generated tokens divided by the time between the first and last streamed chunk. Prompt read time is the time to the first token of a request whose prompt isn't cached.

The two runs were on different days. Between them, one of the five short prompts (a pasted source file, 3,088 tokens against 3,161) and the text of the 98K-token prompt changed; the other four prompts are identical.

Short prompts

Five coding prompts: merging intervals with pytest tests, finding the largest files with Bash, a C ring buffer, explaining mutexes and semaphores, and finding a fragile spot in a pasted CUDA file.

Promptllama.cpp (tok/s)OpenJet (tok/s)Speedup
Python function with tests66.2144.82.19×
Bash script58.1112.41.93×
C ring buffer55.7102.51.84×
Mutex and semaphore explanation61.6122.11.98×
Bug in a 3K-token CUDA file61.8124.52.02×
All five60.7121.22.0×

The all-prompt row is the mean over every request. Speed depends on how predictable the text is: drafts are accepted more often in code with repeated structure than in prose.

The OpenJet run went in four rounds. Its short-prompt decode averaged 135 and 139 tok/s in the first two and about 106 in the last two. Every engine in that session slowed in the last round, which points to other load on the GPU; the third is unexplained. The table includes all four rounds.

Long context

One prompt of about 98,000 tokens of source code, with the same reply settings.

Measurementllama.cppOpenJetSpeedup
Decode with 98K tokens in context36.6 tok/s95.9 tok/s2.6×
Reading the 98K-token prompt112 s (876 tok/s)76 s (1,288 tok/s)1.5×
Reading the 3K-token promptabout 960 tok/sabout 1,640 tok/s1.7×

Correctness alongside speed

Different arithmetic ordering and rounding can change the winning token when probabilities are close. A generated answer can then diverge even when the next-token distributions are very similar. The project therefore compares teacher-forced next-token distributions against llama.cpp generating one token at a time with a q8_0 KV cache, using the same GGUF. llama.cpp's own batched mode, against the same reference, shows how much two correct implementations differ.

Compared with the referenceTop-token agreementMean KL (nats)Perplexity
13,299 positions of code and prose, scored after 1,024 tokens
llama.cpp, batched98.25%0.002253.1529
OpenJet98.74%0.001623.1504
1,980 positions after a 94K-token prompt
llama.cpp, batched99.49%0.00111
OpenJet99.70%0.00093

The reference perplexity is 3.1528. These checks support numerical similarity for the tested inputs; they are not a coding-quality evaluation or proof of identical output on every task. The accuracy run used the engine's v2026.11.11 release binary; the CI build of this source gave the same replies to 12 of 12 seeded requests.

Run the benchmarks yourself

Follow the engine installation guide for Linux, CUDA and the llama.cpp build. Compiling the engine needs about 21 GiB of RAM; a release archive for your GPU avoids that. From megakernel/, with the GPU otherwise idle:

python3 bench/engine_compare.py --seeds 8 --rounds 4 -n 1024 --suite short,long \
  --tokenizer /path/to/qwen/tokenizer.json --output speed.json \
  --engine "openjet=8094=python3 server/mk_server.py -m MODEL --port 8094 -c 106496 --kv q8 --exact-reuse -d 7" \
  --engine "llama.cpp=8094=llama-server -m MODEL --port 8094 --parallel 1 -ngl 99 -c 106496 --jinja \
      --flash-attn on -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 2"

`run_compare.sh` wraps the same comparison together with the accuracy check. For the distribution comparison alone, see Checking a build.

What the results establish

For this RTX 3090, model and configuration, OpenJet decodes 2.0× as fast as llama.cpp with thinking on and 2.6× with 98K tokens in context, and reads long prompts 1.5× as fast, with next-token distributions closer to llama.cpp's own token-by-token reference than llama.cpp's batched mode is.

The two engines were measured on different days, and the tables report means without confidence intervals. Other GPUs, quantizations, concurrent requests and end-to-end agent tasks need their own measurements.

Read the engine results, inspect the benchmark scripts, or continue to the engine documentation and OpenJet vs llama.cpp comparison.