Inside OpenJet Engine: making speculative decoding pay

The bottleneck: reading weights for a small amount of work

Generating a token requires applying the model's projections to a small set of activations. For a single conversation, those projections have little work to reuse across tokens unless the engine deliberately creates that opportunity. The engine README describes both implementations' non-speculative decode as close to the memory-bandwidth limit: OpenJet's 42.9 tokens/s is only about 6% above llama.cpp's 40.4.

Speculative decoding creates more work for each weight read. A cheaper predictor proposes several future tokens. The main model evaluates the proposed continuation together, accepting a prefix and correcting a failed proposal. Each accepted draft moves the conversation forward without requiring a separate full-model pass for that token.

OpenJet uses the model's MTP block to draft, then evaluates the current token and up to four drafts together. These are positions from one conversation. A verification pass can therefore advance the sequence by several tokens while sharing the expensive weight traffic across the positions.

This gives a useful way to reason about the design:

decode throughput ≈ accepted output tokens per cycle / time per cycle

Both sides of that ratio matter. More drafts can increase accepted output, but drafting takes time and rejected proposals do not advance the conversation. The goal is to keep verification cheap enough, and acceptance high enough, for the extra work to pay for itself. The baseline result is evidence for this explanation, not an isolated measurement of each optimisation's contribution.

One GPU launch for a speculative cycle

The decoding implementation specialises the complete cycle to the model's shapes. The source describes 64 main layers: 48 Gated-DeltaNet layers and 16 gated full-attention layers, plus the MTP block. A persistent cooperative CUDA kernel carries out roughly 340 phases, synchronising GPU blocks with a software grid barrier between phases.

Current token + drafts
        ↓
Main model evaluates all positions together
        ↓
Full-vocabulary output head + verification
        ↓
MTP catches up with accepted positions
        ↓
MTP proposes the next drafts
        ↓
Host receives accepted output tokens

Persistent here means that the GPU kernel stays active through the phases of one cycle. There is still a launch for the next cycle. Keeping projections, attention, feed-forward work, verification and drafting inside that launch reduces kernel-launch overhead and host round trips between those operations.

The projection kernels retain the weights' GGUF quantised representations. Q6_K and Q8_0 weights are rearranged at load time for aligned memory access without changing their values. During decode, integer dot products use CUDA's dp4a instruction with int8 activations. Warps reuse activation loads across weight rows, and weight reads serve multiple verification positions. Adding positions therefore increases arithmetic without multiplying the dominant weight traffic in the same way as separate decode passes.

The producing phase also requantises activations for the next consumer, including the next RMS normalisation's weight. This reduces intermediate work and lets consumers use asynchronous copies. These details support the main design: doing more useful work while the weights are already being streamed. Implementation notes.

Cheap drafting, careful state management

The draft output head scores a bundled vocabulary of 32,768 frequent tokens instead of the entire vocabulary. Tokens from the current sequence that are absent from that subset are added dynamically, helping the predictor reuse identifiers in pasted code. The main model's verification still uses the full vocabulary. Restricting draft candidates can change acceptance and performance without restricting which tokens the final model may emit.

Acceptance depends on the continuation. The published Python prompt reaches 139.7 tokens/s with four drafts, while the short-story prompt reaches 70.7. Three drafts are slightly faster than four on the hash-map explanation, at 107.4 versus 106.5 tokens/s. The implementation also reduces a configured four-draft depth to three for a cycle after a rejection. Extra speculation has a cost, even in a kernel designed to share weight reads.

State handling is another part of that cost. Gated-DeltaNet layers maintain a recurrent state, so speculative positions cannot simply become permanent as they are evaluated: some may be rejected. OpenJet commits state only through the position known to be accepted. It keeps a small ring of intermediate values and replays accepted drafts into the recurrent state at the start of the following cycle. Rejected drafts do not require copying and restoring a full state snapshot.

Full-attention layers use split-K flash decoding, with fp16 tensor-core operations and fp32 accumulation. Double-buffered K/V tiles overlap loading the next tile with processing the current tile. This targets a different cost from weight streaming: reading the growing attention history. As context grows, that history takes more work, so short-prompt throughput cannot describe every conversation length. Drafting, replay and attention details.

Prompts need a different execution path

Prompt processing, or prefill, has many known tokens available at once. That gives the GPU enough parallel work to use larger matrix multiplications. OpenJet therefore uses ordinary batched CUDA kernels for prefill rather than feeding the whole prompt through its small-position decode cycle.

The prefill kernels process all but the last prompt token in chunks of up to 2,048 tokens. Quantised projection weights are dequantised in registers for fp16 tensor-core matrix multiplication; the engine avoids expanding the whole model into a separate fp16 weight copy. Gate and up projections share a matrix multiplication whose epilogue prepares the feed-forward down projection's input. DeltaNet uses a chunked formulation, and full attention uses causal flash attention. Prefill design.

Prefill leaves the caches and recurrent state in the layout decode expects. The last prompt token enters the persistent kernel, which produces the first output token and the first drafts. A growing conversation uses the batched path to append new input, including any accepted positions still waiting to be committed to recurrent state.

This matters for a coding agent: reading a pasted file or tool result may dominate the wait before generation starts. In the published second-turn test, processing 2.7K new tokens takes 1.68 seconds in OpenJet versus 2.41 seconds with llama.cpp's prompt cache. At 12K new tokens, the times are 8.10 and 10.45 seconds. These are prompt-processing times; generating the response adds further latency.

What the measurements show

The published results compare the same Qwen3.8-27B Q4_K_M GGUF on an otherwise idle RTX 3090, running back-to-back with one conversation. The mixed-quant UD-Q4_K_XL build is outside this comparison.

The speculative comparisons enable MTP on both sides. llama.cpp uses at most two drafts, flash attention, q8_0 keys and values, 99 GPU layers and one server slot. OpenJet's headline results use four drafts, the restricted draft vocabulary and fp16 KV. These are the project's runtime configurations, with different draft depths and KV formats. The README reports that q8_0 KV slows OpenJet decode by approximately 1–2% and prefill by 2–7% relative to fp16.

Reasoning uses five coding prompts, two seeds and up to 1,024 output tokens, with thinking enabled, xhigh effort, temperature 1.0, top_p 0.95 and top_k 20. The CLI reproduction script aggregates generated tokens over decode time after averaging seeds per prompt; it reports 77.4 versus 50.7 tokens/s. The 78.0 HTTP headline is a separate README measurement. Greedy testing disables thinking, requests 512 tokens per short prompt, keeps the fastest of two repetitions for each prompt and engine, then averages the four prompt rates.

Longer-context results show why these distinctions matter. At 1.8K measured context, OpenJet decodes at 84.8 tokens/s versus 57.4, and prefills at 1,595 versus 1,050 tokens/s. At 29.5K, the corresponding rates are 72.7 versus 52.3 for decode and 1,236 versus 1,062 for prefill. That sweep predates the sequence-token draft vocabulary and double-buffered attention changes; the figures here retain the original measurements without projecting an uplift.

Decode rate excludes model loading, prompt processing and response transport. Faster decode does not imply the same proportional reduction in total request time or coding-task completion time. The benchmark article supplies the complete tables, aggregation rules and reproduction commands, including the fact that the context sweep selects its best prefill and decode rates independently.

Correctness has two separate questions

First, does speculation preserve the target model's selection rule? Greedy verification follows the main model's choice. Sampled verification is designed to preserve its sampling distribution through acceptance and correction; the README reports a 2,000-sample test per position. The smaller draft vocabulary belongs to the proposal mechanism, while verification retains the full output vocabulary.

Second, how closely does the CUDA implementation match llama.cpp numerically? Arithmetic ordering and precision differ. Even changing verification batch size can change a near-tie between tokens, so preserving the sampling rule does not guarantee byte-identical generations across engines or draft settings.

The project compares teacher-forced next-token distributions against llama.cpp generating token by token with q8_0 KV. On 13,299 positions of repository code and prose after 1,024-token prompts, it reports:

Reference perplexity is 2.4263; lower KL means distributions closer to that reference. At 30K and 96K context, OpenJet with q8_0 KV reports 99.65% and 99.60% agreement respectively, each over 1,999 scored positions from a single code sequence. These checks support numerical similarity on the tested material. They do not measure coding-task success or establish identical behaviour on every input. Correctness procedure and results.

Where the design stops helping

Specialisation makes this approach practical and sets its boundaries. The engine targets Qwen3.8-27B shapes and supported tensor quantisation recipes. A GGUF extension or a Q4 label alone does not establish compatibility. The original and mixed-quant engines use separate builds; the loader checks tensor types.

The supported environment is Linux with an NVIDIA sm_86, sm_89 or sm_120 GPU and at least 24 GB VRAM, plus the CUDA toolkit. The engine is tuned on the RTX 3090. These results do not establish its speed on other supported GPUs, other models or quantisations. CPU and Apple GPU execution use other runtime paths.

The API is text-only. It supports temperature, top-p, min-p and top-k up to 32, but ignores presence and repetition penalties with a warning. Editing an early part of conversation history, including context compaction, requires prefill again. There is no slot save/restore. These restrictions affect whether the engine can replace an existing application backend, even when its decode rate is attractive.

The measured workload is one conversation. The current engine documentation also describes parallel serving for up to eight conversations in the Q4_K_M engine, with shared context capacity; that newer capability needs its own performance measurements. The mixed-quant engine serves one conversation at a time.

Finally, the published speed tables have no confidence intervals or complete run manifest covering clocks, power limits, drivers, model checksums, benchmark dates and every engine revision. Best-of-two greedy rates describe favourable observed throughput. Poor draft acceptance, longer attention histories and time spent outside decode can all reduce the practical gain.

OpenJet's engineering result is that a specialised verifier can turn shared weight reads into more accepted tokens per cycle. The published RTX 3090 measurements demonstrate that benefit for the specified workloads. To evaluate it for an application, start with the standalone installation, run the benchmark procedures, and measure prompt latency, generation, numerical behaviour and complete task time on the workload you intend to serve.