OpenJet vs llama.cpp: local LLM inference compared

What is the difference between OpenJet and llama.cpp?

llama.cpp is a general-purpose C/C++ inference project for running and serving local language models across many devices. OpenJet combines a terminal coding agent, hardware-aware model provisioning and a Python SDK with managed inference.

OpenJet can use llama.cpp itself. For supported Qwen3.8-27B configurations on NVIDIA RTX GPUs, it can instead select its own specialized CUDA engine. The comparison therefore has two parts: the complete coding workflow, and the inference engine serving the model.

Features and hardware at a glance

Capability	OpenJet	llama.cpp
Primary use	Local coding workflow, Python embedding and specialized inference	General local LLM inference, CLI and model serving
Model setup	Profiles hardware, recommends a model and configures runtime settings	Direct model and backend control through CLI/server options
Custom engine	Qwen3.8-27B with supported Q4_K_M or a separate UD-Q4_K_XL build	Broad model and quantization support
Hardware	Custom engine: Linux, RTX 30/40/50-series, 24 GB+ VRAM; other setups use llama.cpp or an external server	CPU, Apple Metal, CUDA, HIP, Vulkan and other backends
Coding tools	File edits, approved shell commands, sessions and MCP integration	Inference foundation; pair with your chosen coding harness
HTTP API	Standalone engine exposes OpenAI-compatible chat completions; one request at a time	llama-server exposes an OpenAI-compatible API with configurable serving features
License	AGPL-3.0-only core	MIT

Qwen3.8-27B benchmarks on an RTX 3090

The following are OpenJet's published repository measurements for Qwen3.8-27B Q4_K_M on the same RTX 3090. llama.cpp uses OpenJet's runtime flags, including MTP speculative decoding. Each result is one conversation.

Workload	llama.cpp	OpenJet Engine
Reasoning decode over HTTP	50.9 tok/s	78.0 tok/s (1.53×)
Greedy decode, four-prompt mean	63.1 tok/s	106.7 tok/s (1.69×)
Greedy Python code prompt	72.5 tok/s	139.7 tok/s (1.93×)
Prompt processing, 2–8K tokens	About 1,050–1,140 tok/s	About 1,530–1,600 tok/s

Reasoning measurements use five coding prompts and two seeds, up to 1,024 output tokens, thinking enabled, temperature 1.0, top_p 0.95 and top_k 20. Greedy results use a different four-prompt workload with 512 output tokens. Speed varies with prompt predictability, draft acceptance and other GPU load.

These figures describe this model, quantization, GPU and test configuration. They do not establish the speed of other models, GPUs, batching workloads or future releases. Read the benchmark settings and reproduction scripts before comparing with your own measurements.

Which should you choose?

Choose OpenJet for a managed local coding agent

Use OpenJet when you want hardware detection, model provisioning, repository edits and approved terminal commands in one workflow. You can keep llama.cpp as the runtime or use the custom engine on a matching machine.

Choose OpenJet Engine for its supported Qwen configuration

Consider the standalone engine when your application serves Qwen3.8-27B on a supported Linux NVIDIA system and its text-only, single-request API matches your workload. The engine uses persistent CUDA decoding, on-device speculative verification and tensor-core prefill.

Choose llama.cpp for broader inference requirements

Use llama.cpp directly when you need broad model and accelerator coverage, direct control over server settings, or a runtime to integrate into an existing stack. Check its current server documentation for concurrency, multimodal and endpoint support. You can still connect OpenJet's agent to an existing llama-server.

Installation and API serving

For the coding agent, install OpenJet and let setup select a compatible runtime:

pipx install open-jet
openjet setup
openjet

For standalone inference, follow the engine build instructions, then run:

openjet-engine --model /absolute/path/to/Qwen3.8-27B-Q4_K_M.gguf --port 18080

Point an OpenAI-compatible chat client to http://127.0.0.1:18080/v1. Use the model ID returned by /v1/models. The engine does not implement every parameter or endpoint of other servers; review the HTTP API and limitations.

Sources and limitations

This comparison uses the OpenJet README, engine benchmarks and limitations, and llama.cpp's official repository. The measurements are reported by the OpenJet project, rather than an independent benchmark. For current llama.cpp serving capabilities, see llama-server documentation.