OpenJetGet Started

Python SDK

Put a local model in your program in one line.

OpenJet is a harness for running open models on the machine you already have. The terminal agent is one way in. The SDK is the other: provision a model into your project at build time, then open an inference session and call it from your own code. The model runs on-device, so nothing you send it goes anywhere — and nothing downloads on your user’s machine either.

python
import asyncio

from openjet.sdk import create_inference_session


async def main() -> None:
    session = await create_inference_session()
    try:
        result = await session.run("Summarise this support ticket in one line.")
        print(result.text)
    finally:
        await session.close()


asyncio.run(main())

What the harness handles for you

The hard part of local inference is rarely the inference call. It is profiling the machine, picking a model that fits the memory you actually have, fetching the right GGUF build, configuring the runtime, and then keeping session and context state sane. OpenJet does that once and exposes the result to both the terminal agent and the SDK, so the two share a backend profile rather than each needing their own setup.

Scaffold it into an existing project

Point OpenJet at a project with a use case, a target device class, and a memory budget in gigabytes. It recommends a configuration the target can run instead of leaving you to guess:

bash
pip install open-jet
cd your-project
openjet project --use-case dialogue --target handheld --budget 4

It asks different questions from openjet setup: the device you ship to rather than this machine, the memory slice your app concedes rather than whatever is free, and first-token latency for your use case rather than raw coding ability. The model lands in .openjet/models/ instead of a machine-wide store, so your build bundles it. OpenJetSession and recommend_hardware_config are exported from openjet.sdk if you want to drive it yourself.

Give the session a role

A session takes a system prompt, which is usually all you need to turn a general model into a component of your product — an NPC, a triage step, a summariser:

python
from openjet.sdk import create_inference_session

session = await create_inference_session(system_prompt="You are a shopkeeper. Two sentences max.")
print((await session.run("The player asks what you have for sale.")).text)

Where this fits

  • Products that ship to devices. Games, handhelds, kiosks, and desktop apps where a per-token bill per user is not viable and the device is sometimes offline.
  • Data that cannot leave. Regulated, sovereign, or air-gapped environments where sending prompts to a hosted API is a policy problem rather than a cost problem.
  • Narrow, high-volume tasks. Classification, extraction, rewriting, and dialogue, where a small local model is adequate and a frontier API is overkill.
  • Prototypes you do not want metered. No key to provision, no quota to blow through while iterating.

Models it runs

Qwen3.5 (4B and 9B), Qwen3.8 27B, Qwen3.6 35B-A3B, and Gemma 4 26B, all as GGUF builds compatible with llama.cpp. Smaller members of that set are the realistic targets for embedded use; the larger ones are aimed at the terminal coding agent on a well-specced machine.

FAQ

How do I embed a local LLM in a Python app?
Two steps. At build time, run openjet project in your repo to provision a model into .openjet/models/. At runtime, import create_inference_session from openjet.sdk and await it to get a session you call with run(). There is no inference server to deploy and no API key to manage.
Does the SDK download the model on the user's machine?
No. Provisioning is a build-time step, so your build bundles the model. If the provisioned model is missing, the session raises immediately rather than reaching for the network on a user's machine — your users never fetch a file, see a config, or need an internet connection.
Can an embedded model run shell commands or read files?
No. The SDK is text in, text out, with every tool refused. An embedded model cannot reach the shell or the filesystem no matter what it generates. Tool use is the terminal agent's job, and there it is approval-gated.
Do I need a GPU?
Not necessarily. The SDK is hardware-aware: openjet project takes a use case, a target device class, and a memory budget, and recommends a configuration that fits. Small models on modest hardware are viable for narrow tasks like dialogue, classification, and summarisation; the terminal coding agent is the part that wants Apple silicon with 24GB+ unified memory or a GPU with 14GB+ VRAM.
Does any data leave the machine?
No. Inference runs on the device your code runs on. There are no API calls to a hosted model provider, which is what makes this workable for offline products, regulated data, and air-gapped deployments.
How is this different from calling llama.cpp directly?
You can call llama.cpp directly — the work is everything around it: profiling the machine, choosing and fetching a GGUF that fits, configuring the runtime, and managing session and context state. OpenJet is that harness, and the SDK is the one-line entry point into it.
What licence is it under?
The core is AGPL-3.0-only. External contributions are accepted under CLA terms.
Install OpenJetView on GitHub

Want the terminal agent instead? See the offline coding agent or the Claude Code comparison.