vLLM Beta

4 min read

The vLLM connection works for vLLM and for any engine that speaks the OpenAI API: LM Studio, llama.cpp (llama-server) or SGLang. The model runs on a machine you control instead of through a cloud provider. In return, you are responsible for that machine's resources, availability and security.

Experimental

The vLLM integration is being tested and may change. Try it first in a controlled environment.

Before connecting it

You need the engine running with the model loaded and an address the agent container can reach. Check that it answers with:

curl http://<address>:8000/v1/models

It should return the model under the exact name you will use in Kujira. If the engine only listens on localhost, make it listen on a network interface (in vLLM, --host 0.0.0.0) and protect access with your private network, a VPN or a firewall.

Kujira talks to the engine through /v1/messages, so you need a vLLM version that includes it. If you start it with --api-key, put that same key in the connection; otherwise leave it empty.

Do not expose the engine unprotected

Without --api-key, anyone who knows the address can use your GPU and see your models. Do not publish the port directly on the internet: use a VPN or an authenticating proxy with HTTPS.

Create a connection

Under Settings > Models > vLLM, click Add connection. A connection is a specific machine and model combination, not a provider account: three machines are three connections.

Name the connection

The name is required and it is the only thing that tells one machine from another in the list and in each agent's configuration. Use something recognizable, for example "Server vLLM".

Enter the address

Enter the URL the agent can reach the engine at, without a trailing /v1: for example, http://192.168.1.85:8000. For an engine on the same server as the agent, http://host.docker.internal:8000 is usually used.

Declare where the server is

Under Server location, choose Private network if it is on the same machine, your local network or a VPN, or Public access if you reach it through a tunnel, your own domain or dynamic DNS. Public access requires HTTPS.

Pick the model and its context

Under Model, pick one from the list: Kujira asks the engine—from your server—which models it serves, and shows each one's context window. Picking one fills in the window. Models that support 64k tokens or less appear dimmed and cannot be picked. If the list does not show up, type the exact name /v1/models returns and give its window, which must be more than 64k.

Assign the connection to the agent

In the agent's Configuration > Model, pick that connection in the Connection dropdown. Since the connection brings its own model, that is the one that will run.

Other compatible services

The connection is not tied to a specific engine: it works for any address that answers what Kujira needs, whether an engine, a model server or your own gateway (for example, LiteLLM).

It must meet three requirements:

• Serve /v1/messages in Anthropic format, with streaming and tools.
• List its models at /v1/models, which is what Test uses.
• Accept the key as Authorization: Bearer, if it asks for one.

If the service does not publish its models' window, enter it yourself on the connection: it must be more than 64k tokens.

Kujira does not check or support what sits behind that address: its terms of use, cost and availability are the responsibility of whoever runs it. Since no cloud provider is involved, this connection's consumption shows as zero in Kujira even if the service behind it charges per token.

Context and performance

On every message an agent sends about 30,000 tokens of instructions and tools before what you write, so Kujira only accepts models with a window of more than 64k tokens. Click Test on the connection: it checks that the engine answers, that it serves the connection's model and that its window is large enough. vLLM publishes each model's window (max_model_len); other engines may not, and then the window you declare on the connection is the one that counts.

The per-token cost does not show up in Kujira because no cloud provider is involved. Consumption still exists: it depends on the GPU, CPU, memory and electricity of your infrastructure.

Common errors

Test says the model is not there

The connection's model name must match exactly the id that /v1/models returns on that machine. Pick it from the list instead of typing it.

The agent cannot connect

Check the address, the port, the firewall and that the engine is listening outside localhost when the agent runs in another container or server.

Credential error

The engine was started with --api-key and the connection does not carry that key, or carries another one. Enter it under Key (optional).

It works for one agent and not another

Agents can run on different servers. With host.docker.internal, each agent talks to the engine on its own server: all of them must have the same model, or you must create a connection for each machine.

See Models for the other options and how a connection is checked.