[Official FAQ] Share your experience or follow-up questions below.

Most local inference tools can expose an endpoint that follows the OpenAI API format, so existing apps usually only need a new base_url and model name.

• Ollama: listens on port 11434 by default. The OpenAI-compatible base URL is typically http://localhost:11434/v1 .
• LM Studio: start the local server in the app. The default port is typically 1234, so the base URL looks like http://localhost:1234/v1 .
• llama.cpp: llama-server -m model.gguf -c 8192 -ngl 99 --port 8080. Here -ngl sets how many layers go on the GPU.
• vLLM: vllm serve <model-name>, typically on port 8000.

Python example:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")
r = client.chat.completions.create(model="your-model", messages=[{"role":"user","content":"Hello"}])

Things to watch

• The API key is normally not checked locally. Any non-empty string works.
• Compatibility varies for streaming, tool calls, JSON mode and embeddings. Test the exact features you use.
• By default, services listen on localhost only. If you expose them to your LAN, add authentication and a firewall. Do not put an unauthenticated model server directly on the public internet.
• For many concurrent users, vLLM is generally a better fit than desktop tools.

Running into an error? Paste the request and response (without secrets) and we will take a look.