● EXPERIMENTAL — Local OpenAI-compatible server

LLM Gateway

A single-file, zero-dependency Node server that exposes local AI CLIs (Claude Code, Antigravity, Gemini) through the standard OpenAI API, so any OpenAI-compatible app or agent can use them without API keys. Covers chat completions, the Responses API, model discovery, real SSE streaming, and tool calling emulated on top of CLIs that have no native tool API.

Node.jsJavaScriptSSEOpenAI APIngrok

~220 linesSingle File

0 depsNode 20 Only

SSE StreamReal-time Output

tool_callsEmulated in Prompt

// Context

Claude Code, Antigravity (agy) and Gemini CLI are logged-in local tools with no HTTP API, while most other apps and agents expect the OpenAI API. The gateway is a thin server in between: a node:http process that spawns the CLI per request, using an argv array and never a shell string, and translates the result into OpenAI-shaped responses.

// What I Built

▪

OpenAI-compatible endpoints: POST /v1/chat/completions, POST /v1/responses, GET /v1/models and /v1/models/{id}. Paths are also accepted without the /v1 prefix — found when a client configured without /v1 got a 404 on /models.

▪

Model routing and discovery: Model ids are backend or backend/model (claude/sonnet, agy/gemini-3.8-flash-low). /v1/models uses static lists for claude and gemini, whose CLIs have no list command, and a dynamic list for agy via agy models with a 5-minute cache.

▪

Real SSE streaming: Child stdout is piped to chat.completion.chunk events as it arrives — several chunks about 130ms apart measured on agy. A client disconnect kills the child process; a 120s timeout bounds every request.

▪

Emulated tool calling: Tool schemas are injected into the prompt with a JSON-only reply protocol. The output is parsed and returned as OpenAI tool_calls with finish_reason tool_calls; role:tool results are folded back into the next prompt.

// Architecture

OpenAI-compatible client→POST /v1/chat/completions→gateway (node:http)→spawn(argv[])

→claude / agy / gemini CLI→stdout→SSE chunks / tool_calls

binds 127.0.0.1 by default — optional bearer API_KEY, ngrok tunnel when used from other apps

// Key Engineering Decisions

▪

Emulate tools in the prompt, validate on the way out: The CLIs have no client-defined tool API. Output is parsed defensively — fences stripped, balanced JSON extracted, tool names checked against the declared tools. Measured 3/3 on agy/gemini-3.8-flash-low and 3/3 on claude/sonnet for a get_weather call; forced tool_choice was less reliable (2/4 on gemini-3.8-flash-low). It is emulation and can fail on weaker models; /v1/responses ignores tools.

▪

Buffer when tools are present: A tool call is only valid once the full JSON is parsed, so with tools declared the output is buffered and then emitted as tool_call chunks instead of streaming raw text.

▪

Local-first security defaults: Binds 127.0.0.1, optional bearer API_KEY, 1MB body cap, prompts never logged, errors in the OpenAI error shape. For use from other apps it is exposed through an ngrok tunnel with the key enabled.

▪

Measure before optimising: About 5s per request on agy with Gemini 3.8 Flash, almost all CLI startup: the CLI run directly takes about 5.0s, so the gateway adds close to nothing. 5 concurrent requests slowed to about 25s (cause not identified), so serialising requests is the known next step.