Handle: omlx
URL: http://localhost:34940

oMLX is an Apple Silicon inference server built on MLX with OpenAI-compatible and Anthropic-compatible APIs, continuous batching, multi-model serving, and paged SSD KV caching. Harbor runs the oMLX server natively on the macOS host for Metal access, then exposes it to Harbor containers through a Caddy proxy.

This service is intended for Apple Silicon systems running macOS 15 or newer. On other platforms the proxy can still point at an externally managed compatible oMLX endpoint, but Harbor cannot provide MLX acceleration inside a Linux container.

harbor up omlx

When HARBOR_OMLX_MANAGE_HOST=true, Harbor automatically starts the oMLX server before the proxy comes up. The server is launched via uv run omlx serve from the services/omlx/ workspace, using the configured model directory and cache directory.

Start a frontend against oMLX:

harbor up webui omlx

Use it with host tools:

harbor launch --backend omlx --model Qwen3.5-4B-4bit codex

The oMLX admin dashboard is available through the proxy:

http://localhost:34940/admin

Following options can be set via harbor config:

# Harbor proxy port
HARBOR_OMLX_HOST_PORT              34940

# Proxy image
HARBOR_OMLX_IMAGE                  caddy
HARBOR_OMLX_VERSION                2-alpine

# Host workspace and data paths
HARBOR_OMLX_WORKSPACE              ./services/omlx
HARBOR_OMLX_BASE_PATH              ./services/omlx
HARBOR_OMLX_MODEL_DIR              ./services/omlx/models
HARBOR_OMLX_CACHE_DIR              ./services/omlx/cache

# Host endpoint
HARBOR_OMLX_UPSTREAM_URL           http://host.docker.internal:8096
HARBOR_OMLX_RUNNER_PORT            8096

# Default model and pull source
HARBOR_OMLX_MODEL                  Qwen3.5-4B-4bit
HARBOR_OMLX_HF_PATH                mlx-community/Qwen3.5-4B-4bit

# Host lifecycle and server options
HARBOR_OMLX_API_KEY                sk-omlx
HARBOR_OMLX_EXTRA_ARGS
HARBOR_OMLX_MANAGE_HOST            true

The Harbor omlx service mounts only services/omlx/Caddyfile into the proxy container. The host runner uses:

• services/omlx/pyproject.toml - project file declaring the pinned oMLX dependency
• services/omlx/.venv/ - virtual environment managed by uv
• services/omlx/models/ - local MLX-format model directories
• services/omlx/cache/ - paged SSD KV cache
• services/omlx/logs/ - host runner log files

harbor models ls --source omlx
harbor models pull --source omlx mlx-community/Qwen3.5-4B-4bit
harbor models rm --source omlx mlx-community/Qwen3.5-4B-4bit

The equivalent source-subcommand form is also supported:

harbor models omlx pull mlx-community/Qwen3.5-4B-4bit

harbor omlx pull downloads Hugging Face model repositories into HARBOR_OMLX_MODEL_DIR. oMLX discovers MLX-format model subdirectories automatically, including two-level paths such as mlx-community/Qwen3.5-4B-4bit.

To change Harbor's default model selection:

harbor config set omlx.model Qwen3.5-4B-4bit

Harbor exposes oMLX at:

http://localhost:34940/v1

Containers use:

http://omlx:8080/v1

oMLX provides OpenAI-compatible endpoints such as /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/rerank, and /v1/models. It also exposes an Anthropic-compatible /v1/messages endpoint at the same proxy.

omlx is a host-native OpenAI-compatible inference server (listening on port 8080 inside the Harbor proxy container) that many Harbor frontends and satellite tools auto-configure when started together. Wiring is performed through cross-compose overlay files (compose.x.*.omlx.yml and a few always-on config mergers). Containers reach the backend at http://omlx:8080/v1 with key sk-omlx unless noted otherwise.

Frontends and tools that auto-configure for omlx

• webui — mounts ./services/webui/configs/config.omlx.json (registers OpenAI provider at http://omlx:8080/v1 with key sk-omlx).
• chatui — mounts ./services/chatui/configs/chatui.omlx.yml.
• aider — mounts ./services/aider/configs/aider.omlx.yml.
• boost — injects HARBOR_BOOST_OPENAI_URL_OMLX and HARBOR_BOOST_OPENAI_KEY_OMLX.
• litellm — mounts ./services/litellm/litellm.omlx.yaml.
• bifrost — bootstrap sidecar (compose.x.bifrost.omlx.yml) registers omlx as a custom OpenAI provider using HARBOR_BIFROST_OMLX_*.
• opint (Open Interpreter) — overrides the entrypoint to pass --api_base http://omlx:8080/v1 --api_key sk-omlx.
• optillm — sets OPTILLM_BASE_URL=http://omlx:8080/v1 and OPENAI_API_KEY=sk-omlx.
• astrbot — sets HARBOR_ASTRBOT_OMLX_ENDPOINT (also configurable in the AstrBot WebUI).
• cognee — depends_on: [omlx], sets LLM_PROVIDER=openai, LLM_ENDPOINT, LLM_API_KEY, and LLM_MODEL=${HARBOR_OMLX_MODEL}.
• mindsdb — sets LLM_FUNCTION_BASE_URL=http://omlx:8080/v1 and OPENAI_API_KEY=sk-omlx.
• mi — depends_on: [omlx], sets OPENAI_BASE_URL=http://omlx:8080 (no /v1) plus MODEL=${HARBOR_OMLX_MODEL}; compose.x.mi.harbor.omlx.yml supplies the harbor variant defaults.
• ml-intern — depends_on: [omlx], sets OMLX_BASE_URL / LOCAL_LLM_* pairs and ML_INTERN_MODEL=omlx/${HARBOR_OMLX_MODEL}.
• npcsh — depends_on: [omlx], configures chat/vision/reasoning models and OpenAI-compatible base URLs via HARBOR_OMLX_MODEL.
• open-design — depends_on: [omlx], sets protocol, base URL, key, and HARBOR_OPEN_DESIGN_DEFAULT_MODEL=${HARBOR_OMLX_MODEL}.
• opennotebook — sets OPENAI_COMPATIBLE_BASE_URL_LLM and matching API key.
• openterminal — depends_on: [omlx], injects HARBOR_OMLX_URL, HARBOR_OMLX_OPENAI_URL, HARBOR_OMLX_API_KEY, and HARBOR_OMLX_MODEL.
• anythingllm — sets GENERIC_OPEN_AI_BASE_PATH=http://omlx:8080/v1.
• cmdh — sets OPENAI_BASE_URL=http://omlx:8080/v1.
• hermes — depends_on: [omlx], sets OPENAI_BASE_URL=http://omlx:8080/v1.
• plandex — sets OPENAI_API_BASE=http://omlx:8080/v1.
• sillytavern — depends_on: [omlx], sets SILLYTAVERN_OMLX_URL=http://omlx:8080.

Reverse proxy / external access

• When traefik is also running, compose.x.traefik.omlx.yml exposes the proxy at https://omlx.${HARBOR_TRAEFIK_DOMAIN} on the websecure entrypoint and joins traefik-public in addition to harbor-network.

harbor omlx status
harbor omlx logs
harbor logs omlx

Check the host runner log:

harbor omlx logs

Ensure uv is installed and that the machine is an Apple Silicon Mac with macOS 15 or newer. On first run, uv creates a virtual environment and installs oMLX into services/omlx/.venv/.

Check the host runner:

harbor omlx status
curl http://localhost:8096/v1/models

If you manage oMLX yourself, set:

harbor config set omlx.manage.host false
harbor config set omlx.upstream.url http://host.docker.internal:8000

• oMLX
• oMLX website
• MLX