Description
The LLMKube metal agent runs on a Mac and starts inference engines (llama-server, mlx-server, TensorFold, vllm-swift, oMLX, Ollama) for InferenceService objects in the namespaces it watches. Before 0.10.1, the agent trusted those objects, the network and the Mac's shared temporary directory more than it should have. Anyone allowed to create or update InferenceService or Model objects in a watched namespace could act on the Mac as the agent's user and on Kubernetes objects with the agent's credentials, and other users of the Mac or its network could reach further than intended:
- extraArgs argument injection (S1).
spec.extraArgswas appended verbatim to the engine command line. Engine flags can serve a directory over HTTP, write log or cache files to any path, change the listen address, download or swap the model, enable tool execution, and on vllm-swift load Python code. - Model paths outside any allowed root (S2). The agent opened local model sources (absolute paths,
file://, and relative sources containing..) without checking them against an allowed set of directories, and it served Models the controller had already rejected under GHSA-jw3m-8q7m-f35r. - Service and EndpointSlice takeover (S3). The agent created or updated the Service and EndpointSlice named after the InferenceService without checking ownership. An InferenceService named after an existing Service (for example
kubernetesindefault) rewrote that Service and pointed its traffic at the Mac. - Unauthenticated engines on all interfaces (S4). Engines listened on
0.0.0.0with no authentication, exposing inference and engine admin endpoints to anyone who could reach the Mac on the network. - Model store and logs in
/tmp(S5). The default model store (/tmp/llmkube-models, also the default allowed root) and the shipped plist's log (/tmp/llmkube-metal-agent.log) lived in world-writable/tmp, which macOS clears at boot. Another local user could create the store directory first and plant or replace model files, or plant a symlink at the log or engine log path so the agent's user wrote wherever it pointed. - No SSRF guard on agent downloads (S6). The agent fetched
http,https,hfands3Model sources, and the HEAD probes it uses for memory sizing, with a plain HTTP client. A Model author could make the Mac request loopback, link-local (including cloud metadata), private and CGNAT addresses on its network, the same class the controller already blocked under GHSA-jw3m-8q7m-f35r.
S1 to S4 are fixed in 0.10.0. S5 and S6 are fixed in 0.10.1, which is the first release with every issue here fixed.
The agent's own flags, set by the operator in its launchd plist, are trusted; --inference-service-allowlist limited which InferenceService names the agent served but did not limit what those objects could ask for.
Details
S1: spec.extraArgs. Every executor appended extraArgs after its own flags with no filtering, so a later flag overrode the agent's choice and any flag the engine accepts was available. Fixed in #1941: each engine's flags are pinned from its --help and typed; extraArgs are parsed the way each engine parses its own argv (so a flag cannot hide as another flag's value, and abbreviations, _ spellings, dotted keys and vLLM's -O rewrite are handled); listener, registration and multi-node flags are always refused; file serving and logging, auth and TLS material, model downloads, model id overrides, tools, presets and code loading are refused; path flags must resolve inside the allowed roots; unknown flags are refused. --allow-unsafe-extra-args relaxes the non-listener rules for fully trusted authors. 0.10.1 (#1954) also refuses vllm-swift's --trust-request-chat-template, --enable-prompt-embeds and --enable-mm-embeds, requires its tokenizer and config path flags to name an existing path inside the allowed roots, and refuses a --no-X=value spelling as X.
S2: model paths. The controller refuses local sources outside --allowed-host-path-roots (GHSA-jw3m-8q7m-f35r), but the agent opens model files itself on the Mac and did not read the Model's phase. Fixed in #1939: a new --allowed-model-roots flag (the model store is always a root); every local source, relative source and oMLX pagedSSDCacheDir must resolve inside a root after following symlinks; any .. segment is refused whatever the scheme; a Model in phase Failed is not served.
S3: registry ownership. The Service and EndpointSlice writes had no ownership check. Fixed in #1932: the agent only writes or deletes objects labelled llmkube.ai/managed-by=metal-agent (or owned by the InferenceService) and otherwise refuses the start with reason EndpointNameConflict, leaving the existing object untouched.
S4: network exposure. Fixed in #1944 and #1945. Engines the agent starts now bind 127.0.0.1. The agent serves one TLS 1.3 ingress (--ingress-port, default 9443) with a key it generates and keeps on the Mac, and publishes the certificate's SPKI pin on an <isvc>-agent EndpointSlice. For each Metal InferenceService the controller runs a relay pod (router-proxy in --relay mode) that accepts only a certificate matching the pin and presents a per-namespace token from Secret llmkube-metal-relay; the InferenceService's Service is adopted in place and selects the relay, so its name, port and ClusterIP do not change. The ingress answers 401 unless the target is an InferenceService it serves and the token matches that namespace, and 403 for paths outside a pinned per-runtime allowlist (engine admin endpoints such as llama-server /slots and POST /props, vLLM sleep and LoRA loading, and Ollama pull, push and delete stay blocked even for authenticated callers). Relay headers, protocol upgrades and request trailers never reach the engine. In 0.10.1 (#1954) the agent's host-side client port (127.0.0.1:<client-port>) applies the same path allowlist, answers 421 unless the Host is loopback on its own port (which blocks DNS rebinding from a browser on the Mac), rewrites the upstream Host, and refuses protocol upgrades.
S5: model store and logs. Fixed in #1954. The default store is now ~/Library/Application Support/llmkube/models (created 0700) and the shipped plist logs to ~/Library/Logs/llmkube/metal-agent.log. At startup the agent resolves the store, uses the resolved path for the rest of the run, and refuses a store it does not own, one that is group- or other-writable, one with an ancestor that is group- or other-writable without the sticky bit or owned by another non-root user, one that resolves under /tmp or /var/tmp, or a relative path. Store files and engine logs are opened with O_NOFOLLOW and written 0600, and a symlink in a downloaded model's cache slot is refused.
S6: agent downloads. Fixed in #1954. Every download the agent makes for a Model source goes through the same SSRF-guarded client the controller uses (now shared in internal/safehttp): it dials by resolved address and re-checks every redirect hop and proxy target, and refuses loopback, link-local, private, CGNAT, unique-local and unspecified addresses (NAT64 addresses by their embedded IPv4) unless the host is listed in the new --allowed-download-hosts flag. 0.10.1 also enforces Model.spec.sha256 on sources the agent downloads for llama-server: a mismatch deletes the file and refuses the start.
Every refusal starts nothing and is reported as a Warning Event on the InferenceService plus status.schedulingStatus / schedulingMessage (reasons ExtraArgsRejected, ModelSourceNotAllowed, EndpointNameConflict, ServiceNameTooLong, ModelDigestMismatch). A store refused at startup stops the agent with a message naming the path and the fix.
Impact
Who is affected: clusters running the metal agent where someone who is not fully trusted with the Mac's user account can create or update InferenceService or Model objects in a watched namespace (S1 to S3, S6), where the Mac is reachable from a network whose users are not trusted to call its engines directly (S4), or where other people have accounts on the Mac and the agent used the default store or the shipped plist's log path (S5). In-cluster serving on NVIDIA and AMD nodes is not affected; only Macs running the metal agent are.
Single-operator setups where the only InferenceService author is the Mac's owner, on a private network, with no other users on the Mac, have little practical exposure.
Remediation
Upgrade to 0.10.1: the Helm chart first, then each metal agent (a 0.10.x agent with an older controller leaves Metal Services without endpoints unless it runs with the deprecated --legacy-direct-endpoints). deployment/macos/README.md, "Upgrading to 0.10.1", lists every step; in short:
- Re-render the agent's launchd plist. The 0.10.0 plist pinned
--model-store /tmp/llmkube-modelsand a/tmplog, 0.10.1 refuses a/tmpstore, andlaunchctl kickstart -kkeeps the old plist loaded. Runlaunchctl bootout gui/$(id -u)/com.llmkube.metal-agent, wait untillaunchctl printno longer finds the job, thenmake install-metal-agent. - If you pass your own
--model-store, make sure the agent's user owns it, it is not group- or other-writable, and no directory above it is writable by other users (a store on exFAT or FAT fails this). - Add any LAN mirror your Models download from (for example an internal MinIO) to
--allowed-download-hosts. A new agent binary may also need its macOS Local Network permission granted again before LAN downloads work. - Add every directory that holds local models (and the Hugging Face cache, if model store entries link into it) to
--allowed-model-roots, and point a hand-placed model at its absolute path orfile://URI instead of symlinking it into the store. - Remove extraArgs the agent now refuses; each refusal names the flag. Set
--allow-unsafe-extra-argsonly on a Mac whose InferenceService authors are fully trusted. - Rename any InferenceService refused with
EndpointNameConflict. - Open the ingress port (default 9443) in the macOS firewall instead of engine ports; clients keep calling the InferenceService's Service.
- Keep the router-proxy image at the controller's version (the chart default does); the relay runs in it.
- If you run Ollama, unset
OLLAMA_HOST=0.0.0.0so it binds loopback; the agent warns at startup if Ollama or oMLX answer on the host IP. - Consider running the agent on a scoped kubeconfig (
deployment/macos/metal-agent-rbac.yamland the recipe in the README) instead of your own credentials. - Rotate the relay token by updating Secret
llmkube-metal-relayin place (the agent accepts old and new for 10 minutes); deleting it revokes immediately.
Workarounds for older versions
- Restrict create/update on
InferenceServiceandModelin the watched namespaces to people you trust with the Mac's user account. - Keep
--inference-service-allowlistset to the exact names you serve. - Keep the Mac on a trusted network or reachable only over a private overlay (for example Tailscale), with the macOS firewall on.
- Run the agent as a dedicated macOS user with access only to the model directories, and pass it a
--model-storeand log path under that user's home instead of/tmp.
References
- #1931 / #1932 (S3), #1938 / #1939 (S2), #1940 / #1941 (S1), #1943 / #1944 / #1945 (S4), #1949 / #1954 (S5, S6, and the 0.10.1 hardening)
- Releases: github.com/
defilantech/ LLMKube/ releases/ tag/ v0.10.0 and github.com/ defilantech/ LLMKube/ releases/ tag/ v0.10.1 - Write-up: llmkube.com/
blog/ llmkube-0-10-0-metal-trust-boundary - GHSA-jw3m-8q7m-f35r (controller-side host path check and download guard)
deployment/macos/README.md, "Security model" and "Upgrading to 0.10.1"
Affected versions
fixed in 0.10.1