Skip to content

docs: add Rapid-MLX provider guide - #4660

Open
raullenchai wants to merge 3 commits into
docker:mainfrom
raullenchai:docs/rapid-mlx-provider
Open

raullenchai wants to merge 3 commits into
docker:mainfrom
raullenchai:docs/rapid-mlx-provider

Conversation

@raullenchai

Copy link
Copy Markdown

Rapid-MLX is an open-source (Apache-2.0) local inference server for Apple Silicon that serves MLX models behind OpenAI-compatible Chat Completions and Responses endpoints. It isn't covered in the provider guides yet, so users have no documented way to configure it.

This adds a provider guide at docs/providers/rapid-mlx/index.md, following the same structure as the existing local provider guides. It covers:

  • setup and the port behaviour;
  • optional bearer authentication;
  • the Chat and Responses APIs, including the note that extra_body is only forwarded for Chat;
  • tool calling;
  • troubleshooting, including container networking via host.docker.internal.

A runnable example at examples/rapid-mlx.yaml uses the small qwen3.5-4b-4bit model. The provider is added to docs/data/nav.yml and linked from the local and overview provider pages.

Docs and example only; no Go code changes. go test ./pkg/config passes, including parsing and schema validation of the new example, and markdownlint is clean.

I maintain Rapid-MLX and am happy to keep this page current.

@raullenchai
raullenchai requested a review from a team as a code owner October 19, 2026 22:55
@aheritier aheritier added area/docs Documentation changes kind/docs Documentation-only changes status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key labels Oct 19, 2026
@aheritier

Copy link
Copy Markdown
Collaborator

👋 Some commits in this PR are not signed and verified by GitHub. Please sign your commits with a GPG or SSH key registered in your GitHub account, then force-push.

Commits that are not verified: ab26ff5

See GitHub's guide on signing commits for setup instructions. I've added status/needs-signed-commits; it will be removed automatically once every commit in this PR carries a valid GitHub-verified signature.

aheritier
aheritier previously approved these changes Oct 20, 2026
@raullenchai

Copy link
Copy Markdown
Author

Thanks @aheritier — the commit is now signed and verified, and rebased on the latest main (only change: nav entry placed alphabetically before Runcrate).

@aheritier aheritier removed the status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key label Nov 12, 2026
@aheritier
aheritier marked this pull request as ready for review November 12, 2026 15:26
Copilot AI balanced review requested due to automatic review settings November 12, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation needs Python 3.12+ as the prerequisite for the current Rapid-MLX release and a trusted-host setting for host.docker.internal to prevent HTTP 400 errors in the container setup.

2 open issues
What changed in this PR

This PR documents how to connect Docker Agent to Rapid-MLX’s local inference server on Apple Silicon using a custom provider definition. It adds documentation and an example without changing Go code.

Changes:

  • Adds a guide covering setup, authentication, Chat and Responses APIs, tool calling, and troubleshooting.
  • Provides a runnable configuration using qwen3.5-4b-4bit.
  • Adds navigation and links from the provider overview and local-model guide.
File Description
examples/​rapid-mlx.yaml Adds a local chat configuration with optional authentication.
docs/​providers/​rapid-mlx/​index.md Documents Rapid-MLX setup and integration.
docs/​providers/​overview/​index.md Links to the new provider guide.
docs/​providers/​local/​index.md Includes Rapid-MLX among local model servers.
docs/​data/​nav.yml Adds the Rapid-MLX navigation entry.

Review effort: Balanced

Comment thread docs/providers/rapid-mlx/index.md Resolved
Comment thread docs/providers/rapid-mlx/index.md Resolved

@aheritier aheritier left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files against the custom-provider implementation and Rapid-MLX release sources.

Prior threads:

  • 4552493046: still open [should-fix] (high confidence) — Rapid-MLX 1.0.3 requires Python >=3.12; change the Python 3.10 prerequisite to 3.12+ to avoid installing an older release.
  • 4552493144: still open [blocking] (high confidence) — The documented container URL receives HTTP 400 because host.docker.internal is absent from the default Host allowlist; add it with --trusted-hosts alongside the bind address and API-key configuration.

@aheritier
aheritier marked this pull request as draft November 12, 2026 17:17
@raullenchai

Copy link
Copy Markdown
Author

Thanks for the careful review. Both findings are addressed:

  • The prerequisite now says Python 3.12+.
  • The container setup now binds to 0.0.0.0, sets an API key, and adds --trusted-hosts host.docker.internal.

I checked this against the Rapid-MLX source and --help, and over live HTTP. A request with the Docker Host header goes from 400 to 200 with the flag, and a request without auth still gets 401. Markdown lint, the docs policy checks and the config/provider tests pass. I couldn't run the full Task checks or test actual Docker networking locally.

@raullenchai
raullenchai force-pushed the docs/rapid-mlx-provider branch from eb3f0c4 to a7db056 Compare November 14, 2026 17:24
@aheritier
aheritier marked this pull request as ready for review November 17, 2026 22:16
Copilot AI balanced review requested due to automatic review settings November 17, 2026 22:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Rapid-MLX guide still has unreliable setup and container-authentication instructions, inaccurate model and tool-support assumptions, and an API-key exposure risk.

6 open issues
2 resolved since last review

Review effort: Balanced

Comment thread docs/providers/rapid-mlx/index.md Outdated
```bash
$ brew install rapid-mlx
# or
$ pip install rapid-mlx
Comment thread docs/providers/rapid-mlx/index.md Outdated
Comment on lines +32 to +34
2. Start the server with a model. For a small first download, use the
`qwen3.5-4b-4bit` alias
([`mlx-community/Qwen3.5-4B-MLX-4bit`](https://huggingface.co/mlx-community/Qwen3.5-4B-MLX-4bit)):
Comment on lines +111 to +114
$ export RAPID_MLX_API_KEY=your-server-api-key
$ rapid-mlx serve qwen3.5-4b-4bit --port 8000
$ curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $RAPID_MLX_API_KEY"
$ docker agent run rapid-mlx.yaml --exec "Hello"
Comment thread docs/providers/rapid-mlx/index.md Outdated
Comment on lines +122 to +126
The example is chat-only. Rapid-MLX parses tool calls for the models it
serves, and the parser is selected automatically from the model alias, but
reliability still depends on the model, instructions, and tool schemas. A 4B
model is useful for testing the connection; choose a larger tool-capable model
for agentic coding work.
Comment thread docs/providers/rapid-mlx/index.md Outdated
Host allowlist:

```bash
$ rapid-mlx serve qwen3.5-4b-4bit --port 8000 --host 0.0.0.0 --api-key your-server-api-key --trusted-hosts host.docker.internal
Comment thread docs/providers/rapid-mlx/index.md Outdated
Comment on lines +153 to +155
Configure Docker Agent with the same API key as described in
[Authentication](#authentication). `--trusted-hosts` replaces the default
loopback allowlist; add `localhost 127.0.0.1` if you also need local access.
raullenchai and others added 3 commits November 17, 2026 18:42
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Raullen <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Raullen <[email protected]>
@raullenchai
raullenchai force-pushed the docs/rapid-mlx-provider branch from a7db056 to 974feb8 Compare November 18, 2026 02:42
@raullenchai

Copy link
Copy Markdown
Author

Thanks for the review — addressed all six points in 974feb8 (rebased on main):

  • Install now uses a Python 3.12 venv so rapid-mlx is on PATH.
  • Default model is qwen3-1.7b-4bit (fits 8 GB Macs); noted that qwen3.5-4b-4bit needs 16 GB.
  • Auth section restarts the server with RAPID_MLX_API_KEY instead of --api-key on the command line, and uses a separate client terminal.
  • Tool-calling wording is qualified: automatic parser only for recognized aliases, --tool-call-parser otherwise, with a link to the upstream tool-calling guide.
  • Container example passes the key via env and mounts the YAML.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes kind/docs Documentation-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants