From 3ad27d5fef941fdcaa5cb079d70d2f1eda9251ef Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 8 Nov 2026 19:52:27 +0000 Subject: [PATCH 01/12] docs(patterns): add vLLM Ray Inference Server sandbox pattern Add an overview page and a getting-started page for the rhvp-vllm-ray-inference-server sandbox pattern, plus its architecture diagram. --- .../_index.adoc | 101 +++++++++ .../getting-started.adoc | 206 ++++++++++++++++++ .../architecture.svg | 1 + 3 files changed, 308 insertions(+) create mode 100644 content/patterns/rhvp-vllm-ray-inference-server/_index.adoc create mode 100644 content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc create mode 100644 static/images/rhvp-vllm-ray-inference-server/architecture.svg diff --git a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc new file mode 100644 index 000000000..6059f0c8d --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc @@ -0,0 +1,101 @@ +--- +# The directory name, the aliases, and the GitHub links in this page set use the +# current repository name, rhvp-vllm-ray-inference-server. That name can change, +# pending two open questions to the Validated Patterns team: whether the "rhvp-" +# prefix is acceptable under the Red Hat trademark guidelines, and whether the +# repository moves to the validatedpatterns-sandbox organization. If either +# changes, rename the directory, the aliases, the image folder, and every link +# in both pages together. +title: vLLM Ray Inference Server +date: 2026-11-08 +validated: false +tier: sandbox +summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat OpenShift GitOps +# partners: left out on purpose. The pattern deploys software from NVIDIA (the +# GPU Operator and the CUDA base image) and from the Ray and vLLM projects, but +# no partner organization takes part in the pattern. A partners entry suggests a +# partnership, and the site prints a partner trademark notice for it. Add an +# entry only when a partner organization joins the pattern. +industries: + - General +focus_areas: + - AI +aliases: /rhvp-vllm-ray-inference-server/ +links: + github: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server + install: getting-started + arch: /images/rhvp-vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +--- +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +// The pattern name can change. See the comment at the top of the front matter. + +[id="about-vllm-ray-inference-server-pattern"] += About the {vllm-ray-pattern} + +The {vllm-ray-pattern} deploys a private, OpenAI-compatible large language model (LLM) endpoint on {rh-ocp}. {rh-gitops} deploys every component from your fork of the pattern repository, and the cluster builds the server image itself from upstream packages. Eugo maintains the pattern as community content in the {sandbox-tier-first}. + +[WARNING] +==== +On {AWS}, the default installation creates one billable GPU instance: a `g6.2xlarge` instance with one NVIDIA L4 GPU. AWS bills the instance for as long as it runs. To skip the instance, set `global.gpuMachineSet: none` in the `values-global.yaml` file in your fork, and push the change before you install the pattern. On other platforms, the pattern creates no instance. Uninstalling the pattern does not delete the instance. For the steps to delete it, see link:getting-started/#uninstalling-the-vllm-ray-pattern[Uninstalling the pattern]. +==== + +[id="background-vllm-ray"] +== Background + +A team that wants a private, OpenAI-compatible LLM endpoint on its own {ocp} cluster usually assembles the parts by hand: GPU enablement, a serving runtime, a model cache, a secret for gated models, and a way to rebuild the server image when the software stack changes. This pattern describes all of these parts in one Git repository. You fork the repository, set the model, and push the change, and the cluster converges to a running endpoint. + +The pattern pins every Python package by hash and the base image by digest, so each rebuild installs the same software stack, and each change is a reviewable commit. The pattern uses upstream open source components only: vLLM, Ray Serve, and Python wheels from PyPI, on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. + +[id="architecture-vllm-ray"] +== Architecture + +.Architecture of the {vllm-ray-pattern} +image::/images/rhvp-vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] + +The pattern works in the following way: + +* The {validated-patterns-op} installs {rh-gitops} and points the Argo CD instance at your fork. The `clustergroup` Helm chart then creates every application that the `prod` variant lists. +* {hashicorp-vault} and the {eso-op} deliver an optional Hugging Face token to the server as a Kubernetes secret. The default model is not gated and needs no token. +* The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. +* A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. +* A persistent volume claim keeps the model weights across pod restarts. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `rhvp-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. + +[id="components-vllm-ray"] +== Components that the pattern deploys + +The pattern deploys the following Operators: + +* {rh-gitops} +* The {validated-patterns-op}, a community Operator +* The {eso-op} for Red{nbsp}Hat OpenShift +* The {rh-nfd} +* The NVIDIA GPU Operator, from the certified Operators catalog + +The pattern deploys the following applications: + +* {hashicorp-vault}, from the {solution-name-upstream} Helm chart. +* The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. + +The link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. + +[id="support-vllm-ray"] +== Support + +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. + +[id="next-steps-vllm-ray"] +== Next steps + +* link:getting-started[Getting started] diff --git a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc new file mode 100644 index 000000000..9d1592400 --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc @@ -0,0 +1,206 @@ +--- +# The pattern name in this page can change. See the comment at the top of the +# front matter in _index.adoc in this folder. +title: Getting started +weight: 10 +aliases: /rhvp-vllm-ray-inference-server/getting-started/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +[id="deploying-the-vllm-ray-pattern"] +== Deploying the {vllm-ray-pattern} + +You deploy the pattern from your own fork of the pattern repository. {rh-gitops} deploys the content of your fork on GitHub, not the files on your workstation, so commit and push every change before you install the pattern. + +.Prerequisites + +* An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. +* The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: ++ +[source,terminal] +---- +$ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.managementState}' +---- +* A GPU node: +** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. +** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. +** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. + +.Procedure + +. Fork the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server[rhvp-vllm-ray-inference-server] repository on GitHub. + +. Clone your fork: ++ +[source,terminal,subs="+quotes"] +---- +$ git clone https://github.com/____/rhvp-vllm-ray-inference-server.git +---- + +. Go to the root directory of the repository: ++ +[source,terminal] +---- +$ cd rhvp-vllm-ray-inference-server +---- + +. Edit the `values-global.yaml` file: +** Set `global.gpuMachineSet` to `create` to let the pattern create one AWS `g6.2xlarge` GPU node, or to `none` to use a GPU node that you provide. +** Optional: To serve a different model from Hugging Face, set `global.model.id`, `global.model.source`, and `global.model.revision`. + +. Commit and push the change: ++ +[source,terminal] +---- +$ git commit -am "Configure the pattern" +$ git push +---- + +. Optional: To serve a gated model, create the secrets file for the pattern and add your Hugging Face token: +.. Copy the secrets template: ++ +[source,terminal] +---- +$ mkdir -p ~/.config/validated-patterns +$ cp values-secret.yaml.template \ + ~/.config/validated-patterns/values-secret-rhvp-vllm-ray-inference-server.yaml +---- +.. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. + +. Log in to the cluster: ++ +[source,terminal,subs="+quotes"] +---- +$ oc login --token=____ --server=____ +---- + +. Install the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make install +---- ++ +The installer waits up to 60 minutes for every Argo CD application to report `Synced` and `Healthy`. A first installation can take longer, because the cluster builds a large server image, the GPU node joins the cluster, and the server downloads the model. If the wait times out, the pattern stays installed. Check the progress with the commands in the next step, and then run `./pattern.sh make argo-healthcheck`. + +. Watch the deployment: ++ +[source,terminal] +---- +$ oc get applications.argoproj.io -A +$ oc get builds -n rhvp-inference +$ oc get pods -n rhvp-inference -w +---- ++ +The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. + +[id="verifying-the-vllm-ray-pattern"] +== Verifying the inference server + +The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inference.svc:8000`. To send a test request from your workstation, forward the server port. + +.Procedure + +. Forward the server port to your workstation: ++ +[source,terminal] +---- +$ oc port-forward svc/ray-serve-llm 8000:8000 -n rhvp-inference +---- + +. In another terminal, list the models that the server serves: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/models +---- + +. Send a chat request: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "qwen2.5-0.5b-instruct", + "messages": [{"role": "user", "content": "Say hello in one sentence."}], + "max_tokens": 64 + }' +---- ++ +The `model` field takes the value of `global.model.id`. Any OpenAI-compatible client works with the base URL `http://localhost:8000/v1`. + +[id="opening-the-notebook-vllm-ray"] +== Opening the notebook + +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `rhvp-notebook` namespace, and cluster administrators, can open it. + +.Procedure + +. Get the URL of the notebook route: ++ +[source,terminal] +---- +$ oc get route -n rhvp-notebook +---- + +. Open the URL in a browser and log in with your OpenShift account. + +. Open the `stock_vs_eugo_inference.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. + +. Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: ++ +[source,terminal,subs="+quotes"] +---- +$ oc adm policy add-role-to-user edit ____ -n rhvp-notebook +---- + +[id="uninstalling-the-vllm-ray-pattern"] +== Uninstalling the pattern + +The {solution-name-upstream} framework marks uninstallation as experimental. Uninstalling the pattern leaves the {validated-patterns-op} and {rh-gitops} installed, and it does not delete the AWS GPU node. AWS bills the GPU instance until you delete its machine set. + +.Procedure + +. Uninstall the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make uninstall +---- + +. List the machine sets with their instance types: ++ +[source,terminal] +---- +$ oc get machinesets -n openshift-machine-api \ + -o custom-columns=NAME:.metadata.name,TYPE:.spec.template.spec.providerSpec.value.instanceType +---- + +. Delete the machine set of the `g6.2xlarge` instance type. Delete it only after the uninstallation finishes, or after the `global.gpuMachineSet: none` setting syncs. Otherwise, the job creates the machine set again. ++ +[source,terminal,subs="+quotes"] +---- +$ oc delete machineset ____ -n openshift-machine-api +---- + +. Check for a leftover persistent volume from the `rhvp-inference/ray-serve-llm-models` claim: ++ +[source,terminal] +---- +$ oc get pv +---- + +[id="next-steps-vllm-ray-getting-started"] +== Next steps + +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/rhvp-vllm-ray-inference-server/architecture.svg b/static/images/rhvp-vllm-ray-inference-server/architecture.svg new file mode 100644 index 000000000..9ddcbee4b --- /dev/null +++ b/static/images/rhvp-vllm-ray-inference-server/architecture.svg @@ -0,0 +1 @@ +OpenShift clusterNamespacerhvp-notebookNamespacerhvp-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamrhvp-notebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file From 4faa817797bdb0d8673128d9a40fc1f3799d1a41 Mon Sep 17 00:00:00 2001 From: Ben Date: Wed, 11 Nov 2026 22:46:08 +0000 Subject: [PATCH 02/12] docs(patterns): rename to vllm-ray-inference-server, list Eugo as a partner, apply review notes The pattern repository is now named vllm-ray-inference-server. Move both page folders and the image folder to that name, and update the aliases, the architecture image path, the GitHub links, the clone and cd commands, the secrets file name and the notebook name to match. List Eugo, the maintainer, under partners, so the overview page prints the partner trademark notice. Apply the review notes: drop the front matter comments about a possible rename and the left-out partners entry, and drop validated: false. --- .../_index.adoc | 31 ++++++------------- .../getting-started.adoc | 18 +++++------ .../architecture.svg | 0 3 files changed, 17 insertions(+), 32 deletions(-) rename content/patterns/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/_index.adoc (70%) rename content/patterns/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/getting-started.adoc (87%) rename static/images/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/architecture.svg (100%) diff --git a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc similarity index 70% rename from content/patterns/rhvp-vllm-ray-inference-server/_index.adoc rename to content/patterns/vllm-ray-inference-server/_index.adoc index 6059f0c8d..c1e3b0cd7 100644 --- a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -1,34 +1,23 @@ --- -# The directory name, the aliases, and the GitHub links in this page set use the -# current repository name, rhvp-vllm-ray-inference-server. That name can change, -# pending two open questions to the Validated Patterns team: whether the "rhvp-" -# prefix is acceptable under the Red Hat trademark guidelines, and whether the -# repository moves to the validatedpatterns-sandbox organization. If either -# changes, rename the directory, the aliases, the image folder, and every link -# in both pages together. title: vLLM Ray Inference Server date: 2026-11-08 -validated: false tier: sandbox summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. rh_products: - Red Hat OpenShift Container Platform - Red Hat OpenShift GitOps -# partners: left out on purpose. The pattern deploys software from NVIDIA (the -# GPU Operator and the CUDA base image) and from the Ray and vLLM projects, but -# no partner organization takes part in the pattern. A partners entry suggests a -# partnership, and the site prints a partner trademark notice for it. Add an -# entry only when a partner organization joins the pattern. +partners: + - Eugo industries: - General focus_areas: - AI -aliases: /rhvp-vllm-ray-inference-server/ +aliases: /vllm-ray-inference-server/ links: - github: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server + github: https://github.com/eugo-inc/vllm-ray-inference-server install: getting-started - arch: /images/rhvp-vllm-ray-inference-server/architecture.svg - bugs: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues + arch: /images/vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/vllm-ray-inference-server/issues feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform --- :toc: @@ -37,8 +26,6 @@ links: include::modules/comm-attributes.adoc[] :vllm-ray-pattern: vLLM Ray Inference Server pattern -// The pattern name can change. See the comment at the top of the front matter. - [id="about-vllm-ray-inference-server-pattern"] = About the {vllm-ray-pattern} @@ -60,7 +47,7 @@ The pattern pins every Python package by hash and the base image by digest, so e == Architecture .Architecture of the {vllm-ray-pattern} -image::/images/rhvp-vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] +image::/images/vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] The pattern works in the following way: @@ -88,12 +75,12 @@ The pattern deploys the following applications: * The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. * The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. -The link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. +The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. [id="support-vllm-ray"] == Support -The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. [id="next-steps-vllm-ray"] == Next steps diff --git a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc similarity index 87% rename from content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc rename to content/patterns/vllm-ray-inference-server/getting-started.adoc index 9d1592400..2e8e07f4c 100644 --- a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -1,9 +1,7 @@ --- -# The pattern name in this page can change. See the comment at the top of the -# front matter in _index.adoc in this folder. title: Getting started weight: 10 -aliases: /rhvp-vllm-ray-inference-server/getting-started/ +aliases: /vllm-ray-inference-server/getting-started/ --- :toc: @@ -31,25 +29,25 @@ $ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec ** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. ** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. ** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. -* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. * On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. .Procedure -. Fork the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server[rhvp-vllm-ray-inference-server] repository on GitHub. +. Fork the link:https://github.com/eugo-inc/vllm-ray-inference-server[vllm-ray-inference-server] repository on GitHub. . Clone your fork: + [source,terminal,subs="+quotes"] ---- -$ git clone https://github.com/____/rhvp-vllm-ray-inference-server.git +$ git clone https://github.com/____/vllm-ray-inference-server.git ---- . Go to the root directory of the repository: + [source,terminal] ---- -$ cd rhvp-vllm-ray-inference-server +$ cd vllm-ray-inference-server ---- . Edit the `values-global.yaml` file: @@ -71,7 +69,7 @@ $ git push ---- $ mkdir -p ~/.config/validated-patterns $ cp values-secret.yaml.template \ - ~/.config/validated-patterns/values-secret-rhvp-vllm-ray-inference-server.yaml + ~/.config/validated-patterns/values-secret-vllm-ray-inference-server.yaml ---- .. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. @@ -154,7 +152,7 @@ $ oc get route -n rhvp-notebook . Open the URL in a browser and log in with your OpenShift account. -. Open the `stock_vs_eugo_inference.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. +. Open the `inference_timings.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. . Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: + @@ -202,5 +200,5 @@ $ oc get pv [id="next-steps-vllm-ray-getting-started"] == Next steps -* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. * For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/rhvp-vllm-ray-inference-server/architecture.svg b/static/images/vllm-ray-inference-server/architecture.svg similarity index 100% rename from static/images/rhvp-vllm-ray-inference-server/architecture.svg rename to static/images/vllm-ray-inference-server/architecture.svg From 60102f448a6ef2745f9187a7ea12146e2f4ebd72 Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 00:05:57 +0000 Subject: [PATCH 03/12] docs(patterns): follow the pattern's namespace rename and list decode speed The pattern's namespaces are now vllm-inference (the server) and vllm-notebook (the notebook). Update the oc commands, the in-cluster server URL, the notebook access text and the leftover-volume check on both pages, and the namespace labels in the architecture diagram, whose notebook build box now names the BuildConfig and ImageStream notebook. The overview page's notebook entry also lists the single-stream decode speed that the notebook measures. --- .../vllm-ray-inference-server/_index.adoc | 4 ++-- .../getting-started.adoc | 18 +++++++++--------- .../vllm-ray-inference-server/architecture.svg | 2 +- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index c1e3b0cd7..e6b637f4c 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -56,7 +56,7 @@ The pattern works in the following way: * The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. * A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. * A persistent volume claim keeps the model weights across pod restarts. -* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `rhvp-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `vllm-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. [id="components-vllm-ray"] == Components that the pattern deploys @@ -73,7 +73,7 @@ The pattern deploys the following applications: * {hashicorp-vault}, from the {solution-name-upstream} Helm chart. * The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. -* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at several concurrency levels. The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. diff --git a/content/patterns/vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc index 2e8e07f4c..3a2206316 100644 --- a/content/patterns/vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -94,8 +94,8 @@ The installer waits up to 60 minutes for every Argo CD application to report `Sy [source,terminal] ---- $ oc get applications.argoproj.io -A -$ oc get builds -n rhvp-inference -$ oc get pods -n rhvp-inference -w +$ oc get builds -n vllm-inference +$ oc get pods -n vllm-inference -w ---- + The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. @@ -103,7 +103,7 @@ The server pod stays in the `Pending` state until a node advertises the `nvidia. [id="verifying-the-vllm-ray-pattern"] == Verifying the inference server -The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inference.svc:8000`. To send a test request from your workstation, forward the server port. +The server is reachable inside the cluster at `http://ray-serve-llm.vllm-inference.svc:8000`. To send a test request from your workstation, forward the server port. .Procedure @@ -111,7 +111,7 @@ The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inferen + [source,terminal] ---- -$ oc port-forward svc/ray-serve-llm 8000:8000 -n rhvp-inference +$ oc port-forward svc/ray-serve-llm 8000:8000 -n vllm-inference ---- . In another terminal, list the models that the server serves: @@ -139,7 +139,7 @@ The `model` field takes the value of `global.model.id`. Any OpenAI-compatible cl [id="opening-the-notebook-vllm-ray"] == Opening the notebook -The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `rhvp-notebook` namespace, and cluster administrators, can open it. +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `vllm-notebook` namespace, and cluster administrators, can open it. .Procedure @@ -147,18 +147,18 @@ The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can + [source,terminal] ---- -$ oc get route -n rhvp-notebook +$ oc get route -n vllm-notebook ---- . Open the URL in a browser and log in with your OpenShift account. . Open the `inference_timings.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. -. Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: +. Optional: To give another user access to the notebook, grant the `edit` role in the `vllm-notebook` namespace: + [source,terminal,subs="+quotes"] ---- -$ oc adm policy add-role-to-user edit ____ -n rhvp-notebook +$ oc adm policy add-role-to-user edit ____ -n vllm-notebook ---- [id="uninstalling-the-vllm-ray-pattern"] @@ -190,7 +190,7 @@ $ oc get machinesets -n openshift-machine-api \ $ oc delete machineset ____ -n openshift-machine-api ---- -. Check for a leftover persistent volume from the `rhvp-inference/ray-serve-llm-models` claim: +. Check for a leftover persistent volume from the `vllm-inference/ray-serve-llm-models` claim: + [source,terminal] ---- diff --git a/static/images/vllm-ray-inference-server/architecture.svg b/static/images/vllm-ray-inference-server/architecture.svg index 9ddcbee4b..012e72bf3 100644 --- a/static/images/vllm-ray-inference-server/architecture.svg +++ b/static/images/vllm-ray-inference-server/architecture.svg @@ -1 +1 @@ -OpenShift clusterNamespacerhvp-notebookNamespacerhvp-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamrhvp-notebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file +OpenShift clusterNamespacevllm-notebookNamespacevllm-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamnotebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file From 55fd71dc5bece9ae9c5961b87f64122db764052a Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 04:03:49 +0000 Subject: [PATCH 04/12] docs(patterns): frame the vLLM Ray Inference Server beside Red Hat AI products - Background: say that the pattern does not replace or integrate with the Red Hat AI products, such as Red Hat OpenShift AI, which parts are pinned and which are not, the one-time install step and what OpenShift GitOps does not do after it, and what the notebook measures. Drop the claims that each rebuild installs the same software stack and that the pattern uses upstream open source components only. - Prerequisites: use a cluster that does not already run the Node Feature Discovery Operator or the NVIDIA GPU Operator. The pattern installs both itself and can conflict with existing installations. It has not been tested on a cluster that runs Red Hat OpenShift AI. --- .../patterns/vllm-ray-inference-server/_index.adoc | 12 ++++++++++-- .../vllm-ray-inference-server/getting-started.adoc | 1 + 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index e6b637f4c..d90094b4a 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -39,9 +39,17 @@ On {AWS}, the default installation creates one billable GPU instance: a `g6.2xla [id="background-vllm-ray"] == Background -A team that wants a private, OpenAI-compatible LLM endpoint on its own {ocp} cluster usually assembles the parts by hand: GPU enablement, a serving runtime, a model cache, a secret for gated models, and a way to rebuild the server image when the software stack changes. This pattern describes all of these parts in one Git repository. You fork the repository, set the model, and push the change, and the cluster converges to a running endpoint. +The {vllm-ray-pattern} does not replace the Red{nbsp}Hat AI products, such as {rhoai}, and it does not integrate with them. The pattern is a small, community-maintained reference point: the upstream vLLM engine on Ray Serve, built from known versions and measured the same way each time. The pattern does not need a {rhoai} subscription. -The pattern pins every Python package by hash and the base image by digest, so each rebuild installs the same software stack, and each change is a reviewable commit. The pattern uses upstream open source components only: vLLM, Ray Serve, and Python wheels from PyPI, on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. +Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command once. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. + +The pattern pins every Python package by hash and installs wheels only. It pins the base images of the server and the notebook by digest, and the default model by its Hugging Face commit. Some parts are not pinned, among them the system packages that the server image installs with `dnf`, the Operator versions, and the patch versions of the Validated Patterns Helm charts. Two builds from the same commit therefore install the same Python packages, but their system packages can differ. + +The notebook measures the server in your cluster: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at 1, 8, and 32 concurrent requests. The notebook chooses its prompts from a fixed seed, and every request uses a temperature of 0 and a fixed number of output tokens. The time to `Ready` is in whole seconds and can trail the server by up to about 2 seconds. + +The server runs Ray Serve LLM and the vLLM engine in one pod of a standard deployment, without KServe or KubeRay. Ray, vLLM, and PyTorch come from PyPI and run on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. The pattern creates no KServe `InferenceService` or `ServingRuntime` resource. + +The pattern installs the {rh-nfd} and the NVIDIA GPU Operator itself. On a cluster that already runs these Operators, the pattern can conflict with them, and the pattern has not been tested on a cluster that runs {rhoai}. Install the pattern on a cluster that does not run these Operators. [id="architecture-vllm-ray"] == Architecture diff --git a/content/patterns/vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc index 3a2206316..ff220e8a6 100644 --- a/content/patterns/vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -18,6 +18,7 @@ You deploy the pattern from your own fork of the pattern repository. {rh-gitops} .Prerequisites * An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A cluster that does not already run the {rh-nfd} or the NVIDIA GPU Operator. The pattern installs both Operators itself, with its own `OperatorGroup` objects and settings, and can conflict with existing installations of these Operators. The pattern has not been tested on a cluster that runs {rhoai}. * A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. * The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: + From d98e6932bc7cc665afee9851ed6b88a47ccfa531 Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 04:06:30 +0000 Subject: [PATCH 05/12] docs(patterns): say one time instead of once in the vLLM Ray Inference Server background --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index d90094b4a..9f03307a5 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -41,7 +41,7 @@ On {AWS}, the default installation creates one billable GPU instance: a `g6.2xla The {vllm-ray-pattern} does not replace the Red{nbsp}Hat AI products, such as {rhoai}, and it does not integrate with them. The pattern is a small, community-maintained reference point: the upstream vLLM engine on Ray Serve, built from known versions and measured the same way each time. The pattern does not need a {rhoai} subscription. -Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command once. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. +Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command one time. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. The pattern pins every Python package by hash and installs wheels only. It pins the base images of the server and the notebook by digest, and the default model by its Hugging Face commit. Some parts are not pinned, among them the system packages that the server image installs with `dnf`, the Operator versions, and the patch versions of the Validated Patterns Helm charts. Two builds from the same commit therefore install the same Python packages, but their system packages can differ. From ad90260cf9bb4a407c3e5f727f5eeb836f1313c7 Mon Sep 17 00:00:00 2001 From: Ben Date: Sat, 14 Nov 2026 20:18:22 +0000 Subject: [PATCH 06/12] docs(patterns): add Eugo's partner results to the vLLM Ray Inference Server page Add a "Partner results: Eugo" section before Support: the start-up time and start-up reliability of Eugo's next vLLM image (release pending) against the upstream vLLM image, and a link to the matching section of the pattern README. --- content/patterns/vllm-ray-inference-server/_index.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index 9f03307a5..d8b8a088e 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -85,6 +85,16 @@ The pattern deploys the following applications: The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. +[id="partner-results-vllm-ray"] +== Partner results: Eugo + +Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. + +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). +* *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. + +For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. + [id="support-vllm-ray"] == Support From 1878e28ae76fa58a92bb206f2b360e94dfca8b8b Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 17:50:40 +0000 Subject: [PATCH 07/12] docs(patterns): say which attention kernels the vLLM Ray Inference Server start-up result compared Add one sentence to the start-up bullet of "Partner results: Eugo": Eugo's image picked Triton attention by itself, upstream with --attention-backend TRITON_ATTN was not measured, and the lead against it is likely smaller. --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index d8b8a088e..13a092912 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -90,7 +90,7 @@ The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-st Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. -* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). Eugo's image picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not measured, and the lead against it is likely smaller. * *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. From 001e672971f0cdabb23d9b52975711276865441f Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 18:05:13 +0000 Subject: [PATCH 08/12] docs(patterns): say upstream with Triton attention was not part of the start-up runs The start-up bullet said upstream with --attention-backend TRITON_ATTN "was not measured". The README section it links to describes a separate upstream-only check of that setting, so the bullet now says it was not part of these runs, matching the README. --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index 13a092912..90477b8ad 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -90,7 +90,7 @@ The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-st Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. -* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). Eugo's image picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not measured, and the lead against it is likely smaller. +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). Eugo's image picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not part of these runs, and the lead against it is likely smaller. * *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. From db32a571beea477a392823fc88bb6fcf5de6287c Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 18:22:49 +0000 Subject: [PATCH 09/12] docs(patterns): say the Eugo build in the start-up runs picked Triton attention The start-up bullet said Eugo's image picked Triton attention by itself. That is true of the build these runs used, but the section calls the measured image Eugo's next vLLM image (release pending), and the sentence could be read as that image's default. It now says the Eugo build in these runs picked Triton attention by itself, matching the README. No number changed. --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index 90477b8ad..a385d7626 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -90,7 +90,7 @@ The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-st Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. -* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). Eugo's image picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not part of these runs, and the lead against it is likely smaller. +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). The Eugo build in these runs picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not part of these runs, and the lead against it is likely smaller. * *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. From 5a7f9c9fe03d19fd9acf329a73a59b386313338e Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 19:29:56 +0000 Subject: [PATCH 10/12] docs(patterns): describe image comparison and scale out on the vLLM Ray Inference Server page The overview page left both optional features to the pattern README. A new section says what each one does, that both are off by default, that a comparison scales the inference server to zero pods for its run, and that neither has run on OpenShift yet, and it links the README sections. It says nothing that the README on main does not say, and it adds no partner result. --- content/patterns/vllm-ray-inference-server/_index.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index a385d7626..39b40b98f 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -85,6 +85,16 @@ The pattern deploys the following applications: The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. +[id="compare-scale-out-vllm-ray"] +== Image comparison and scale out + +The pattern has two optional features, and both are off by default: + +* *Image comparison.* A job in the cluster runs two server images through the same benchmark, on the same GPU, with the same model and requests. The job records the time to ready, the time to first answer, and, at each load level, the time to first token, the inter-token latency, the output tokens per second, and the output tokens per joule of GPU energy. Each round also starts an identical copy of each image. A rule that is fixed before the first launch counts a difference between the images only when it is larger than the gap between two starts of one image and significant in an exact sign test over the paired rounds. Each case ends with a verdict, such as FASTER, SLOWER, or TIE, or with NO VERDICT and a reason. A run scales the inference server to zero pods for its whole length. +* *Scale out.* Ray Serve can add vLLM replicas inside the server pod. With `global.scaleOut.mode: keda`, the Red{nbsp}Hat Custom Metrics Autoscaler, the supported build of KEDA, adds whole server pods when the vLLM count of waiting requests per pod rises above a set limit, and removes them after a cooldown. The image comparison also has a scale-out mode: at a fixed request rate, it starts a second replica of each image at a fixed time, and measures how soon that replica gives its first answer and what share of the requests meets the latency goals. For now, this mode runs only with Docker on one machine, not in the cluster. + +No image comparison has run on {ocp} yet, and neither autoscaler has added a real vLLM replica or pod yet. For the settings, the steps, and what has not been checked, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#compare-two-images[Compare two images] and link:https://github.com/eugo-inc/vllm-ray-inference-server#scale-out[Scale out] sections of the pattern README. + [id="partner-results-vllm-ray"] == Partner results: Eugo From 8963719d4bf38260d070df75e0e33bf73955ece6 Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 20:07:46 +0000 Subject: [PATCH 11/12] docs(patterns): give the vLLM Ray Inference Server start-up result for three upstream attention settings The start-up bullet of "Partner results: Eugo" gave one result, 20.2 seconds (22%) sooner than upstream with FlashInfer attention, from an earlier build, and a sentence saying the lead against upstream with Triton attention was likely smaller. It now gives the result against each of three upstream attention settings, measured on Eugo's next vLLM test build, as the pattern README does: 10.9 seconds (18%) sooner than FlashInfer with TRT-LLM attention off, 12.0 seconds (20%) sooner than Triton and 16.4 seconds (25%) sooner than FlashInfer, in every paired round, with p = 0.004 or lower after Holm's correction. The sentence about Triton attention is gone, since that setting is now measured. The intro now says Eugo measured its next vLLM test build instead of its next vLLM image. --- content/patterns/vllm-ray-inference-server/_index.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index 39b40b98f..aa92eae55 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -98,9 +98,9 @@ No image comparison has run on {ocp} yet, and neither autoscaler has added a rea [id="partner-results-vllm-ray"] == Partner results: Eugo -Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. +Eugo measured its next vLLM test build (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. -* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). The Eugo build in these runs picked Triton attention by itself; upstream with `--attention-backend TRITON_ATTN` was not part of these runs, and the lead against it is likely smaller. +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token sooner than the upstream image in every paired round with each of three upstream attention settings: a median of 10.9 seconds (18%) sooner with FlashInfer and its TRT-LLM attention turned off (11 of 11 rounds), 12.0 seconds (20%) sooner with `--attention-backend TRITON_ATTN` (9 of 9), and 16.4 seconds (25%) sooner with `--attention-backend FLASHINFER` (10 of 10), with p = 0.004 or lower after Holm's correction. Both sides loaded the weights eagerly, and the Eugo build chose its attention kernel itself. * *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. From 53fb462e7449320b8b1ccfe77c56fcc07f40f8f4 Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 15 Nov 2026 20:07:46 +0000 Subject: [PATCH 12/12] docs(patterns): count failed starts on the same build as the start-up result The "Starts that succeed first time" bullet (4 of 12 upstream starts and none of 28 Eugo starts) came from the earlier build. It now comes from the same runs as the start-up result, as in the pattern README: the vLLM start-up memory check stopped 15 of 45 upstream starts and none of 11 Eugo starts. --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index aa92eae55..db675bf12 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -101,7 +101,7 @@ No image comparison has run on {ocp} yet, and neither autoscaler has added a rea Eugo measured its next vLLM test build (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. * *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token sooner than the upstream image in every paired round with each of three upstream attention settings: a median of 10.9 seconds (18%) sooner with FlashInfer and its TRT-LLM attention turned off (11 of 11 rounds), 12.0 seconds (20%) sooner with `--attention-backend TRITON_ATTN` (9 of 9), and 16.4 seconds (25%) sooner with `--attention-backend FLASHINFER` (10 of 10), with p = 0.004 or lower after Holm's correction. Both sides loaded the weights eagerly, and the Eugo build chose its attention kernel itself. -* *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. +* *Starts that succeed first time.* The vLLM start-up memory check stopped 15 of 45 upstream starts and none of 11 Eugo starts. Eugo's vLLM carries a fix that mitigates this. For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README.