[Bug]: Multinode TP Errors when TP > number GPUS on 1 node, but <= world size

[zaristei]

Your current environment

vllm_out.log

The output of python collect_env.py

Your output of `python collect_env.py` here

2x GB200 nodes, vllm 0.11.2

🐛 Describe the bug

On head node:
ray start --head --num-gpus 4
Copy output command for worker node
ray start --address='<IP>:<PORT>' --num-gpus 4
Ensure 8 gpus are visible on ray status.
On head node:
vllm serve meta-llama/Llama-3.3-70B-Instruct --gpu-memory-utilization 0.9 --served-model-name llama3.3-70b --tensor-parallel-size 8 --pipeline-parallel-size 1 --data-parallel-size 1 --max-model-len 2048 --port $port

Output log (portions redacted)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[zaristei]

Proposed fix: Change the error in parallel.py into a warning when Ray is detected. This TP pattern was supported behavior prior to the logging changes introduced in #25775

[curious-broccoli]

posting part of your logs here so it can be found via search:

�[1;36m(APIServer pid=1938)�[0;0m pydantic_core._pydantic_core.ValidationError: 1 validation error for ParallelConfig
�[1;36m(APIServer pid=1938)�[0;0m   Value error, Tensor parallel size (8) cannot be larger than the number of available GPUs (4). [type=value_error, input_value=ArgsKwargs((), {'pipeline...'_api_process_rank': 0}), input_type=ArgsKwargs]

[longqu]

same problem here, actually this bug starts with v0.11.1.

When starts DeepSeek with 2 nodes H100(8 cards per node) using ray cluster:

   vllm serve \
    /root/.cache/huggingface \
    --served-model-name "$MODEL_NAME" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 16 \
    --max-model-len 65536 \
    --gpu_memory_utilization 0.9 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max_num_batched_tokens 65536 \
    --trust-remote-code

(This works perfectly until v0.11.0 included)

Here is the output of the error log:

---

(APIServer pid=452) The argument trust_remote_code is to be used with Auto classes. It has no effec
t here and is ignored.
(APIServer pid=452) Traceback (most recent call last):
(APIServer pid=452) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=452) sys.exit(main())
(APIServer pid=452) ^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", li
ne 73, in main
(APIServer pid=452) args.dispatch_function(args)
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", l
ine 60, in cmd
(APIServer pid=452) uvloop.run(run_server(args))
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 96, in
run
(APIServer pid=452) return __asyncio.run(
(APIServer pid=452) ^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=452) return runner.run(main)
(APIServer pid=452) ^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=452) return self._loop.run_until_complete(task)
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 48, in
wrapper
(APIServer pid=452) return await main
(APIServer pid=452) ^^^^^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_serve
r.py", line 2024, in run_server
(APIServer pid=452) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_serve
r.py", line 2043, in run_server_worker
(APIServer pid=452) async with build_async_engine_client(
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=452) return await anext(self.gen)
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_serve
r.py", line 195, in build_async_engine_client
(APIServer pid=452) async with build_async_engine_client_from_engine_args(
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=452) return await anext(self.gen)
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_serve
r.py", line 221, in build_async_engine_client_from_engine_args
(APIServer pid=452) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=452) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1
600, in create_engine_config
(APIServer pid=452) parallel_config = ParallelConfig(
(APIServer pid=452) ^^^^^^^^^^^^^^^
(APIServer pid=452) File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.p
y", line 121, in init
(APIServer pid=452) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_insta
nce=s)
(APIServer pid=452) pydantic_core._pydantic_core.ValidationError: 1 validation error for ParallelConf
ig
(APIServer pid=452) Value error, Tensor parallel size (16) cannot be larger than the number of avai
lable GPUs (8). [type=value_error, input_value=ArgsKwargs((), {'pipeline...'_api_process_rank': 0}), input_type=
ArgsKwargs]
(APIServer pid=452) For further information visit https://errors.pydantic.dev/2.12/v/value_error
srun: error: h800compute0001: task 0: Exited with exit code 1