Expose num_gpu as a first-class argument on generate/chat - #720
Open
shreesha345 wants to merge 1 commit into
Open
Expose num_gpu as a first-class argument on generate/chat#720shreesha345 wants to merge 1 commit into
shreesha345 wants to merge 1 commit into
Conversation
Adds a documented, type-checked num_gpu keyword argument to
Client/AsyncClient generate() and chat(), merged into options via a
new _merge_options() helper so it composes with any user-supplied
options dict/Options instance.
Previously num_gpu was only reachable via options={'num_gpu': N},
which was undiscoverable and easy to confuse with "number of GPUs"
rather than "layers offloaded to GPU". Documents the distinction and
the multi ollama serve / multi Client(host=...) pattern for hard
GPU isolation in the README and a new examples/gpu-selection.py.
Fixes ollama#161, ollama#603
Relates to ollama#62
shreesha345
force-pushed
the
feat/gpu-selection-num-gpu
branch
from
August 17, 2026 12:13
b3e352d to
989a5de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
num_gpukeyword argument toClient/AsyncClientgenerate()andchat()(sync + async), merged intooptionsvia a new_merge_options()helper so it composes cleanly with any user-suppliedoptions=dict/Optionsinstance.num_gpuwas only reachable viaoptions={'num_gpu': N}— undiscoverable, and easily confused with "number of GPUs" rather than its actual meaning: "layers offloaded to GPU". This PR documents that distinction explicitly.README.mdand a newexamples/gpu-selection.py, both explainingnum_gpuand the multi-ollama serve/ multi-Client(host=...)pattern for hard GPU isolation across processes on a shared multi-GPU server (device selection itself is a server-side concern controlled byCUDA_VISIBLE_DEVICESperollama serveprocess — there's no per-request "which physical GPU" field in the REST API).Closes #161, #603
Relates to #62
Test plan
pytest tests/test_client.py -k num_gpu -v— 3 new tests pass (test_client_chat_with_num_gpu,test_client_chat_with_num_gpu_merges_options,test_client_generate_with_num_gpu)pytest ollama tests— full suite passes except 8 pre-existing, unrelated Windows-only failures (tempfile permission / image serialization edge cases), confirmed viagit stashto exist onmainprior to this changeruff check/ruff format --check— clean