tt-device-queue

One FIFO queue per physical Tenstorrent card. Jobs on queue 0 serialize with other jobs and resets on queue 0; queue 1 runs independently. One service owns both queues. There is no automatic card assignment or job migration.

The caller must match the queue number to the device used by their code. The queue does not inspect code, select hardware, or set TT_VISIBLE_DEVICES. For blackhole-py, code using Device(0) goes to queue 0 and code using Device(1) goes to queue 1.

cd ~/tenstorrent/blackhole-py
tt-device-queue run --device 0 --cwd "$PWD" -- 'python3 my_device0_script.py'
tt-device-queue run --device 1 --cwd "$PWD" -- 'python3 my_device1_script.py'

--device selects the queue and defaults to 0. Pass a complete shell program as one quoted argument to preserve assignments, pipes, redirects, and &&.

CLI

tt-device-queue --json queue --device 1 --cwd "$PWD" -- 'python3 script.py'
tt-device-queue job JOB_ID
tt-device-queue result JOB_ID
tt-device-queue logs JOB_ID --offset 0 --limit 16384
tt-device-queue queue-python --device 1 --cwd "$PWD" --wait script.py
tt-device-queue status
tt-device-queue kill JOB_ID

run and result return the command's exit status. --wait-timeout only stops the local wait; use kill to stop a job. Submission options include --timeout, --repeat, and repeatable --env NAME=VALUE. queue-python accepts - for stdin.

Global options (--host, --port, --client-id, --json) precede the subcommand. TT_DEVICE_URL, TT_DEVICE_PORT, and TT_QUEUE_CLIENT_ID override connection and diagnostic client defaults. Jobs run FIFO regardless of client identity.

Resets

tt-device-queue reset --device 0  # reset only card 0
tt-device-queue reset --device 1  # reset only card 1
tt-device-queue reset             # schedule both cards' resets
tt-device-queue reset JOB_ID      # target the card recorded on this job

A reset holds its queue and can interrupt its current job. Duplicate requests coalesce. A job-based request skips recovery if the card has already been reset since that job ran. Reset requests return immediately; check status for completion and reset jobs' logs for details.

Defaults use ~/tenstorrent/.venv:

• Reset: tt-smi -r /dev/tenstorrent/{device} --no_reinit.
• Health check: python3 <repo>/health_check.py {device}, which initializes only that physical card through tt_umd and reads its clock telemetry.

The explicit device path avoids confusing UMD logical IDs with physical IDs. --no_reinit prevents tt-smi from subsequently discovering all cards. The separate health check avoids the all-card scan performed by tt-smi -s. Override with TT_DEVICE_RESET_CMD, TT_DEVICE_HEALTH_CHECK_CMD, and optional TT_DEVICE_DEEP_RESET_CMD. Commands substitute {device}; reset commands without it receive the numeric ID as their final argument. Custom controls must touch only their target card.

Failed recovery marks only that card dead and fails its queued jobs. The other queue continues. Same-boot service restarts preserve dead state; a host reboot clears it. This is cooperative serialization, not hardware access enforcement.

Install and test

Confirm tt-device-queue status is idle, then run ./install.sh. It runs the unit tests, installs the CLI, and restarts the user service. After changing server/core code or the service unit, reinstall while idle.

PYTHONPATH=. .venv/bin/python3 -m unittest discover -s tests -p 'test_*.py'
PYTHONPATH=. .venv/bin/python3 tests/stress_queue.py

The service listens on 127.0.0.1:5741. HTTP submissions use POST /queue with cmd, cwd, and device_id (default "0"). /reset accepts device_id or job_id; neither means all cards. Other endpoints are /status, /kill, /job/<id>, /logs/<id>, and /result/<id>.

Storage and limits

SQLite in logs-v2/ stores durable metadata; bounded output files store logs. Queued jobs survive restarts with their queue target. Interrupted running jobs fail and are never replayed. Legacy unassigned queued jobs recover on queue 0. Only active jobs stay in memory. Persistence failures stop dispatch until the writes recover and appear in status.

Setting	Default	Environment variable
Queues	Numeric /dev/tenstorrent/* nodes	TT_DEVICE_IDS
Job timeout / maximum	3600 / 86400 seconds	TT_DEVICE_DEFAULT_TIMEOUT, TT_DEVICE_MAX_TIMEOUT
Total queued jobs	1000	TT_DEVICE_MAX_QUEUED_JOBS
Output per job	16 MiB	TT_DEVICE_MAX_LOG_BYTES
CLI result output	1 MiB	TT_DEVICE_RESULT_BYTES
Request body	1 MiB	TT_DEVICE_MAX_REQUEST_BYTES
HTTP handlers	16	TT_DEVICE_HTTP_WORKERS
Completed retention	30 days / 10,000 jobs	TT_DEVICE_RETENTION_DAYS, TT_DEVICE_MAX_COMPLETED_JOBS

Omitted or zero timeout selects the default; unbounded jobs are unsupported. The original pre-v2 database is incompatible; use logs-v2/.