CI-1235: Add workflow debugging feature by sminot · Pull Request #203 · CirroBio/Cirro-SDK-Python

sminot · 2026-04-17T21:09:22Z

Adds the ability to inspect and debug failed Nextflow workflow executions directly from the Cirro SDK and CLI.

What's new for users

cirro debug — a new CLI command to inspect a failed dataset. Prints the last 50 lines of the execution log, identifies the primary failed task automatically, and shows its script, log, input files, and output files. Pass -i/--interactive to enter a menu-driven exploration mode where you can browse inputs and outputs, drill into source tasks, and read file contents directly in the terminal (as text, JSON, or CSV).

CLI

Command	Description
`cirro debug --project <name> --dataset <name>`	Non-interactive: print task debug summary, recurse through input chain
`cirro debug -i`	Interactive: menu-driven task and file exploration

New SDK classes

DataPortalTask (cirro/sdk/task.py)

Represents a single task from a Nextflow workflow execution. Metadata is read from the WORKFLOW_TRACE artifact; logs and files are fetched on demand.

Attribute	Description
`name`, `status`, `exit_code`, `hash`, `work_dir`, `task_id`	Trace-derived metadata
`logs`	Task stdout/stderr (via execution API, with `.command.log` fallback)
`script`	The shell script Nextflow ran (`.command.sh`, with log-artifact fallback)
`inputs`	`WorkDirFile` list parsed from `.command.run`, each linked to its `source_task`
`outputs`	Non-hidden files in the task's S3 work directory

WorkDirFile (cirro/sdk/task.py)

Represents a file in a Nextflow S3 work directory or dataset staging area.

Attribute / Method	Description
`name`, `size`, `source_task`	File metadata
`read()`, `readlines()`	Read as text (supports gzip)
`read_json()`	Parse as JSON
`read_csv()`	Parse as a Pandas DataFrame (auto-infers `.gz`/`.bz2`/`.xz`/`.zst` compression)

Additions to existing SDK classes

Addition	Description
`DataPortalDataset.executor`	Executor type (`NEXTFLOW`, `CROMWELL`) for the dataset's process
`DataPortalDataset.logs`	Top-level execution log via Cirro API (CloudWatch)
`DataPortalDataset.tasks`	Full list of `DataPortalTask` objects from the trace artifact
`DataPortalDataset.primary_failed_task`	Auto-identifies the root-cause failed task by cross-referencing exit codes with the execution log; returns `None` gracefully for non-Nextflow executors, empty traces, or successful runs

Internal changes

FileAccessContext.scratch_download() — new classmethod for accessing Nextflow scratch bucket files
FileService._get_scratch_read_credentials() — cached credential fetch for scratch bucket reads
Null-guard added in ExecutionService for resp.events when log responses are empty

sonarqubecloud · 2026-05-14T00:42:35Z

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
44.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Add # NOSONAR to 15 broad except Exception patterns and 2 cognitive complexity hotspots (run_debug, _file_menu) introduced in this PR. All catches are intentional resilience patterns that return defaults when S3/file access fails. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

_load_tasks_from_api() calls get_tasks_for_execution and maps the API Task model (name, status, native_job_id) onto DataPortalTask trace_row dicts. _load_tasks_cromwell now uses this instead of raising NotImplementedError. _load_tasks for Nextflow falls back to the API when the WORKFLOW_TRACE artifact is unavailable (e.g. dataset still running or artifact upload failed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nathanthorpe · 2026-05-23T00:15:18Z

            client=self._api_client
        )

+        if resp is None or resp.events is None:


I don't think this is ever the case, we shouldn't have the checks. If we do need it then the backend needs to be fixed.

nathanthorpe · 2026-05-23T00:16:20Z

+    from cirro.sdk.task import DataPortalTask
+
+
+def parse_inputs_from_command_run(content: str) -> List[str]:


can be removed right?

nathanthorpe · 2026-05-23T00:17:06Z

                    df.sort_index(axis=1).to_csv(index=False)
                )

+    @unittest.skipIf(os.environ.get('CI') == 'true', "Skipping S3 integration test in CI")


these tests should be run by CI

- Update poetry.lock to pin cirro_api_client 1.5.0 - Switch from custom TaskFilesResponse/TaskFile to official GetTaskFilesResponse/FileEntry - Add command_line and log_location properties to DataPortalTask (exposed from Task model) - Fix hash property (not in 1.5.0 Task model; always returns '') - Update _build_inputs/_build_outputs to read uri/sourceTask from FileEntry.additional_properties Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Move all debug-related code from controller.py into cirro/cli/debug.py - Remove null checks from get_task_logs (backend guarantees non-null response) - Remove parse_inputs_from_command_run from nextflow_utils.py and its tests - Remove CI skip decorator from test_preprocess.py so S3 tests run in CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Restore Path import removed during debug extraction (used in run_upload_reference) - Remove unused List/Optional typing imports from controller.py - Remove unused top-level ask_project/ask_dataset imports shadowed by local imports - Replace _BINARY_EXTENSIONS blacklist with _TEXT_EXTENSIONS whitelist in debug.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Fix ImportError: run_debug imported from wrong module in cli/__init__.py - Add dataset_id param to FileAccessContext.scratch_download (was TypeError) - Guard get_task_logs against None response, matching get_execution_logs - Fix RUNNING check skipped on name-lookup failure in debug.py (capture original value before ID resolution) - Apply .lstrip('/') to S3 key in get_file_stats, matching get_file_from_path - Move _read_token_cache/_scratch_token_cache/_get_token_lock to instance level in FileService (was class-level, cross-contaminating credentials across instances) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sonarqubecloud · 2026-05-27T04:51:25Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
45.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

…th is the full uri

sminot and others added 6 commits April 17, 2026 14:08

Move over debug workflow branch (#202)

5331d3d

Don't catch naked exceptions

bbabc29

cirro debug with no arguments -> --help

c5b6f6b

Update test for 404

a2b5913

Update exceptions

7858b83

Tests

02486a9

sminot changed the title ~~Add workflow debugging feature~~ CI-1235: Add workflow debugging feature Apr 23, 2026

nathanthorpe requested a review from a team May 14, 2026 00:08

scratch file access context does not depend on dataset id

0ad527c

sminot and others added 6 commits May 22, 2026 09:57

Fix doc string

7d757ce

Use API for tasks in any case

3eda282

Notebook example for debug

ccca935

Fix ext

16a369d

sminot requested a review from nathanthorpe May 22, 2026 20:37

sminot and others added 4 commits May 22, 2026 13:43

Fix docstring for logs

dcbc7b2

Address SonarQube issues

cff2a0c

Use execution API for task details and files

1f43c87

Require cirro_api_client 1.5.0 for task files endpoint

8ddffc6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nathanthorpe reviewed May 23, 2026

View reviewed changes

Comment thread cirro/cli/controller.py Outdated

nathanthorpe reviewed May 23, 2026

View reviewed changes

sminot and others added 5 commits May 26, 2026 11:34

Regenerate poetry.lock to match pyproject.toml

35055aa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Specify file extensions to include, not binary to exclude

b2ec1dd

sminot and others added 2 commits May 26, 2026 13:45

don't check for resp.events

2e29848

nathanthorpe added 5 commits May 26, 2026 21:55

move nextflow utils

5d4117d

this only tests wiring

6b54411

move file reading methods over to mixin, remove 'prefer uri' since pa…

cc21cc5

…th is the full uri

export DataPortalFile so it shows up in API docs in the main page

fc4b29f

remove test

225a7ca

nathanthorpe approved these changes May 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI-1235: Add workflow debugging feature#203

CI-1235: Add workflow debugging feature#203
sminot wants to merge 29 commits into
mainfrom
debug-workflow

sminot commented Apr 17, 2026

Uh oh!

sonarqubecloud Bot commented May 14, 2026

Uh oh!

Uh oh!

nathanthorpe May 23, 2026

Uh oh!

nathanthorpe May 23, 2026

Uh oh!

nathanthorpe May 23, 2026

Uh oh!

sonarqubecloud Bot commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		from cirro.sdk.task import DataPortalTask


		def parse_inputs_from_command_run(content: str) -> List[str]:

Conversation

sminot commented Apr 17, 2026

Uh oh!

sonarqubecloud Bot commented May 14, 2026

Quality Gate passed

Uh oh!

Uh oh!

nathanthorpe May 23, 2026

Choose a reason for hiding this comment

Uh oh!

nathanthorpe May 23, 2026

Choose a reason for hiding this comment

Uh oh!

nathanthorpe May 23, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented May 27, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants