fix(api): stop users.list pagination on a repeated cursor

[SebTardif]

What Problem This Solves

Fixes an issue where users running slacrawl sync (or doctor/repair paths that load the member directory) would hang until the process was killed when Slack kept returning the same users.list next_cursor. Channel, history, thread, and DM pagers already stop on a repeated token. The member walk still handed the full walk to slack-go, which follows next_cursor until it is empty.

Why This Change Was Made

A stuck Slack cursor is a hang, not a slow page. #159 added a seen-map on conversations.list, conversations.history, conversations.replies, and DM listing. getUsers still called GetUsersContext as one shot, so a repeated member-directory cursor kept slacrawl sync requesting forever (each request still had the 60s HTTP timeout). This pages users.list in slacrawl with the same fail-closed check.

The unguarded member walk dates to 588584e (feat: sync post-bootstrap updates, 2026-03-08). It has been present for 180 days. Same class as openclaw/notcrawl#102 and openclaw/discrawl#181.

User Impact

A wedged users.list pager now stops with a clear users.list repeated cursor error instead of spinning until someone kills the sync. Successful pages (empty cursor, or a new cursor) are unchanged.

Evidence

terminal output from the patched Slack client. An httptest users.list pager that always returns next_cursor=stuck now fails closed after two pages.

Before this patch the same pager never returned. A 5s deadline expired with:

$ GOWORK=off go test ./internal/slackapi/ -count=1 -timeout 15s -run TestGetUsersRejectsRepeatedCursor
Error: Error "Post \"http://127.0.0.1:59290/users.list\": context deadline exceeded" does not contain "users.list repeated cursor \"stuck\""
FAIL    github.com/openclaw/slacrawl/internal/slackapi  5.258s

After this patch (2f79bb80f4f2) the same pager returns immediately:

$ GOWORK=off go test ./internal/slackapi/ -count=1 -timeout 15s -run TestGetUsersRejectsRepeatedCursor -v
=== RUN   TestGetUsersRejectsRepeatedCursor
    api_test.go:1953: getUsers stuck next_cursor: users.list repeated cursor "stuck"
--- PASS: TestGetUsersRejectsRepeatedCursor (0.00s)
ok      github.com/openclaw/slacrawl/internal/slackapi  0.255s

Two distinct pages then an empty cursor still collect both members:

$ GOWORK=off go test ./internal/slackapi/ -count=1 -timeout 15s -run TestGetUsersWalksDistinctCursors -v
=== RUN   TestGetUsersWalksDistinctCursors
--- PASS: TestGetUsersWalksDistinctCursors (0.00s)
ok      github.com/openclaw/slacrawl/internal/slackapi  0.253s

Real behavior proof

• Behavior or issue addressed: slacrawl sync (and doctor/repair that call getUsers) hung when Slack repeated a users.list next_cursor, because slack-go GetUsersContext walks pages with no seen-set.
• Real environment tested: Windows 11, Go 1.27.0, slacrawl at 2f79bb80f4f2 on branch fix/users-list-seen-token.
• Exact steps or command run after this patch: GOWORK=off go test ./internal/slackapi/ -count=1 -timeout 15s -run TestGetUsersRejectsRepeatedCursor -v and the matching distinct-cursor command against an httptest Slack users.list endpoint.
• Evidence after fix: terminal output from the patched Slack client is in Evidence above. The stuck pager returns users.list repeated cursor "stuck" after two users.list POSTs instead of waiting out the deadline.
• Observed result after fix: getUsers returns users.list repeated cursor "stuck" in 0.00s after two pages. Two distinct cursors then an empty cursor still return both members.
• What was not tested: a live Slack workspace that actually repeats users.list next_cursor. Rate-limit retry on a single users.list page was not re-run here; that path uses the same retry helper as the other Slack pagers.

Related: #159

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed September 5, 2026, 3:23 PM ET / 19:23 UTC.

ClawSweeper review

What this changes

Stops Slack member-directory pagination when a cursor repeats, retries individual rate-limited pages, and adds regression coverage and documentation.

Merge readiness

✅ Ready for maintainer review

This PR remains useful: current main and v0.8.6 still lack the member-pagination guard. The focused implementation and supplied production-path HTTP proof support landing it; no blocking defect was found.

Priority: P2
Reviewed head: 0eac09dbb2afa211374485f52e3f0839f013d0bd

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, maintainable fix with production-boundary HTTP proof and useful cycle and retry coverage.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (terminal): The supplied Windows terminal transcript exercises production getUsers through the real slack-go HTTP client against an injected repeated-cursor endpoint: it stops after two requests, while distinct pages still return both users. The production file is unchanged from the demonstrated revision, satisfying the internal-reliability proof boundary.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The supplied Windows terminal transcript exercises production getUsers through the real slack-go HTTP client against an injected repeated-cursor endpoint: it stops after two requests, while distinct pages still return both users. The production file is unchanged from the demonstrated revision, satisfying the internal-reliability proof boundary.
Evidence reviewed	8 items	Repository policy and identity: The origin identifies openclaw/slacrawl. No root or nested AGENTS.md or maintainer notes were found. CONTRIBUTING.md and SPEC.md support focused, tested changes and the existing workspace-user snapshot contract. Current-main and release gap: Both fetched main and v0.8.6 still call GetUsersContext inside one retry operation, without a slacrawl cursor guard. The release was inspected directly; this requested fix is not present there. Released implementation: The latest supplied release retains the same whole-walk GetUsersContext implementation.
Findings	None	None.
Security	None	None.

How this fits together

Slacrawl mirrors Slack data into a local SQLite archive. Its member-directory fetch feeds user records and direct-message names into sync, so an endless page walk prevents sync from completing.

flowchart TD
  A[Slack sync] --> B[Fetch member directory]
  B --> C[Slack HTTP page]
  C --> D{Page result}
  D -->|Rate limited| C
  D -->|New cursor| C
  D -->|Repeated cursor| E[Return clear error]
  D -->|No next cursor| F[Store users and name DMs]

Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test growth	production +31/-3; tests +105/-0	The bounded pagination loop justifies the production growth, with four focused regression cases.

Technical review

Best possible solution:

Keep member pagination locally bounded while preserving normal member collection, cancellation, and page-level rate-limit retries.

Do we have a high-confidence way to reproduce the issue?

Yes: a users.list endpoint that repeatedly returns the same nonempty cursor exercises the affected production fetch. Current-main source retains the unguarded call, and the supplied before/after transcript demonstrates the failure and fix; this reviewer did not execute it.

Is this the best way to solve the issue?

Yes: owning the page loop is a focused extension of the existing Slack pagination pattern, and per-page retries preserve completed progress without introducing configuration or storage changes.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against b9ff7d91485f.

Labels

Label justifications:

• P2: Repairs a bounded Slack sync hang triggered by repeated member-directory cursors.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The supplied Windows terminal transcript exercises production getUsers through the real slack-go HTTP client against an injected repeated-cursor endpoint: it stops after two requests, while distinct pages still return both users. The production file is unchanged from the demonstrated revision, satisfying the internal-reliability proof boundary.
• proof: sufficient: Contributor real behavior proof is sufficient. The supplied Windows terminal transcript exercises production getUsers through the real slack-go HTTP client against an injected repeated-cursor endpoint: it stops after two requests, while distinct pages still return both users. The production file is unchanged from the demonstrated revision, satisfying the internal-reliability proof boundary.

Evidence

What I checked:

• Repository policy and identity: The origin identifies openclaw/slacrawl. No root or nested AGENTS.md or maintainer notes were found. CONTRIBUTING.md and SPEC.md support focused, tested changes and the existing workspace-user snapshot contract. (CONTRIBUTING.md:1, 0eac09dbb2af)
• Current-main and release gap: Both fetched main and v0.8.6 still call GetUsersContext inside one retry operation, without a slacrawl cursor guard. The release was inspected directly; this requested fix is not present there. (internal/slackapi/api.go:947, b9ff7d91485f)
• Released implementation: The latest supplied release retains the same whole-walk GetUsersContext implementation. (internal/slackapi/api.go:947, dd2608b91f4f)
• Production boundary and dependency signal: Sync calls getUsers before storing member records; NewWithOptions constructs the real slack-go client with the supplied HTTP client. The patch directly uses slack-go pagination, pinned to v0.29.0 in go.mod. Repeated cursors return an error before partial member results reach storage; normal completion retains the existing caller contract. (internal/slackapi/api.go:947, 0eac09dbb2af)
• After-fix HTTP proof: The captured PR body, sourceRevision 04399596c5f6054ef3fd52d065811adfb56e146c2d9446490f64a9c5914c5120, records Windows 11 execution at 2f79bb8. The inspected harness calls production getUsers through the real slack-go HTTP client against an injected users.list endpoint. Before fixing, the deadline expired; afterward, two requests produce the repeated-cursor error immediately. A separate run returns both members across distinct pages. This is production-boundary fault proof, rather than a mocked transport client. (internal/slackapi/api_test.go:1935, 0eac09dbb2af)
• Re-review continuity and coverage: The production file has identical blob 31d96d62318c633ec8cac6a2d7228d51ebf0ca8c at the previously reviewed head and current head, preserving applicability of the supplied proof. Current source includes four focused cases: repeated cursor, distinct cursors, multi-token cycle, and retry of only the rate-limited page. The previous review projection contains no findings or published rank-up moves. (internal/slackapi/api_test.go:1935, 0eac09dbb2af)

Likely related people:

• SebTardif: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)

• reviewed 2026-09-05T05:02:40.825Z sha 2f79bb8 :: needs maintainer review before merge. :: none
• reviewed 2026-09-05T13:00:42.081Z sha 2f79bb8 :: needs maintainer review before merge. :: none

[steipete]

Maintainer verification at 0eac09dbb2afa211374485f52e3f0839f013d0bd:

• Built and ran the real cmd/slacrawl sync command against a synthetic loopback Slack HTTP server. A temporary Go build overlay set the existing App.apiURL test seam; the pagination implementation and CLI execution path were unchanged. No Slack credentials or workspace data were used.
• Before: the main-branch pager made 38 member requests in two seconds and still required interruption (exit 130).
• After: a repeated cursor exited 1 after exactly two requests; an A/B/A cycle exited 1 after three requests, both with users.list repeated cursor. Advancing pages exited 0 and stored both members. A 429 on page two retried that same page, made three requests total, and stored each member once.
• Added regression coverage for longer cursor cycles and page-local rate-limit retry. Updated command documentation and Unreleased credit for @SebTardif.
• Full local GOWORK=off GOMAXPROCS=2 GOFLAGS=-p=2 make check passed, including tests, vet, vulnerability/dead-code checks, stock CLI smoke, and the four-target release snapshot.
• Independent Codex branch autoreview against origin/main: scoped-clean at P0–P2.
• Exact-head CI: https://github.com/openclaw/slacrawl/actions/runs/33986803836

Prepared for squash merge; left open for the owner-authorized landing step. This proves deterministic fault handling with an injected HTTP endpoint, not a live Slack service returning a bad cursor.

[steipete]

Maintainer review: reproduced the users.list cursor loop with the built CLI, verified rejection + rate-limit retry; exact-head CI green; autoreview clean.