Skip to content

Add x/followers domain skill (followers/profile scraping)#397

Open
MoayadAlismail wants to merge 1 commit into
browser-use:mainfrom
MoayadAlismail:add-x-followers-skill
Open

Add x/followers domain skill (followers/profile scraping)#397
MoayadAlismail wants to merge 1 commit into
browser-use:mainfrom
MoayadAlismail:add-x-followers-skill

Conversation

@MoayadAlismail
Copy link
Copy Markdown

@MoayadAlismail MoayadAlismail commented May 31, 2026

Adds agent-workspace/domain-skills/x/followers.md — a playbook for reading other users' followers/following and profile data on X.

Covers:

  • Auth check via ct0+twid (auth_token is HttpOnly) and the login-wall redirect.
  • The public follower-list pagination cap (observed 73 unique on /followers for a large account, plus ~23 from /verified_followers → ~85–90 unique) — it's a per-request limit, not a soft-block.
  • DOM extraction via [data-testid="UserCell"] (preferred over GraphQL), with the small-step (~800px / 1.4s) scroll pattern needed because X virtualizes the list.
  • Stable profile-page selectors, GraphQL capture notes, and gotchas.
  • Shared-daemon tab-stealing failure mode + the Target.createTarget(url=...) / js(..., target_id=...) pinning fix (hit live when another agent shared the daemon).
  • Explicit "don't collect avatars" guidance (no face-based filtering).

🤖 Generated with Claude Code


Summary by cubic

Adds agent-workspace/domain-skills/x/followers.md, a practical guide for scraping X followers/following and profile data via the DOM. It clarifies auth checks, the real pagination cap, reliable selectors, and a fix for shared-daemon tab stealing.

  • New Features
    • Auth detection via ct0 + twid and the login-wall redirect check.
    • Public list cap documented: ~73 on /followers plus ~20 from /verified_followers (~85–90 unique); following similar.
    • DOM-first approach with stable selectors and small-step scrolling to avoid virtualization misses.
    • GraphQL capture tips (required headers, cursors) and note that replaying does not bypass the cap.
    • Shared-daemon fix using Target.createTarget(...) + js(..., target_id=...); includes guidance to avoid collecting avatars.

Written for commit 1281796. Summary will update on new commits.

Review in cubic

Observed 73 unique on /followers (not ~50-60) + 23 verified = ~89 on a
large account. Add the concurrent-daemon tab-stealing failure mode and
the Target.createTarget(url=...) + js(target_id=...) pinning fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="agent-workspace/domain-skills/x/followers.md">

<violation number="1" location="agent-workspace/domain-skills/x/followers.md:67">
P1: Follower-harvesting loop uses `prev` before initialization, causing a `NameError` on the first iteration.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

for c in json.loads(js(EXTRACT)):
if c.get("at") and c["at"] not in seen:
seen[c["at"]] = c
new = len(seen) - prev; prev = len(seen)
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot May 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Follower-harvesting loop uses prev before initialization, causing a NameError on the first iteration.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agent-workspace/domain-skills/x/followers.md, line 67:

<comment>Follower-harvesting loop uses `prev` before initialization, causing a `NameError` on the first iteration.</comment>

<file context>
@@ -0,0 +1,105 @@
+    for c in json.loads(js(EXTRACT)):
+        if c.get("at") and c["at"] not in seen:
+            seen[c["at"]] = c
+    new = len(seen) - prev; prev = len(seen)
+    no_growth = no_growth + 1 if new == 0 else 0
+    js("window.scrollBy(0, 800)")
</file context>
Fix with Cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant