fix facebook feed too_short zero rows

[lawmight]

Description

Live smoke after PR #4: articles=2, mainTextLength=972, rejected=3, too_short=2, decoy_content=1 — Adolph decoy correctly caught, but two short role=article chrome stubs blocked extraction while real posts may live in parent FeedUnits or below the fold.

Type of Change

• 🐛 Bug fix

Changes

• resolveArticleContainer: promote short role=article headers to parent FeedUnit when post menu/author/permalink evidence exists
• feedUnitContainers: explicit FeedUnit candidates in extraction pipeline
• Hydration-aware scroll: incremental scrollBy (12 passes, 1.2s sleep); continue when needsMoreFeedHydration (too_short + mainTextLength≥400 + post markers)
• Diagnostics: textLength on rejection samples; too_short(len=N) in error hints; chrome-stub note when stubs coexist with decoys
• Error routing: too_short chrome + decoy-only substantive rejections → anti-scrape error explaining stubs vs decoy posts

Checklist

• npm test -- clis/facebook/ — 117 passed (French, Adolph/decoy, messenger, short-article fixture)

Post-merge verification

opencli facebook feed --limit 5 on tom.coustols should return LVLUP (or similar) OR fail with too_short(len=…) chrome stub explanation + decoy — not a bare no rows with unexplained shorts.

  

Summary by Sourcery

Restore reliable Facebook feed extraction when short chrome placeholders and decoy posts coexist with real feed content.

Bug Fixes:

• Fix Facebook feed extraction returning zero rows when short article chrome stubs obscure readable posts.
• Improve error classification and diagnostics for feeds containing chrome stubs and anti-scrape decoy content.

Enhancements:

• Recognize parent FeedUnits as extraction candidates and continue hydration-aware incremental scrolling when readable posts may still be below the fold.

Tests:

• Add coverage for short article chrome promotion, rejection text lengths, hydration-aware scrolling, and anti-scrape error reporting.

---

Summary by cubic

Fixes Facebook feed extraction returning zero rows when short role=article chrome stubs (Stories, composer) coexist with decoy posts. The extractor now promotes those stubs to their parent FeedUnit when post evidence exists, so real posts like LVLUP are extracted instead of being rejected as too short.

Changes

• Scrolls incrementally (12 passes, 1.2s sleep) and continues when hydration is detected (too_short + mainTextLength≥400 + post markers).
• Adds feedUnitContainers as explicit extraction candidates.
• Records textLength on rejection samples and includes too_short(len=N) in error hints.
• Routes too_short chrome + decoy-only rejections to an anti-scrape error explaining stubs vs decoy posts.

^{Written for commit 661236c. Summary will update on new commits.}

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Fixes zero-row Facebook feed extraction by promoting short article chrome to parent FeedUnits, continuing incremental scrolling during likely lazy hydration, and providing diagnostics that distinguish chrome stubs from decoy or anti-scrape content; regression tests cover each behavior.

Sequence diagram for promoting chrome stubs to FeedUnits

sequenceDiagram
    participant Extractor
    participant FeedDOM
    participant FeedUnit
    participant Diagnostics

    Extractor->>FeedDOM: querySelectorAll role=article
    FeedDOM-->>Extractor: short article header
    Extractor->>FeedUnit: resolveArticleContainer
    FeedUnit-->>Extractor: parent with post body and post markers
    Extractor->>Extractor: extractPost
    Extractor->>Diagnostics: record too_short samples for unresolved stubs
    Extractor-->>Extractor: return readable post rows

Loading

Flow diagram for Facebook feed extraction with hydration recovery

flowchart TD
    A[Load Facebook feed] --> B[Incremental scrollBy]
    B --> C[Extract feed candidates]
    C --> D{Rows found?}
    D -->|Yes| E[Return feed rows]
    D -->|No| F{"needsMoreFeedHydration(payload)?"}
    F -->|Yes| B
    F -->|No after stalled passes| G[Classify rejection diagnostics]
    G --> H[Raise readable error or anti-scrape error]

Loading

File-Level Changes

Change	Details	Files
Promote short Facebook article chrome to real FeedUnit extraction candidates and add explicit FeedUnit discovery.	Resolve short article nodes through FeedUnit or aria-posinset parents when post evidence is present. Deduplicate and combine promoted article, FeedUnit, action-anchored, and fallback candidates. Avoid treating promoted chrome headers as independent rejection candidates.	clis/facebook/feed.js
Make feed loading continue through lazy hydration when visible markup has substantive content but no extractable rows.	Replace large scroll jumps with bounded incremental scrolling across 12 passes with 1.2-second waits. Continue scrolling when long page text, too_short rejections, and post markers indicate hydration may still be incomplete. Reset stalled-pass detection while hydration evidence remains.	clis/facebook/feed.js
Improve rejection diagnostics and classify chrome-plus-decoy failures as anti-scrape responses.	Record rejected node text lengths and include them in rejection summaries. Explain short Stories/composer stubs separately from substantive decoy posts. Route feeds containing only decoy substantive candidates and chrome stubs to a targeted anti-scrape error.	clis/facebook/feed.js
Add regression coverage for short article stubs, FeedUnit promotion, hydration scrolling, diagnostics, and error routing.	Add a fixture with Stories/composer stubs, a decoy FeedUnit, and a readable LVLUP FeedUnit. Test textLength reporting, hydration continuation criteria, and anti-scrape hints. Verify the promoted readable post is extracted while decoys and stubs are handled correctly.	clis/facebook/__fixtures__/feed-short-article-chrome.html clis/facebook/feed.test.js

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Approved.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook