4.x review follow-ups: isProbablyReaderable unit tests, docs for the content wrapper and rewrite

README.md

@@ -6,7 +6,7 @@ PHP port of *Mozilla's* **[Readability.js](https://github.com/mozilla/readabilit
 
 ![Screenshot](https://raw.githubusercontent.com/fivefilters/readability.php/assets/screenshot.png)
 
-Version 4.0 is a ground-up rewrite on PHP's native HTML parser ([Lexbor, included in PHP 8.4's DOM extension](https://blog.keyvan.net/p/parsing-html-with-php-84)), transcribed method-for-method from Readability.js v0.6.0. It parses HTML the way modern browsers do, needs no third-party parsing library, and is tested against Mozilla's own test corpus.
+Version 4.0 is a ground-up rewrite, produced using [Claude](https://claude.com/claude-code) (Anthropic's AI coding tool), to bring the code in line with the latest version of Readability.js (v0.6.0, transcribed method-for-method) and to take advantage of the new, faster native HTML parser introduced in PHP 8.4 ([Lexbor, included in the DOM extension](https://blog.keyvan.net/p/parsing-html-with-php-84)) and the new WHATWG URL parser introduced in PHP 8.5. It parses HTML the way modern browsers do, needs no third-party HTML parsing library, and is tested against Mozilla's own test corpus.
 
 **Original Developer**: Andres Rey
 
@@ -82,16 +82,29 @@ foreach ($article->contentElement->querySelectorAll('img[src]') as $img) {
 
 If you already have a `\Dom\HTMLDocument` (for example because you want to pre-process it), use `parseDocument()` instead of `parse()`. Note that the document is modified in place while the article is extracted.
 
-There is also a port of Mozilla's `isProbablyReaderable`, a quick check for whether it's worth running the full parse:
+### Checking if a page is readerable
+
+There is also a port of Mozilla's `isProbablyReaderable`: a quick-and-dirty way of figuring out if it's plausible that a page contains an article, without the cost of running the full parse. Like the original, it can produce both false positives and false negatives, but it's cheap enough to run on pages as they come in:
 
 ```php
 use fivefilters\Readability\Readerable;
 
+// Only run the full parse if we suspect it will produce a meaningful result.
 if (Readerable::isProbablyReaderable($html)) {
-    // ...
+    $article = new Readability(new Configuration())->parse($html);
 }
 ```
 
+It accepts an HTML string or a `\Dom\HTMLDocument`, and takes the same optional tuning parameters as Readability.js (same defaults):
+
+- **minContentLength**: default `140`, the minimum node content length used to decide if the document is readerable;
+- **minScore**: default `20`, the minimum cumulated 'score' used to determine if the document is readerable;
+- **visibilityChecker**: default `Readerable::isNodeVisible(...)`, the function used to determine if a node is visible.
+
+```php
+Readerable::isProbablyReaderable($html, minScore: 0, minContentLength: 120);
+```
+
 ## Options
 
 Configuration is a readonly object; pass options as named constructor arguments (or as an array via `Configuration::fromArray()`):

UPGRADE.md

@@ -181,7 +181,13 @@ Debug messages are sent to the logger independently of the `debug` flag (which o
 ## Behavior changes to be aware of
 
 - **`parse()` never returns `false`/`bool`.** It returns an `Article` or throws `ParseException` — including for empty input, documents over `maxElemsToParse`, and pages where no article could be found (all cases where Readability.js returns `null`).
-- **The content is wrapped** in `<div id="readability-page-1" class="page">…</div>`, exactly as Readability.js outputs. If you post-process the HTML, account for the wrapper.
+- **The content is wrapped** in `<div id="readability-page-1" class="page">…</div>`, exactly as Readability.js outputs. 3.x serialized the extracted elements with no wrapper around them. If you post-process the HTML, account for the wrapper — or reproduce the unwrapped 3.x output with:
+
+  ```php
+  $html = $article->contentElement->firstElementChild->innerHTML;
+  ```
+
+  (`firstElementChild` is the wrapper `div` itself — always the only child of `contentElement` — and `innerHTML` serializes everything inside it, so nothing is lost when the article consists of several top-level elements.)
 - **The byline is always extracted, and inline bylines are removed from the content by default** (see `keepInlineByline` above). A 3.x install that never set `articleByline` kept the byline in the content.
 - **Relative URL fixing follows the WHATWG URL Standard** (what browsers and Readability.js do), via PHP 8.5's native `Uri\WhatWg\Url` or rowbot/url on PHP 8.4. Edge-case outputs may differ slightly from 3.x's RFC 3986 resolution (e.g. `https://example.com` serializes as `https://example.com/`).
 - **Encoding:** input is parsed with the encoding declared in the document, defaulting to UTF-8 (as a browser would). The 3.x mb_* guessing hacks are gone — supply UTF-8 or make sure the document declares its charset.

src/Readerable.php

@@ -17,14 +17,14 @@ final class Readerable
      * Mirrors isProbablyReaderable.
      *
      * @param \Dom\HTMLDocument|string $document the document, or an HTML string
-     * @param int $minScore the minimum cumulated 'score' used to determine if the document is readerable
+     * @param float $minScore the minimum cumulated 'score' used to determine if the document is readerable
      * @param int $minContentLength the minimum node content length used to decide if the document is readerable
      * @param callable(\Dom\Element): bool|null $visibilityChecker the function used to determine if a node is visible
      * @return bool Whether or not we suspect Readability::parse() will succeed at returning an article
      */
     public static function isProbablyReaderable(
         \Dom\HTMLDocument|string $document,
-        int $minScore = 20,
+        float $minScore = 20,
         int $minContentLength = 140,
         ?callable $visibilityChecker = null,
     ): bool {

test/ReaderableTest.php

@@ -0,0 +1,100 @@
+<?php
+
+declare(strict_types=1);
+
+namespace fivefilters\Readability\Test;
+
+use fivefilters\Readability\Readerable;
+use PHPUnit\Framework\TestCase;
+
+/**
+ * Unit tests for Readerable::isProbablyReaderable, mirroring Mozilla's
+ * test/test-isProbablyReaderable.js. The corpus-wide check (every test page
+ * against the "readerable" key in its expected-metadata.json) lives in
+ * ReadabilityTest::testReaderable.
+ */
+class ReaderableTest extends TestCase
+{
+    private static function makeDoc(string $source): \Dom\HTMLDocument
+    {
+        return \Dom\HTMLDocument::createFromString($source, LIBXML_NOERROR);
+    }
+
+    /** "hello there " repeated: 1× = 11 chars, 11× = 132, 12× = 144, 50× = 600. */
+    private static function makeParagraphDoc(int $repeat): \Dom\HTMLDocument
+    {
+        return self::makeDoc('<html><p id="main">' . str_repeat('hello there ', $repeat) . '</p></html>');
+    }
+
+    public function testOnlyLargeDocumentsAreReaderableWithDefaultOptions(): void
+    {
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(1)), 'very small doc'); // score: 0
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(11)), 'small doc'); // score: 0
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(12)), 'large doc'); // score: ~1.7
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(50)), 'very large doc'); // score: ~21.4
+    }
+
+    public function testSmallAndLargeDocumentsAreReaderableWithLowerMinContentLength(): void
+    {
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(1), minScore: 0, minContentLength: 120), 'very small doc');
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(11), minScore: 0, minContentLength: 120), 'small doc');
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(12), minScore: 0, minContentLength: 120), 'large doc');
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(50), minScore: 0, minContentLength: 120), 'very large doc');
+    }
+
+    public function testOnlyLargestDocumentIsReaderableWithHigherMinContentLength(): void
+    {
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(1), minScore: 0, minContentLength: 200), 'very small doc');
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(11), minScore: 0, minContentLength: 200), 'small doc');
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(12), minScore: 0, minContentLength: 200), 'large doc');
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(50), minScore: 0, minContentLength: 200), 'very large doc');
+    }
+
+    public function testSmallAndLargeDocumentsAreReaderableWithLowerMinScore(): void
+    {
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(1), minScore: 4, minContentLength: 0), 'very small doc'); // score: ~3.3
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(11), minScore: 4, minContentLength: 0), 'small doc'); // score: ~11.4
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(12), minScore: 4, minContentLength: 0), 'large doc'); // score: ~11.9
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(50), minScore: 4, minContentLength: 0), 'very large doc'); // score: ~24.4
+    }
+
+    public function testOnlyLargeDocumentsAreReaderableWithHigherMinScore(): void
+    {
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(1), minScore: 11.5, minContentLength: 0), 'very small doc'); // score: ~3.3
+        $this->assertFalse(Readerable::isProbablyReaderable(self::makeParagraphDoc(11), minScore: 11.5, minContentLength: 0), 'small doc'); // score: ~11.4
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(12), minScore: 11.5, minContentLength: 0), 'large doc'); // score: ~11.9
+        $this->assertTrue(Readerable::isProbablyReaderable(self::makeParagraphDoc(50), minScore: 11.5, minContentLength: 0), 'very large doc'); // score: ~24.4
+    }
+
+    public function testUsesProvidedVisibilityCheckerNotReaderable(): void
+    {
+        $called = false;
+        $result = Readerable::isProbablyReaderable(self::makeParagraphDoc(50), visibilityChecker: function () use (&$called): bool {
+            $called = true;
+            return false;
+        });
+        $this->assertFalse($result);
+        $this->assertTrue($called);
+    }
+
+    public function testUsesProvidedVisibilityCheckerReaderable(): void
+    {
+        $hiddenDoc = self::makeDoc('<html><p id="main" style="display: none">' . str_repeat('hello there ', 50) . '</p></html>');
+        $this->assertFalse(Readerable::isProbablyReaderable($hiddenDoc), 'hidden with default checker');
+
+        $called = false;
+        $result = Readerable::isProbablyReaderable($hiddenDoc, visibilityChecker: function () use (&$called): bool {
+            $called = true;
+            return true;
+        });
+        $this->assertTrue($result, 'hidden with always-visible checker');
+        $this->assertTrue($called);
+    }
+
+    /** PHP-specific convenience: an HTML string is accepted in place of a document. */
+    public function testAcceptsHtmlString(): void
+    {
+        $this->assertTrue(Readerable::isProbablyReaderable('<html><p>' . str_repeat('hello there ', 50) . '</p></html>'));
+        $this->assertFalse(Readerable::isProbablyReaderable('<html><p>hello there</p></html>'));
+    }
+}