v4.0: Rewrite on PHP 8.4's native DOM API, parity with Readability.js 0.6.0

.github/workflows/main.yml

@@ -1,46 +1,35 @@
-# This is a basic workflow to help you get started with Actions
-
 name: CI
 
-# Controls when the workflow will run
 on:
-  # Triggers the workflow on push or pull request events but only for the master branch
   push:
     branches: [master]
   pull_request:
     branches: [master]
-
-  # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
 
-# A workflow run is made up of one or more jobs that can run sequentially or in parallel
 jobs:
-  # This workflow contains a single job called "build"
-  build:
-    # The type of runner that the job will run on
+  test:
     runs-on: ubuntu-latest
 
     strategy:
       matrix:
-        php: ['8.1', '8.2', '8.3', '8.4']
-        libxml: ['2.9.14']
+        php: ['8.4', '8.5']
 
-    # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
-      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v7
 
       - name: Set up PHP
         uses: shivammathur/setup-php@v2
         with:
           php-version: ${{matrix.php}}
+          extensions: dom, mbstring
           tools: composer:v2
 
       - name: Install dependencies
         run: composer install
 
-      # Runs a set of commands using the runners shell
       - name: Run tests
-        run: |
-          docker build --build-arg PHP_VERSION=${{matrix.php}} --build-arg LIBXML_VERSION=${{matrix.libxml}} -t gh-action - < ./docker/php/Dockerfile
-          docker run --volume $PWD:/app --workdir="/app" --env XDEBUG_MODE=coverage gh-action php ./vendor/bin/phpunit --coverage-clover /app/test/clover.xml
+        run: ./vendor/bin/phpunit
+
+      - name: Static analysis
+        run: ./vendor/bin/psalm --output-format=github --no-progress

.gitignore

@@ -2,4 +2,8 @@
 vendor
 composer.lock
 /test.*
-/test/changed/
+/test/changed/
+/test/tools/node_modules/
+/test/tools/js-output/
+/test/tools/package-lock.json
+/.phpunit.result.cache

CHANGELOG.md

@@ -1,6 +1,34 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
+## [v4.0.0-beta.1](https://github.com/fivefilters/readability.php/releases/tag/v4.0.0-beta.1)
+
+Ground-up port from the latest Readability.js (v0.6.0) using Claude's Fable model. Uses PHP 8.4's new DOM API and native parser. See [UPGRADE.md](UPGRADE.md) for the full 3.x → 4.0 migration guide.
+
+### Changed
+- Requires PHP >= 8.4; parsing and serialization use `Dom\HTMLDocument` (the WHATWG-spec Lexbor parser bundled with PHP), replacing HTML5-PHP and the legacy libxml path
+- `parse()` now returns a readonly `Article` value object (`title`, `content`, `textContent`, `length`, `excerpt`, `byline`, `siteName`, `dir`, `lang`, `publishedTime`, `image`, `images`, `contentElement`). When no article content is found (where Readability.js returns null), the `Article` still carries the extracted title and metadata with null content — check `Article::hasContent()`. `ParseException` is thrown only for empty input or documents over `maxElemsToParse`
+- Options are passed directly to the `Readability` constructor as named arguments, like the options object in Readability.js (`new Readability(fixRelativeURLs: true)`), and are all optional; a readonly `Configuration` object taking the same named arguments can be passed instead. `maxTopCandidates` renamed to `nbTopCandidates` (matching Readability.js)
+- Article output is wrapped in `<div id="readability-page-1" class="page">`, as in Readability.js
+- Byline is always extracted into `Article::$byline`; the `articleByline` option becomes `keepInlineByline`, which only controls whether an inline byline stays in the content (default removes it, as in Readability.js)
+- Image extraction moved onto the result object: `getImage()`/`getImages()` become `$article->image` / `$article->images`
+- PSR-3 logging: pass a `LoggerInterface` as the `logger` option instead of `setLogger()`
+- Relative URL resolution now uses a real WHATWG URL parser — PHP 8.5's native `Uri\WhatWg\Url` when available, [rowbot/url](https://github.com/TRowbotham/URL-Parser) on PHP 8.4 — matching the `new URL()` behavior Readability.js relies on; replaces league/uri
+- Test corpus replaced with Mozilla's 130 test pages verbatim; content comparison ports Mozilla's structural DOM comparison
+
+### Added
+- Parity with Readability.js 0.6.0: `lang` and `publishedTime` output; `maxElemsToParse`, `classesToPreserve`, `allowedVideoRegex`, `linkDensityModifier` and `debug` options; aria-modal dialog removal; ad/loading-indicator stripping; parsely/`article:author`/`itemprop` metadata sources; JSON-LD `@graph`, `@context`-object and array handling; Unicode comma scoring; updated regexes (mathjax, bilibili, en/em-dash title separators)
+- `Readerable::isProbablyReaderable()`, a port of Readability-readerable.js
+- `parse()` accepts an already-parsed `Dom\HTMLDocument` as well as an HTML string
+- Cross-check harness (`test/tools/`) that diffs this port's output against Readability.js over the whole corpus
+- Static analysis with [Psalm](https://psalm.dev/) (`composer analyse`), run in CI alongside the test suite
+
+### Removed
+- HTML5-PHP dependency; `ext-xml` requirement
+- Options that existed as libxml workarounds: `parser`, `substituteEntities`, `normalizeEntities`, `summonCthulhu`
+- The custom DOM subclass layer (`src/Nodes/`) and its workarounds (attribute-based state, shifting-aware iteration)
+- The Docker-based local test setup (`docker-compose.yml`, `Makefile`, `docker/`); tests and static analysis run directly on PHP 8.4/8.5, locally and in CI
+
 ## [v3.3.3](https://github.com/fivefilters/readability.php/releases/tag/v3.3.3)
 - Fix type error - extends type support to add DOMProcessingInstruction in more method signatures (reported by @reinierkors)
 

CONTRIBUTING.md

@@ -21,11 +21,25 @@ For anything else, we accept contributions via Pull Requests on [Github](https:/
 - **Don't forget to add yourself to AUTHORS.md** - If you want to be credited, make sure you add your information (whatever you want to include) in `AUTHORS.md`.
 
 
+## Syncing with a new Readability.js release
+
+The private methods in `src/Readability.php` mirror the prototype methods of Readability.js in name (minus the underscore prefix) and order, so syncing is a mechanical diff exercise:
+
+1. Diff the new `Readability.js` against the previous synced version (git master at the time of the last sync — see `test/tools/known-divergences.md`).
+2. For each changed `_method`, apply the same change to the matching method in `src/Readability.php`. Regex changes go to `src/RegExps.php` (constants are the UPPER_SNAKE forms of the JS `REGEXPS` keys). `Readability-readerable.js` changes go to `src/Readerable.php`.
+3. Copy any added/changed directories from Mozilla's `test/test-pages/` into `test/test-pages/` verbatim (drop each page's `expected-images.json`-era leftovers if any; sources, `expected.html` and `expected-metadata.json` are used as-is).
+4. Run `./vendor/bin/phpunit`, then the cross-check harness in `test/tools/` (bump the `@mozilla/readability` version in its `package.json`), and update `known-divergences.md`.
+
+Things that intentionally differ from the JS (don't "fix" these): scoring state lives in `SplObjectStorage` maps instead of node expandos; `getAllNodesWithTag` materializes querySelectorAll snapshots; URL resolution goes through `src/Url.php`, which wraps a real WHATWG URL parser (PHP 8.5's native `Uri\WhatWg\Url`, or rowbot/url on PHP 8.4) and returns `null` where JS `new URL()` throws; JS `''`/`undefined` metadata maps to PHP `null`; `parse()` throws instead of returning null.
+
 ## Running Tests
 
 ``` bash
-$ make test-all #requires docker and docker-compose
+$ ./vendor/bin/phpunit      # requires PHP 8.4+
+$ ./vendor/bin/psalm        # static analysis; CI runs this too
 ```
 
+CI runs both on PHP 8.4 and 8.5.
+
 
 **Happy coding**!