Rewrite the fast run mode with scoped SPARQL queries

[LeMyst]

Rewrite of the fast run mode: the mechanism that lets a synchronisation bot skip the entities that are already up to date, without loading each of them through the MediaWiki API.

This PR replaces the previous implementation rather than patching it. #1001 was the alternative proposal, patching the existing architecture in place; it has been closed in favour of this one, and the one thing it still had over this branch (datatype coverage) has been ported here.

Why a rewrite

The previous implementation loaded the whole data corpus with a single paginated SPARQL query per property, pulling statements, qualifiers, references and ranks in one go. That query is the scaling wall: on a benchmark of only 95 entities it repeatedly hit consecutive WDQS 504 timeouts, and at the scale of a large corpus it simply cannot run. Once it succeeds the comparison is very fast, but a cold load that cannot complete is not a usable trade-off.

This implementation scopes its queries instead: the statements of a property are loaded for the corpus, and the qualifiers, references and ranks are loaded lazily, per statement, only when a deep comparison is actually needed. Each query stays small enough to complete.

What it does

entity.write_required(base_filter=...) returns False only when at least one entity already holds every compared claim with the same value, and the same qualifiers, references and rank depending on the options. When the entity ID is known (the entity was loaded with wbi.item.get()), the comparison is automatically restricted to that entity, so a bot editing a known entity no longer needs a unique identifying value in its data.

The base filter defines the corpus and accepts three forms: a property with a value, a property with any value, and a property path given as a list of two datatypes.

The README section Examples (in "fast run" mode) documents the whole API, with a full bot example, the options, the language-data checking and the measured performance figures.

Breaking changes

The fastrun API is not backwards compatible. Anything using wbi_fastrun directly needs updating:

• FastRunContainer.get_items() becomes get_entities().
• FastRunContainer.__init__() now takes base_filter as its first, mandatory argument; base_data_type becomes optional. use_refs becomes use_references, joined by use_qualifiers (default True), use_rank and cache.
• FastRunContainer.write_required() gains entity_filter and property_filter and drops cqid.
• BaseEntity.write_required() now requires base_filter instead of defaulting to None.
• Data types gained a PTYPE class attribute (their URI in the Wikibase ontology) and a from_sparql_value() method taking the SPARQL binding dict. It replaces parse_sparql_value(), which fastrun no longer calls.

This needs a changelog entry and a version note.

Datatype coverage

BaseDataType.from_sparql_value() was a stub returning None, so any datatype not implementing it was skipped with a warning and its statements were always reported as requiring a write. The last commit gives it a working generic implementation for the data types whose RDF representation is the literal value itself, plus URI-specific implementations that extract the value a local claim actually holds:

• Property, Lexeme, Form, Sense: the entity ID
• GeoShape, TabularData: the Commons page title, percent-decoded
• CommonsMedia: the file name, percent-decoded

CommonsMedia was the worst case, and a silent one: it inherited the URL implementation, stored the whole Commons URL and could therefore never match a local claim, without even emitting the warning the other data types produce.

Every reachable datatype is now compared. Only EntitySchema is left out, since it has no PTYPE and is never resolved from the SPARQL property type.

Known limitations

Documented in the README:

• The SPARQL endpoint lags behind the live data, so a write that just happened may not be visible yet and could be reported as required again.
• Attributes missing from the SPARQL simple values are rebuilt with their default value (the precision of a time value is inferred from the timestamp, quantity bounds are not compared). Statements using non-default attributes are always reported as requiring a write. In particular, a timestamp whose precision cannot be inferred is skipped with a warning.
• Qualifiers, references and ranks cost one SPARQL query per statement to check. This is fast when most entities are up to date, but slows down when many entities share the same values.

Tests

253 tests pass, all offline: the 48 fastrun tests run against the MockWikibase emulation with a per-query-type SPARQL dispatcher, so no network access is involved. mypy, isort, codespell and flynt are clean.

Rebased on master.