feat(game): stream validation for game metadata

README.md

@@ -1,50 +1,47 @@
 # TechAPI
 
-> **Curated, open dataset for consumer electronics specs.** Free, share-alike, machine-readable.
+> Curated, open dataset for consumer electronics specifications and game metadata.
 
 [![validate-data](https://github.com/GetTechAPI/TechAPI/actions/workflows/validate-data.yml/badge.svg)](https://github.com/GetTechAPI/TechAPI/actions/workflows/validate-data.yml)
- Data: **CC-BY-SA 4.0**
+Data: **CC-BY-SA 4.0**
 
-This repo holds the curated **dataset** and the **public site** (Astro intro +
-playground). The API server, ingestion crawlers, coverage checks, and the
-static-dump generator live in
-[**TechEngine**](https://github.com/GetTechAPI/TechEngine).
+This repository owns the versioned dataset and the public Astro site. The API
+server, ingestion crawlers, coverage checks, and static-dump generator live in
+[TechEngine](https://github.com/GetTechAPI/TechEngine), included here as the
+`TechEngine` submodule.
 
 ## Layout
 
 ```text
-data/brand/<country>/<slug>.json                       # e.g. data/brand/kr/samsung.json
-data/soc/<manufacturer>/<year>/<slug>.json             # data/soc/qualcomm/2024/snapdragon-8-elite.json
-data/smartphone/<brand>/<year>/<slug>.json             # data/smartphone/samsung/2025/galaxy-s25.json
-data/gpu/<manufacturer>/<year>/<segment>/<slug>.json   # data/gpu/nvidia/2025/consumer/geforce-rtx-5090.json
-data/cpu/<manufacturer>/<year>/<segment>/<slug>.json   # data/cpu/intel/2023/consumer/core-i9-14900k.json
+data/brand/<country>/<slug>.json
+data/cpu/<manufacturer>/<year>/<segment>/<slug>.json
+data/gpu/<manufacturer>/<year>/<segment>/<slug>.json
+data/smartphone/<brand>/<year>/<slug>.json
+data/game/<slug[:2]>/<slug>.json
+site/                       # Astro Pages site
+site/public/v1/             # committed static API dump from TechEngine
+app/validate.py             # stdlib-only, fast repository self-check
 ```
 
-All paths use singular folder names. Slugs are kebab-case and unique within each category.
+The game directory is sharded by normalized slug rather than genre or import
+batch. See [the data layout guide](docs/DATA_LAYOUT.md) before adding or moving
+game records.
 
-The Astro site lives under `site/` and is the deploy target for GitHub Pages. It
-consumes the static JSON dump produced by TechEngine's `weekly-refresh` workflow.
-
-## Self-Check
-
-A lightweight bundled validator lives at `app/validate.py`. It runs on every PR via
-[`validate-data.yml`](.github/workflows/validate-data.yml) and is also chained into
-the heavier TechEngine validation workflow as a downstream job.
+## Local workflow
 
 ```bash
 python -m app.validate
+cd site && npm ci && npm run build
 ```
 
-The validator uses only the Python standard library; no install step required.
-
-## Contributing
+Use TechEngine for ingest, API, and dump generation. Do not change its code or
+submodule pointer as part of a data-only change. The committed
+`site/public/v1/` dump is deployed by GitHub Pages; its contents must remain in
+the repository even when it is too large to browse comfortably locally.
 
-Open a PR with the new/updated JSON file. The PR template walks through what to
-include. The validator must pass. All records (`brand`, `soc`, `smartphone`,
-`gpu`, and `cpu`) must include `source_urls` with at least one canonical
-reference (vendor product page, Wikipedia infobox, datasheet).
+For a focused workflow and the boundary between this repository and TechEngine,
+see [docs/LOCAL_WORKFLOW.md](docs/LOCAL_WORKFLOW.md).
 
 ## License
 
-Data is licensed **CC-BY-SA 4.0**; attribute "Data from TechAPI" and share alike.
-The bundled validator code is [MIT](LICENSE).
+Data is licensed **CC-BY-SA 4.0**. The bundled validator is [MIT](LICENSE).

app/validate.py

@@ -7,7 +7,9 @@
 
 from __future__ import annotations
 
+import argparse
 import json
+import os
 import re
 import sys
 from pathlib import Path
@@ -112,6 +114,29 @@
     "verified",
 }
 
+GAME_LIST_FIELDS = (
+    "platforms",
+    "genres",
+    "stores",
+    "developers",
+    "publishers",
+    "tags",
+)
+
+GAME_ESRB_RATINGS = {
+    "Everyone",
+    "Everyone 10+",
+    "Teen",
+    "Mature",
+    "Adults Only",
+    "Rating Pending",
+}
+
+
+def _game_shard(slug: str) -> str:
+    """Return the stable on-disk shard for a game slug."""
+    return slug[:2]
+
 SOFTWARE_REQUIRED = {
     "slug",
     "name",
@@ -159,6 +184,37 @@ def _check_date(name: str, value: object, errors: list[str]) -> None:
         errors.append(f"{name}: release_date '{value}' must be ISO 8601 YYYY-MM-DD (§14.2)")
 
 
+def _check_title(name: str, value: object, errors: list[str]) -> None:
+    if not isinstance(value, str) or not value.strip():
+        errors.append(f"{name}: name must be a non-empty string")
+
+
+def _check_bool(name: str, field: str, value: object, errors: list[str]) -> None:
+    if not isinstance(value, bool):
+        errors.append(f"{name}: {field} must be a boolean")
+
+
+def _check_string_list(name: str, field: str, value: object, errors: list[str]) -> None:
+    """Validate optional game classification lists without constraining vocabulary."""
+    if value is None:
+        return
+    if not isinstance(value, list) or not all(
+        isinstance(item, str) and item.strip() for item in value
+    ):
+        errors.append(f"{name}: {field} must be a list of non-empty strings")
+        return
+    normalized = [item.casefold() for item in value]
+    if len(set(normalized)) != len(normalized):
+        errors.append(f"{name}: {field} contains duplicate values")
+
+
+def _check_optional_url(name: str, field: str, value: object, errors: list[str]) -> None:
+    if value is not None and (
+        not isinstance(value, str) or not value.startswith(("http://", "https://"))
+    ):
+        errors.append(f"{name}: {field} must be an http(s) URL when present")
+
+
 def _check_unique_slugs(
     category: str, records: list[tuple[str, dict[str, Any]]], errors: list[str]
 ) -> None:
@@ -230,6 +286,83 @@ def _check_variant_path(
         errors.append(f"{fname}: filename must match slug '{rec.get('slug')}'")
 
 
+def _check_game_path(fname: str, rec: dict[str, Any], errors: list[str]) -> None:
+    """Keep imports in the documented ``game/<shard>/<slug>.json`` layout."""
+    parts = Path(fname).parts
+    slug = rec.get("slug")
+    if not isinstance(slug, str):
+        return
+    if len(parts) != 3:
+        errors.append(f"{fname}: game records must live at 'game/<shard>/<slug>.json'")
+        return
+    _, shard, filename = parts
+    expected_shard = _game_shard(slug)
+    if shard != expected_shard:
+        errors.append(
+            f"{fname}: lives in shard '{shard}' but slug '{slug}' belongs in '{expected_shard}'"
+        )
+    if filename != f"{slug}.json":
+        errors.append(f"{fname}: filename must match slug '{slug}'")
+
+
+def _validate_game_record(fname: str, rec: dict[str, Any], errors: list[str]) -> None:
+    _check_required(fname, rec, GAME_REQUIRED, errors)
+    _check_source_urls(fname, rec, errors)
+    _check_slug(fname, rec.get("slug"), errors)
+    _check_title(fname, rec.get("name"), errors)
+    _check_bool(fname, "verified", rec.get("verified"), errors)
+    if rec.get("release_date") is not None:
+        _check_date(fname, rec["release_date"], errors)
+    if rec.get("rating") is not None:
+        _check_range(fname, "rating", rec.get("rating"), 0, 5, errors)
+    if rec.get("rating_count") is not None:
+        _check_range(fname, "rating_count", rec.get("rating_count"), 0, 2_000_000_000, errors)
+    if rec.get("metacritic") is not None:
+        _check_range(fname, "metacritic", rec.get("metacritic"), 0, 100, errors)
+    if rec.get("playtime_hours") is not None:
+        _check_range(fname, "playtime_hours", rec.get("playtime_hours"), 0, 100_000, errors)
+    for field in GAME_LIST_FIELDS:
+        _check_string_list(fname, field, rec.get(field), errors)
+    esrb_rating = rec.get("esrb_rating")
+    if esrb_rating is not None and esrb_rating not in GAME_ESRB_RATINGS:
+        errors.append(f"{fname}: esrb_rating '{esrb_rating}' not in {sorted(GAME_ESRB_RATINGS)}")
+    _check_optional_url(fname, "background_image", rec.get("background_image"), errors)
+    _check_game_path(fname, rec, errors)
+
+
+def validate_games(data_dir: Path | None = None) -> list[str]:
+    """Stream game records so the import-scale dataset is not retained in memory."""
+    errors: list[str] = []
+    root = (data_dir or DATA_DIR) / "game"
+    if not root.exists():
+        return errors
+
+    seen: dict[str, str] = {}
+    for directory, directories, filenames in os.walk(root):
+        directories.sort()
+        for filename in sorted(filenames):
+            if not filename.endswith(".json"):
+                continue
+            path = Path(directory) / filename
+            fname = str(path.relative_to(data_dir or DATA_DIR))
+            try:
+                rec = json.loads(path.read_text(encoding="utf-8-sig"))
+            except (OSError, json.JSONDecodeError) as exc:
+                errors.append(f"{fname}: invalid JSON ({exc})")
+                continue
+            if not isinstance(rec, dict):
+                errors.append(f"{fname}: game record must be a JSON object")
+                continue
+            slug = rec.get("slug")
+            if isinstance(slug, str):
+                if slug in seen:
+                    errors.append(f"{fname}: duplicate game slug '{slug}' (also in {seen[slug]})")
+                else:
+                    seen[slug] = fname
+            _validate_game_record(fname, rec, errors)
+    return errors
+
+
 def validate() -> list[str]:
     errors: list[str] = []
 
@@ -416,15 +549,7 @@ def validate() -> list[str]:
         _check_variant_path(fname, rec, "monitor", errors, allow_flat=True)
 
     for fname, rec in games:
-        _check_required(fname, rec, GAME_REQUIRED, errors)
-        _check_source_urls(fname, rec, errors)
-        _check_slug(fname, rec.get("slug"), errors)
-        if rec.get("release_date") is not None:
-            _check_date(fname, rec["release_date"], errors)
-        if rec.get("rating") is not None:
-            _check_range(fname, "rating", rec.get("rating"), 0, 5, errors)
-        if rec.get("metacritic") is not None:
-            _check_range(fname, "metacritic", rec.get("metacritic"), 0, 100, errors)
+        _validate_game_record(fname, rec, errors)
 
     for fname, rec in software:
         _check_required(fname, rec, SOFTWARE_REQUIRED, errors)
@@ -436,12 +561,20 @@ def validate() -> list[str]:
     return errors
 
 
-def run() -> int:
+def run(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description="Validate TechAPI record data")
+    parser.add_argument(
+        "--category",
+        choices=("all", "game"),
+        default="all",
+        help="validate all records (default) or stream the game dataset only",
+    )
+    args = parser.parse_args(argv)
     try:
         sys.stdout.reconfigure(encoding="utf-8")  # type: ignore[union-attr]
     except Exception:
         pass
-    errors = validate()
+    errors = validate_games() if args.category == "game" else validate()
     if errors:
         print(f"❌ Data validation failed ({len(errors)} issue(s)):")
         for err in errors:

docs/DATA_LAYOUT.md

@@ -0,0 +1,44 @@
+# Data layout
+
+Versioned records stay in `data/`; the directory name is the record family and
+the JSON file is the individual record.  Do not group records by a transient
+import run or by a scraper name.
+
+## Games
+
+Game records use a deterministic shard:
+
+```
+data/game/<slug first two characters>/<slug>.json
+```
+
+For example, `hades` belongs at `data/game/ha/hades.json`.  This keeps a
+large import stable and prevents one directory from becoming a bottleneck.
+
+## Local checkout policy
+
+The default checkout deliberately excludes `data/` and generated `site/`
+content.  They remain tracked and are still present on the remote; omitting
+them only prevents routine Git polling from scanning millions of files.
+
+For a focused data import, temporarily add only the family you need:
+
+```
+git sparse-checkout add --cone data/cpu
+```
+
+For a full game import, add `data/game` for the duration of the import, then
+return to the small default checkout:
+
+```
+git sparse-checkout add --cone data/game
+git sparse-checkout set --cone .github app docs TechEngine
+```
+
+Never use `assume-unchanged` or `skip-worktree` as an import workflow.  Run
+`python -m app.validate` only while the needed data family is checked out.
+For the million-record game import, use the streaming validator:
+
+```
+python -m app.validate --category game
+```

docs/LOCAL_WORKFLOW.md

@@ -0,0 +1,33 @@
+# Local workflow
+
+## Ownership boundary
+
+TechAPI owns versioned record files, the small standard-library validator, and
+the Astro Pages site. TechEngine owns ingestion, the API service, schemas,
+database models, dump generation, and heavy validation. Keep a schema change
+coordinated across both repositories; do not modify the `TechEngine` submodule
+pointer for an ordinary data cleanup.
+
+## Game records
+
+Games are individual JSON files under `data/game/<shard>/`. The shard is the
+first two characters of `slug` (or the whole slug when it has one character).
+The layout is intentionally independent of genre, platform, developer, and
+import source so that imports stay stable and directories stay bounded.
+
+Before an import:
+
+1. Put each record in its slug shard.
+2. Keep `slug`, `name`, `source_urls`, and `verified` present.
+3. Run `python -m app.validate`.
+4. Generate or verify the static dump through TechEngine when the API output
+   needs to change.
+
+## Generated files
+
+- `site/public/v1/` is committed API output used by GitHub Pages; do not delete
+  it as a local cleanup shortcut.
+- `site/public/v1/history.json` and `verification.json` are build outputs and
+  are ignored.
+- `.claude/archive/` contains local historical work notes and is ignored.
+- Python caches (`.pytest_cache`, `.mypy_cache`, `.ruff_cache`) are disposable.