Skip to content

data(game): add game category + import 168,883 Wikidata games (→277k records)#115

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging
Jul 9, 2026
Merged

data(game): add game category + import 168,883 Wikidata games (→277k records)#115
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

Adds the new game category and imports 168,883 video games from Wikidata — the first big step of the 1M-record goal (dataset 108,575 → 277,458).

  • Source: Wikidata SPARQL, ?g wdt:P31 wd:Q7889 (174,919 games). Per game: name, release_date (P577, earliest), platforms (P400), genres (P136), developers (P178), publishers (P123). source_urls cites each game's Wikidata entity. No fabrication — every field is straight from Wikidata; games with no English label or unparseable data were skipped.
  • Category: wired via TechEngine PR ci(pr): cap metadata file scans for large imports #41 (merged) — standalone, unscored, no brand FK (makers are free-text developer/publisher lists). This PR bumps the engine submodule to that pin and mirrors the GAME_REQUIRED rule into app/validate.py.
  • Layout: sharded data/game/<xx>/<slug>.json; slugs kebab-cased, name-collisions disambiguated by Wikidata QID.
  • All records verified: false. python -m app.validate ✅ passed across all 277,458 records. Public dump regenerated — site/public/v1/games/ (168,883 detail pages + list) + openapi.json + v1/index.json.

Next toward 1M: RAWG (474k) once downloaded, then IGDB.

Closes #1

Adds the new `game` category (TechEngine PR #41): 168,883 video games sourced
from Wikidata (P31=Q7889) via SPARQL — name, release_date, platforms, genres,
developers, publishers; source_urls cite each Wikidata entity. All verified:false.
Bumps the engine submodule to the game-category pin and mirrors the GAME_REQUIRED
rule into app/validate.py. Sharded under data/game/<xx>/. No fabrication —
every field comes from Wikidata.

Refs #1
@Seungpyo1007 Seungpyo1007 added data Dataset changes enhancement New feature or request labels Jul 9, 2026
@github-actions github-actions Bot added the app Validator or application code changes label Jul 9, 2026
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · data validation

category    added     verified   total records
game        168,883    false        277,458
  • python -m app.validate✅ passed (0 errors) across all 10+1 categories.
  • New game category seeded from Wikidata; engine submodule bumped to the game-category pin (TechEngine PR ci(pr): cap metadata file scans for large imports #41).
  • Dump regenerated: 168,883 game detail pages + list + openapi. Other collections untouched (surgical staging).
  • Slug fix applied: 10 truncation-trailing-hyphen slugs corrected before commit.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · verify (Tier 0)

Real data, not fabricated:

  • Every game is a Wikidata entity (P31 = Q7889); its source_urls links straight back to wikidata.org/wiki/<QID> for provenance.
  • Fields are the entity's own statements: publication date (P577), platforms (P400), genres (P136), developer (P178), publisher (P123) — aggregated one row per game, English labels only.
  • Games lacking an English label or with unparseable dates were dropped rather than guessed. Duplicate QIDs from the resumed fetch were de-duplicated.

verified stays false; promotion remains the verify-network job's call. Safe to rebase-merge (submodule bump matches the auto-bump already on main).

@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jul 9, 2026
@Seungpyo1007 Seungpyo1007 merged commit 44a7f59 into main Jul 9, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app Validator or application code changes data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants