Skip to content

data(software,game): add software category + 18k Wikidata records (→296k)#116

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging
Jul 10, 2026
Merged

data(software,game): add software category + 18k Wikidata records (→296k)#116
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

Adds the new software category (TechEngine PR #42) + more Wikidata games — dataset 277,458 → 295,869 on the road to 1M.

  • software11,611 software products from Wikidata (P31 = Q7397): name, release_date, developers, publishers, operating_systems, programming_languages, licenses, genres. Standalone/unscored, no brand FK. source_urls cite each Wikidata entity.
  • game6,800 additional video games recovered from Q7889 subclasses (browser/arcade/mobile/etc.) that the initial direct-P31 pass (data(game): add game category + import 168,883 Wikidata games (→277k records) #115) missed. Deduped against the existing 168,883 by QID + slug (collisions get a QID suffix).
  • Bumps engine submodule to the software-category pin; mirrors SOFTWARE_REQUIRED into app/validate.py.
  • CI note: TechEngine PR data(mobile): import source variant supplements #42 also fixed a CI outage — GitHub switched runners to Node 24, which crashes actions/setup-python + cache: pip ("Maximum call stack size exceeded"); fixed by dropping the pip cache in test.yml.

All verified: false. python -m app.validate ✅ passed (295,869 records). Dump regenerated — v1/software/ (11,611) + 6,800 new v1/games/ pages + openapi + index (existing 168,883 game pages left untouched — surgical staging). No fabrication; every field from Wikidata.

Closes #1

New `software` category (TechEngine PR #42): 11,611 software products from
Wikidata (P31=Q7397) — name, release_date, developers, publishers,
operating_systems, programming_languages, licenses, genres. Plus 6,800 more
video games recovered from Q7889 subclasses (browser/arcade/mobile games) the
direct-P31 pass missed. Bumps engine submodule to the software-category pin +
mirrors SOFTWARE_REQUIRED into app/validate.py. All from Wikidata, no
fabrication; verified:false; sharded.

Refs #1
@github-actions github-actions Bot added app Validator or application code changes data Dataset changes enhancement New feature or request labels Jul 10, 2026
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · data validation

category    added     total records
software    11,611       295,869
game (+)     6,800
  • python -m app.validate✅ passed (0 errors) across all 12 categories.
  • New software category seeded from Wikidata Q7397; engine submodule bumped (TechEngine PR data(mobile): import source variant supplements #42).
  • Dump: 11,611 software pages + 6,800 new game pages + openapi/index. Existing 168,883 game pages untouched.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · verify (Tier 0)

Real data + a CI fix:

  • Every software/game record is a Wikidata entity; source_urls links to wikidata.org/wiki/<QID>. Fields are the entity's own statements (developer P178, publisher P123, OS P306, language P277, license P275, genre P136, date P577) — nothing invented.
  • Extra games were de-duplicated against the existing catalogue by QID and slug before writing.
  • Bundled CI fix: GitHub's runner Node 20→24 switch broke actions/setup-python pip-cache (stack overflow); removed cache: pip in the engine test.yml.

verified stays false. Safe to rebase-merge (submodule bump matches the auto-bump already on main).

@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jul 10, 2026
@Seungpyo1007 Seungpyo1007 merged commit e4342c3 into main Jul 10, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app Validator or application code changes data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants