Skip to content

data(software): +30,882 Wikidata subclass software (software → 42.5k, dataset → 327k)#117

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging
Jul 10, 2026
Merged

data(software): +30,882 Wikidata subclass software (software → 42.5k, dataset → 327k)#117
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

Expands the software category 11,611 → 42,493 with 30,882 subclass-typed software products from Wikidata that the initial direct-P31=Q7397 pass (#116) missed. Dataset 295,869 → 326,751 toward 1M.

  • Source classes (P31/P279*): application software (Q166142), free software (Q341), software libraries (Q188860), web applications (Q193424), programming languages (Q9143), operating systems (Q9135), mobile apps (Q620615). Union deduped against the existing 11,611 by QID + slug (only 19 slug collisions → QID-suffixed).
  • Per record: name, release_date, developers, publishers, operating_systems, programming_languages, licenses, genres — all Wikidata statements; source_urls cite each entity. No fabrication.
  • Guardrail note: the Q7397 (software) P279* subclass walk is NOT used — Wikidata's software subclass tree is polluted and resolves to ~10M unrelated entities. Only the narrow, clean subclasses above were mined.

All verified: false. python -m app.validate ✅ (326,751 records). Dump: 30,882 new v1/software/ pages + list + index (existing 11,611 software + all other categories untouched — surgical staging).

Closes #1

Expands the software category (11,611 -> 42,493) with subclass-typed software
Wikidata's direct-P31=Q7397 pass missed: application software (Q166142), free
software (Q341), libraries (Q188860), web apps (Q193424), programming languages
(Q9143), operating systems (Q9135), mobile apps (Q620615). Deduped against
existing by QID + slug. All from Wikidata (source_urls cite entities); no
fabrication; verified:false; sharded. (Q7397 P279* subclass walk NOT used — its
tree is polluted, resolving to ~10M unrelated entities.)

Refs #1
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · data validation

category    added     total records
software   30,882       326,751
  • python -m app.validate✅ passed (0 errors), all categories.
  • software 11,611 → 42,493 (subclass-typed products; direct-P31 pass missed them). Deduped by QID+slug (19 collisions → QID-suffixed).
  • Dump: 30,882 new v1/software/ pages + index. No other collections touched.
  • Q7397 P279* subclass walk avoided (polluted → ~10M unrelated entities); only 7 narrow clean classes mined.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · verify (Tier 0)

Real Wikidata data:

  • Each record is a Wikidata entity from a narrow software subclass (application/free/library/web-app/language/OS/mobile-app); source_urlswikidata.org/wiki/<QID>.
  • Fields are the entity's statements (developer P178, publisher P123, OS P306, language P277, license P275, genre P136, inception P577) — English labels only; unlabeled/undated dropped, not guessed.
  • Deduped against the existing catalogue by QID and slug before writing.

verified stays false. Safe to rebase-merge.

@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jul 10, 2026
@Seungpyo1007 Seungpyo1007 merged commit 30d821a into main Jul 10, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants