Skip to content

data(mobile): import source variant supplements#42

Merged
Seungpyo1007 merged 6 commits into
mainfrom
data/import-staging
Jun 20, 2026
Merged

data(mobile): import source variant supplements#42
Seungpyo1007 merged 6 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jun 20, 2026

Copy link
Copy Markdown
Member

Summary

  • import AIToolBuzz, Beridzeg45 GSM Arena, and AllPhones source variants as unverified mobile records
  • preserve source-specific variants separately with source_key-backed slugs
  • refresh public v1 smartphone/tablet/watch list and detail snapshots

Imported data

  • total added records: 22,702
  • smartphones: 20,993
  • tablets: 1,554
  • watches: 155
  • AIToolBuzz: 11,221
  • Beridzeg45 GSM Arena: 4,613
  • AllPhones: 6,868
  • source years: 1994-2026
  • verification state: all imported records use verified: false

Public dump after refresh

  • smartphones: 90,118
  • tablets: 3,002
  • watches: 349
  • all collections now exceed 100,000 total records

Verification

  • python -m app.validate PASS
  • python TechEngine\integrity_check.py data --strict PASS
  • git diff --check origin/main...HEAD PASS
  • cd site && npm.cmd run build PASS

Closes #1

@Seungpyo1007 Seungpyo1007 added data Dataset changes enhancement New feature or request labels Jun 20, 2026
@github-actions github-actions Bot mentioned this pull request Jun 20, 2026
Open
@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jun 20, 2026
@TechEngineBot

TechEngineBot commented Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
python integrity_check.py TechAPI/data --strict PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 20993 0 0 0 20993 6362
tablet 1554 0 0 0 1554 505
watch 155 0 0 0 155 1
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

smartphone added

  • smartphone/acer/2009/dx650/acer-dx650-aitoolbuzz-17-1gb-2g.json - Acer DX650
  • smartphone/acer/2009/dx650/acer-dx650-beridzeg45-3564-1gb.json - Acer DX650
  • smartphone/acer/2009/m900/acer-m900-aitoolbuzz-97-1gb-3g.json - Acer M900
  • smartphone/acer/2009/m900/acer-m900-beridzeg45-6812-1gb.json - Acer M900
  • smartphone/acer/2011/allegro/acer-allegro-aitoolbuzz-4-8gb-8gb-3g.json - Acer Allegro
  • smartphone/acer/2011/allegro/acer-allegro-beridzeg45-6805-8gb.json - Acer Allegro
  • smartphone/acer/2012/cloudmobile-s500/acer-cloudmobile-s500-aitoolbuzz-16-1gb-8gb-3g.json - Acer CloudMobile S500
  • smartphone/acer/2012/cloudmobile-s500/acer-cloudmobile-s500-beridzeg45-3863-1gb.json - Acer CloudMobile S500
  • smartphone/acer/2012/liquid-gallant-duo/acer-liquid-gallant-duo-aitoolbuzz-60-1gb-4gb-3g.json - Acer Liquid Gallant Duo
  • smartphone/acer/2012/liquid-gallant-duo/acer-liquid-gallant-duo-beridzeg45-132-1gb.json - Acer Liquid Gallant Duo
  • smartphone/acer/2012/liquid-gallant-e350/acer-liquid-gallant-e350-aitoolbuzz-61-1gb-4gb-3g.json - Acer Liquid Gallant E350
  • smartphone/acer/2012/liquid-gallant-e350/acer-liquid-gallant-e350-beridzeg45-3086-1gb.json - Acer Liquid Gallant E350
  • smartphone/acer/2013/liquid-c1/acer-liquid-c1-aitoolbuzz-51-1gb-4gb-3g.json - Acer Liquid C1
  • smartphone/acer/2013/liquid-c1/acer-liquid-c1-beridzeg45-941-1gb.json - Acer Liquid C1
  • smartphone/acer/2013/liquid-e1/acer-liquid-e1-aitoolbuzz-53-1gb-4gb-3g.json - Acer Liquid E1
  • ... 20978 more

tablet added

  • tablet/acer/2011/iconia-tab-a100/acer-iconia-tab-a100-aitoolbuzz-31-1gb-8gb-2g.json - Acer Iconia Tab A100
  • tablet/acer/2011/iconia-tab-a101/acer-iconia-tab-a101-aitoolbuzz-32-1gb-8gb-3g.json - Acer Iconia Tab A101
  • tablet/acer/2011/iconia-tab-a500/acer-iconia-tab-a500-aitoolbuzz-41-1gb-16gb-2g.json - Acer Iconia Tab A500
  • tablet/acer/2011/iconia-tab-a501/acer-iconia-tab-a501-aitoolbuzz-42-1gb-16gb-3g.json - Acer Iconia Tab A501
  • tablet/acer/2012/iconia-tab-a110/acer-iconia-tab-a110-aitoolbuzz-33-1gb-8gb-2g.json - Acer Iconia Tab A110
  • tablet/acer/2012/iconia-tab-a200/acer-iconia-tab-a200-aitoolbuzz-36-1gb-8gb.json - Acer Iconia Tab A200
  • tablet/acer/2012/iconia-tab-a210/acer-iconia-tab-a210-aitoolbuzz-37-1gb-8gb-2g.json - Acer Iconia Tab A210
  • tablet/acer/2012/iconia-tab-a510/acer-iconia-tab-a510-aitoolbuzz-43-1gb-16gb-2g.json - Acer Iconia Tab A510
  • tablet/acer/2012/iconia-tab-a511/acer-iconia-tab-a511-aitoolbuzz-44-1gb-16gb-3g.json - Acer Iconia Tab A511
  • tablet/acer/2012/iconia-tab-a700/acer-iconia-tab-a700-aitoolbuzz-45-1gb-16gb-2g.json - Acer Iconia Tab A700
  • tablet/acer/2012/iconia-tab-a701/acer-iconia-tab-a701-aitoolbuzz-46-1gb-16gb-3g.json - Acer Iconia Tab A701
  • tablet/acer/2013/iconia-tab-a1-810/acer-iconia-tab-a1-810-aitoolbuzz-34-1gb-8gb-2g.json - Acer Iconia Tab A1-810
  • tablet/acer/2013/iconia-tab-a1-811/acer-iconia-tab-a1-811-aitoolbuzz-35-1gb-8gb-3g.json - Acer Iconia Tab A1-811
  • tablet/acer/2013/iconia-tab-a3/acer-iconia-tab-a3-aitoolbuzz-38-1gb-16gb.json - Acer Iconia Tab A3
  • tablet/acer/2013/iconia-tab-b1-710/acer-iconia-tab-b1-710-aitoolbuzz-47-1gb-8gb-2g.json - Acer Iconia Tab B1-710
  • ... 1539 more

watch added

  • watch/allview/2016/allwatch/allview-allwatch-aitoolbuzz-523-1gb-2g.json - Allview Allwatch
  • watch/allview/2018/allwatch-hybrid-s/allview-allwatch-hybrid-s-aitoolbuzz-524-1gb-2g.json - Allview Allwatch Hybrid S
  • watch/allview/2018/allwatch-hybrid-t/allview-allwatch-hybrid-t-aitoolbuzz-525-1gb-2g.json - Allview Allwatch Hybrid T
  • watch/allview/2018/allwatch-v/allview-allwatch-v-aitoolbuzz-526-1gb-2g.json - Allview Allwatch V
  • watch/apple/2015/watch-38mm/apple-watch-38mm-aitoolbuzz-840-0-5gb-8gb-2g.json - Apple Watch 38mm (1st gen)
  • watch/apple/2015/watch-42mm/apple-watch-42mm-aitoolbuzz-841-0-5gb-8gb-2g.json - Apple Watch 42mm (1st gen)
  • watch/apple/2015/watch-edition-38mm/apple-watch-edition-38mm-aitoolbuzz-842-0-5gb-8gb-2g.json - Apple Watch Edition 38mm (1st gen)
  • watch/apple/2015/watch-edition-42mm/apple-watch-edition-42mm-aitoolbuzz-843-0-5gb-8gb-2g.json - Apple Watch Edition 42mm (1st gen)
  • watch/apple/2015/watch-sport-38mm/apple-watch-sport-38mm-aitoolbuzz-874-0-5gb-8gb-2g.json - Apple Watch Sport 38mm (1st gen)
  • watch/apple/2015/watch-sport-42mm/apple-watch-sport-42mm-aitoolbuzz-875-0-5gb-8gb-2g.json - Apple Watch Sport 42mm (1st gen)
  • watch/apple/2016/watch-edition-series-2-38mm/apple-watch-edition-series-2-38mm-aitoolbuzz-844-0-5gb-8gb-2g.json - Apple Watch Edition Series 2 38mm
  • watch/apple/2016/watch-edition-series-2-42mm/apple-watch-edition-series-2-42mm-aitoolbuzz-845-0-5gb-8gb-2g.json - Apple Watch Edition Series 2 42mm
  • watch/apple/2016/watch-series-1-aluminum-38mm/apple-watch-series-1-aluminum-38mm-aitoolbuzz-852-0-5gb-8gb-2g.json - Apple Watch Series 1 Aluminum 38mm
  • watch/apple/2016/watch-series-1-aluminum-42mm/apple-watch-series-1-aluminum-42mm-aitoolbuzz-853-0-5gb-8gb-2g.json - Apple Watch Series 1 Aluminum 42mm
  • watch/apple/2016/watch-series-2-38mm/apple-watch-series-2-38mm-aitoolbuzz-856-0-5gb-8gb-2g.json - Apple Watch Series 2 38mm
  • ... 140 more

Heuristic review

  • Added records by manufacturer/brand: samsung: 2422, lg: 1382, vivo: 1369, motorola: 1278, xiaomi: 1056, huawei: 1009, oppo: 916, nokia: 877

  • Added records by source class: other: 15834, kaggle: 6868

  • Heuristic warnings: 9 total; showing first 9.

    • smartphone: smartphone/jolla/2013/jolla/jolla-jolla-beridzeg45-1933-1gb.json: repeated adjacent word in name
    • smartphone: smartphone/oppo/2018/f9/oppo-f9-aitoolbuzz-7609-6gb-64gb-4g-lte.json: repeated adjacent word in name
    • smartphone: smartphone/oppo/2018/f9/oppo-f9-allphones-6033-6gb-64gb-4g-lte.json: repeated adjacent word in name
    • smartphone: smartphone/oppo/2018/f9/oppo-f9-beridzeg45-1803-4gb.json: repeated adjacent word in name
    • smartphone: smartphone/samsung/2010/smiley/samsung-smiley-aitoolbuzz-9052-1gb-3g.json: unbalanced parentheses in name
    • smartphone: smartphone/samsung/2010/smiley/samsung-smiley-allphones-7304-1gb-2g.json: unbalanced parentheses in name
    • smartphone: smartphone/vivo/2018/v11/vivo-v11-aitoolbuzz-12102-6gb-64gb-4g-lte.json: repeated adjacent word in name
    • smartphone: smartphone/vivo/2018/v11/vivo-v11-allphones-7691-6gb-64gb-4g-lte.json: repeated adjacent word in name
    • smartphone: smartphone/vivo/2018/v11/vivo-v11-beridzeg45-2644-6gb.json: repeated adjacent word in name

@TechEngineBot

TechEngineBot commented Jun 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 189 0 60 129 60 0.0%
soc 2104 58 2046 0 2104 2.8%
smartphone 90118 184 89934 0 90118 0.2%
tablet 3002 0 3002 0 3002 0.0%
watch 349 0 349 0 349 0.0%
pda 110 0 110 0 110 0.0%
gpu 2030 0 2030 0 2030 0.0%
cpu 3977 976 3001 0 3977 24.5%
all 101879 1218 100532 129 101750 1.2%

Warning

Tracked verified coverage is below 50% for brand 0.0% (0/60), tablet 0.0% (0/3002), watch 0.0% (0/349), pda 0.0% (0/110), gpu 0.0% (0/2030), smartphone 0.2% (184/90118), all 1.2% (1218/101750), soc 2.8% (58/2104), and 1 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 152
GPU cross-source ratio outliers + sanity 18

@Seungpyo1007 Seungpyo1007 force-pushed the data/import-staging branch from b40d045 to 078f398 Compare June 20, 2026 19:41
@Seungpyo1007 Seungpyo1007 force-pushed the data/import-staging branch from 078f398 to bc3c9a9 Compare June 20, 2026 19:52
@Seungpyo1007 Seungpyo1007 merged commit 1aff4ec into main Jun 20, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jun 20, 2026
Seungpyo1007 added a commit that referenced this pull request Jul 10, 2026
@Seungpyo1007
data(software,game): add 11,611 software + 6,800 games from Wikidata
f7d4c2a 
New `software` category (TechEngine PR #42): 11,611 software products from
Wikidata (P31=Q7397) — name, release_date, developers, publishers,
operating_systems, programming_languages, licenses, genres. Plus 6,800 more
video games recovered from Q7889 subclasses (browser/arcade/mobile games) the
direct-P31 pass missed. Bumps engine submodule to the software-category pin +
mirrors SOFTWARE_REQUIRED into app/validate.py. All from Wikidata, no
fabrication; verified:false; sharded.

Refs #1
@Seungpyo1007 Seungpyo1007 mentioned this pull request Jul 10, 2026
Merged
Seungpyo1007 added a commit that referenced this pull request Jul 10, 2026
@Seungpyo1007
data(software,game): add 11,611 software + 6,800 games from Wikidata
5822cc1 
New `software` category (TechEngine PR #42): 11,611 software products from
Wikidata (P31=Q7397) — name, release_date, developers, publishers,
operating_systems, programming_languages, licenses, genres. Plus 6,800 more
video games recovered from Q7889 subclasses (browser/arcade/mobile games) the
direct-P31 pass missed. Bumps engine submodule to the software-category pin +
mirrors SOFTWARE_REQUIRED into app/validate.py. All from Wikidata, no
fabrication; verified:false; sharded.

Refs #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Seungpyo1007 Seungpyo1007

@TechEngineBot TechEngineBot

Labels

data Dataset changes enhancement New feature or request

Projects

TechAPI-Project
Status: Done

Milestone

Massive dataset rebuild (1989-2026)

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants

@Seungpyo1007 @TechEngineBot