Flet 0.86 on-device compatibility: forge fix_wheel fixes, recipe loader/packaging fixes, deploy-via-dispatch#106
Merged
Merged
Conversation
When python_build_run_id is set, the recipe-tester currently only used that run's CROSS-COMPILE support tree; the ON-DEVICE libpython was still downloaded by serious_python from a hardcoded flet-dev/python-build RELEASE -- so a runtime fix (e.g. the 3.13+ mimalloc x86_64 seccomp open() crash) couldn't be validated on-device without publishing a release. Now: setup.sh keeps the run's python-*-dart-* tarballs in downloads/, and the android recipe-tester step seeds them into serious_python's cache ($FLET_CACHE_DIR/python-build/v<ver>/) with a fresh mtime. serious_python's download task is onlyIfModified(true), so it 304-skips and bundles THIS run's runtime. No release needed. Safe: a re-download just yields the released runtime (never a false pass). Scoped to android (where the mimalloc crash manifests, on CI's x86_64 emulator). iOS seeding (darwin cache is date-keyed + marker-based) is a follow-up for the _pyrepl leg.
Mirror the android runtime seed for iOS. The darwin cache is date-keyed ($FLET_CACHE_DIR/python-build/v<ver>-<date>/) and its download is a plain [ ! -f tarball ] check, so pin SERIOUS_PYTHON_BUILD_DATE to a sentinel we control (per-step, iOS job only) and drop the fixed python-ios-dart-<ver> tarball at that exact path; prepare_ios.sh finds it, skips the download, and extracts THIS run's runtime -- carrying the _pyrepl fix to the iOS 3.14 sim. Only fires when python_build_run_id supplied the tarball (setup.sh kept it); the sentinel date is never fetched because the file already exists.
Flet 0.86 bundles site-packages as a compressed sitepackages.zip imported via zipimport. Packages that read a bundled DATA file through a real __file__ path (rather than importlib.resources) then crash on-device with NotADirectoryError, because __file__ resolves inside the zip. This regressed a batch of recipes that were green on-device before the 0.86 migration (#104). serious_python exposes an Android escape hatch — [tool.flet.android].extract_packages (flet build --android-extract-packages) — that ships listed packages extracted to disk, placed on sys.path before the zip, so __file__-relative data reads resolve. Wire it per-recipe: - pyproject.toml.tpl: new [tool.flet.android] extract_packages = [__EXTRACT_PACKAGES__] - stage_recipe.sh: expand the token from recipes/<pkg>/extract_packages.txt (one relative path per line; absent file => [] no-op) - recipes/{matplotlib,astropy,scikit-learn,thinc,spacy}/extract_packages.txt Verified on-device (android arm64, flet 0.86.0.dev2): - matplotlib: NotADirectoryError(matplotlib/mpl-data/matplotlibrc) -> 1 passed - astropy: NotADirectoryError(astropy/CITATION) -> 5 passed scikit-learn (sklearn/utils/_repr_html/estimator.css), thinc (thinc/backends/_custom_kernels.cu) and spacy (imports thinc) use the identical data-file-inside-package pattern and are covered by the same mechanism. Not covered by extract_packages (separate 0.86 issues, tracked separately): - python-magic: magic.mgc lives in flet-libmagic's opt/ tree; serious_python's copyOpt copies only **/*.so, so the data file is dropped entirely. - opencv-python: cv2's loader requires the literal config.py SOURCE, which 0.86 strips when compiling bundled packages to .pyc.
Replaced hardcoded version strings with `{% set version %}` variables across several recipes to streamline maintenance and update consistency.
Move the path-hungry extract list from the recipes/<pkg>/extract_packages.txt sidecar (previous commit) into a first-class `extract_packages:` field in the recipe's meta.yaml — its natural home next to patches/requirements, and one fewer per-recipe file convention. - schema/meta-schema.yaml: document `extract_packages` (array of strings, default []); forge's build ignores it, the recipe-tester consumes it. - read_extract_packages.py: render meta.yaml the way forge does (Jinja -> YAML) and print the list; run hermetically via `uv run --with jinja2 --with pyyaml` so the deps are present regardless of the caller's environment. - stage_recipe.sh: read the field via that helper instead of the sidecar file. - move matplotlib/astropy/scikit-learn/thinc/spacy lists into their meta.yaml; delete the 5 sidecars. Output is unchanged: the generated pyproject.toml is byte-identical to the sidecar version (matplotlib -> ["matplotlib"], spacy -> ["spacy","thinc"], numpy -> []), the schema still validates metas with and without the field, and matplotlib re-verified green on-device (arm64, flet 0.86.0.dev2).
Adds a conda-style `test:` section to the recipe schema (forge's schema is "a subset of conda syntax", which has a `test:` block) and reads test-only deps from `test.requires` instead of a per-recipe `tests/requirements.txt` — one declarative config surface (meta.yaml), with `tests/` left as pure test code. - schema/meta-schema.yaml: new `test.requires` (array of PEP 508 specs, default []). - read_extract_packages.py -> read_meta_list.py: generalized to print any list-valued meta field by dotted key (`extract_packages`, `test.requires`), rendered Jinja->YAML the way forge loads meta. - stage_recipe.sh: read both lists via a shared `meta_list()` helper; a leftover tests/requirements.txt now errors (pointing at test.requires) instead of being silently ignored. No recipes currently declare test deps (no tests/requirements.txt existed), so nothing to migrate. Verified: schema validates with and without the field; the generated pyproject injects test.requires into `dependencies` and keeps extract_packages, structurally identical to the prior requirements.txt path; the legacy guard fires. extract_packages stays TOP-LEVEL — it is a deployment property of the wheel (a real app that bundles the package needs it too), not a test-only concern.
Flet 0.86's serious_python Android packaging only relocates a native module into
jniLibs and writes the `.soref` marker its on-device importer needs for
extensions whose filename carries a CPython ABI tag (`*.cpython-*.so` /
`*.abi3.so`). A bare `NAME.so` is treated as a plain dependency library, gets no
`.soref`, and fails to import on-device with ModuleNotFoundError. CMake / SWIG /
Cython / nanobind builds routinely emit un-tagged extensions because they can't
derive the target SOABI when cross-compiling — this broke ncnn (ncnn.ncnn),
faiss-cpu (_swigfaiss), coolprop (CoolProp/_constants) and onnxruntime (x86_64
leg only) on 0.86.
fix_wheel now renames each bare `.so` that exports `PyInit_<basename>` (a genuine
CPython extension) to `<basename>.cpython-3X.so`, using llvm-nm to distinguish
extensions from real dependency libraries (left untouched). One fix covers the
whole class — present and future — instead of per-recipe CMake patches.
Verified:
- on-device (android arm64): a manually ABI-tagged ncnn wheel -> serious_python
relocates it (lib/.../libncnn-ncnn.so + ncnn/ncnn.soref) and `import ncnn`
works; recipe-tester ncnn 3/3 passed (was ModuleNotFoundError: ncnn.ncnn).
- the tagging logic, run standalone over the published wheels, tags exactly
ncnn/ncnn.so, faiss/_swigfaiss.so and CoolProp/{CoolProp,_constants}.so, and
SKIPS flet-libcpp-shared's opt/lib/libc++_shared.so (no PyInit_).
…roid)
Both packages load their native lib via ctypes.util.find_library('sodium' /
'opaque'), which returns None on Android (no ld cache — find_library relies on
gcc/ld/ldconfig) and on iOS (the lib is framework-ized). LoadLibrary(None) then
yields a nameless handle and __init__ raises "Unable to find libsodium/libopaque".
flet-libsodium / flet-libopaque bundle the libraries and serious-python surfaces
them into the APK's jniLibs, so fall back to loading the bundled soname
(libX.so / libX.fwork / libX.dylib) when find_library() comes up empty.
- pysodium: new mobile.patch (recipe had none) + register it in meta.yaml.
- opaque: extend the existing mobile.patch (the setup.py install_requires fix)
with the same loader accommodation; opaque imports pysodium, so both are fixed
together.
Verified on-device (android arm64, flet 0.86.0.dev2): pysodium 2/2 passed,
opaque 2/2 passed (import + a real registration/credential roundtrip exercising
both libopaque and libsodium). Was: ValueError: Unable to find libsodium/libopaque.
…rtlib (0.86) pycryptodome ctypes-loads its native modules (Crypto/Util/_raw_api.py: load_pycryptodome_raw_lib) by probing os.path.isfile() for the .so next to the package's __file__. Under Flet 0.86 the package lives inside sitepackages.zip, so every probe misses and it raises "Cannot load native module 'Crypto.Util._cpuid_c'". The extensions ARE relocated into the APK's jniLibs (they are .abi3.so, which serious_python tags and writes a .soref for), but the loader ctypes-loads by path, not via import, so it never finds them. Extend the existing mobile.patch loader: after the on-disk probes miss, ask the import system (importlib.util.find_spec) for the resolved on-device origin — the serious_python .soref finder returns a spec whose origin is the loadable jniLibs / base.apk!/lib path — and dlopen that. No hardcoded soname mangling; it reuses serious_python's own resolution through the public importlib API. The iOS (.fwork) and desktop paths are unchanged (they succeed in the first loop, before this fallback is reached). pycryptodomex gets the identical fix under the Cryptodome namespace. Verified on-device (android arm64, flet 0.86.0.dev2), extensions resolving from their .soref markers: - pycryptodome 3/3 passed (AES import + AES-CBC roundtrip + SHA-256 vector) - pycryptodomex 2/2 passed (AES-GCM roundtrip + SHA-256 vector) Was: OSError: Cannot load native module ... Not found _cpuid_c.cpython-312.so/.abi3.so/.so/.fwork.
…et 0.86)
llama_cpp/_ctypes_extensions.py:load_shared_library builds candidate paths under
base_path = dirname(__file__)/lib and gates each on Path.exists() before
ctypes.CDLL. Under Flet 0.86 site-packages ships as a zip, so base_path is inside
sitepackages.zip and never exists on disk; serious-python relocated the bundled
libllama/libggml* into the APK's jniLibs. Every probe missed ->
FileNotFoundError: Shared library with base name 'llama' not found.
Extend the existing mobile.patch loader: when no candidate path exists on disk,
fall back to ctypes.CDLL by bare soname (lib<name>.so) for both the ggml
dependency preload and the main libllama load, so the Android linker resolves
them from jniLibs (loaded RTLD_GLOBAL, so the preloaded deps satisfy libllama's
DT_NEEDED). iOS (.fwork) and desktop paths are unchanged — their base_path is a
real directory, so the existing Path.exists() branch still wins first.
Verified on-device (android arm64, flet 0.86.0.dev2): llama-cpp-python 1/1 passed
(test_native_lib_callable), with libllama/libggml{,-base,-cpu}.so resolving from
jniLibs. Was: FileNotFoundError: Shared library with base name 'llama' not found.
llama on iOS is a separate blocker (serious-python #223 — framework-izing the
interdependent ctypes .dylib libs — not this Android fix).
…droid The `jq` Python binding is a top-level extension module `jq`, which serious_python (Flet 0.86 Android) relocates into jniLibs under the mangled name `libjq.so` — the same basename as flet-libjq's shared library. Both mapped to lib/<abi>/libjq.so, the extension clobbered the C lib, and `import jq` failed with "cannot locate symbol jq_teardown referenced by libjq.so" (the C lib carrying jq_teardown was overwritten by the extension). NOT a missing symbol export — libjq.so exports all 177 jq_/jv_ symbols; it was a jniLibs name collision. Fix: static-link jq + oniguruma into the extension so it is self-contained and no libjq.so ships at runtime. - flet-libjq: configure --with-pic (PIC static archives), keep libjq.a/libonig.a, drop the shared libraries. build 10 -> 11. - jq: requirements host -> host_build (flet-libjq linked at build time, not a runtime dep) and mobile.patch links -l:libjq.a/-l:libonig.a (static) instead of -ljq/-lonig (dynamic). build 1 -> 2. Verified on-device (android arm64, flet 0.86.0.dev2): jq 2/2 passed (filter + first). The APK ships a single self-contained lib/arm64-v8a/libjq.so (the ~1 MB extension; DT_NEEDED lists no libjq.so/libonig.so) — no collision. Was: ImportError: dlopen failed: cannot locate symbol "jq_teardown".
…fields Fold this session's findings into the git-tracked skills so they're not re-derived: - forge-error-catalogue: new umbrella entry (the 0.86 Android packaging change and why __file__-relative loaders/data reads that worked under 0.85 fail now) plus the specific symptom -> fix entries — NotADirectoryError -> extract_packages; untagged native .so ModuleNotFoundError / "circular import" -> forge fix_wheel ABI-tag; pycryptodome "Cannot load native module" -> importlib.find_spec().origin; llama FileNotFoundError / ctypes find_library-None -> bare-soname load from jniLibs; a top-level ext colliding with a flet-lib* at lib<pkg>.so -> static-link. - new-mobile-recipe + local-recipe-testing: the new meta.yaml `extract_packages` and `test.requires` fields (test.requires replaced tests/requirements.txt), the Flet 0.86 build-toolchain pin (uvx --prerelease from pypi.flet.dev) for the local loop, and an APK-contents verification gotcha.
….86)
selectolax/__init__.py did `from . import lexbor, modest, parser`, but
`selectolax/modest/` is only a directory of Cython `.pxi` includes (parser.pyx does
`include "modest/*.pxi"`, so they compile into parser.cpython-*.so) — there is no
runtime `modest` module. Under normal CPython `from . import modest` resolves as an
empty PEP 420 namespace package; under Flet 0.86 Android, site-packages ship as
sitepackages.zip and serious-python only synthesizes an __init__ for dirs with an
importable member, so include-only modest/ is invisible to zipimport and the import
fails ("cannot import name 'modest' from partially initialized module 'selectolax'").
Drop the vestigial import — selectolax.modest has no runtime API; the engines are
selectolax.parser (Modest backend) and selectolax.lexbor, both unaffected.
Verified on-device (android arm64, flet 0.86.0.dev2): selectolax 2/2 passed (modest
+ lexbor parsers). Was: ImportError: cannot import name 'modest'. Also documents
the symptom in forge-error-catalogue (the include-only namespace-dir entry).
…id (Flet 0.86)
Under Flet 0.86 Android the native cv2 extension is relocated to jniLibs
(importable only as the submodule cv2.cv2 via a cv2/cv2.soref marker) and
config.py is compiled to config.pyc, so cv2's stock loader dies two ways:
"missing configuration file: ['config.py']", and — if the native is loaded as
cv2.cv2 — "Submodule name should always start with a parent module name. Parent
name: cv2.cv2. Submodule name: cv2" (OpenCV's compiled bindings require the
top-level name "cv2").
mobile.patch note 3 inserts a fast-path in cv2/__init__.py after the recursion
guard: resolve the relocated extension via find_spec('cv2.cv2') and load it
through a fresh ExtensionFileLoader("cv2", origin) so its runtime __name__ is
the required top-level "cv2", relink, and load the extra pure-python submodules.
On desktop/non-0.86 find_spec is None → stock loader runs unchanged. Pairs with
extract_packages: [cv2] (the extra-submodule scan os.listdir()s the package dir).
Verified on-device (arm64) 4/4 against the byte-identical published 4.10.0.84
loader. Skills: dedicated cv2 entry + cross-refs in forge-error-catalogue.
…et 0.86 Android)
Under Flet 0.86 Android the compiled magic database never reached the device:
magic.mgc ships in flet-libmagic's opt/share/misc/ tree, but serious-python's
copyOpt copies only **/*.so out of a flet-lib* opt/ tree (and splitSitePackages
skips opt/), so it landed in neither jniLibs nor sitepackages.zip. The loader
pointed magic_file at <site-packages>/opt/share/misc/magic.mgc, os.path.exists()
was False, and magic_load(cookie, None) failed "could not find any valid magic
files!".
Fix (mobile.patch note 2 + meta script_env), two parts:
(a) setup.py copies magic.mgc INTO the `magic` package at build time (path fed
via the new FLET_MAGIC_MGC script_env, pointing at flet-libmagic's cross-env
opt/) and adds it to package_data, so the DB ships inside python-magic's own
wheel and rides 0.86's all-files sitepackages.zip. Arch-neutral -> wheel
stays pure; flet-libmagic remains the per-platform Requires-Dist for the .so.
(b) magic/__init__.py loads it: real path when one exists (iOS/desktop/
extract_packages), else bind magic_load_buffers (public in file>=5.32) and
load the DB from an in-memory buffer read via importlib.resources (reads
through zipimport). Buffer stashed on the Magic instance (kept alive by the
module-level _instances cache) since libmagic references it in place.
Zero consumer config (no extract_packages needed). Verified on-device (arm64)
2/2: libmagic loads, magic.from_buffer(png) -> image/png. Built libmagic.so
confirmed to export magic_load_buffers; magic.mgc (10.3 MB) confirmed in the
wheel's sitepackages.zip.
…heel + load from memory)
…ython, opaque, pycryptodome, pycryptodomex, pysodium) These recipes' mobile.patch changed (Flet 0.86 ctypes-loader fixes) but their build number was left unchanged, so a published wheel at the old build number would shadow the fix. Bump so the patched wheel wins.
…init__ IS the native ext) + local SP-override test technique
The static-link change used GNU ld's `-l:libjq.a`/`-l:libonig.a` for BOTH platforms, but Apple's ld64 has no `-l:` syntax, so the iOS build failed with `ld: library ':libjq.a' not found` on every Python. The jniLibs `libjq.so` name collision the `-l:` form guards against is Android-only; flet-libjq ships only static `.a` on both platforms, so on iOS a plain `-ljq`/`-lonig` resolves to the static archive just the same. Gate the link flags on `sysconfig.get_platform()`: keep `-l:libjq.a` on Android (the on-device-verified form), use `-ljq`/`-lonig` on iOS. Verified locally on BOTH: iphonesimulator arm64 builds and statically embeds 107 jq_* symbols with no external libjq/libonig dylib dep; android arm64 unchanged (no libjq DT_NEEDED). Build 2 -> 3.
… gate static-link per platform
serious_python's darwin packaging turns every site-packages native .so into a .framework binary that SwiftPM LINKS (a Package.swift binaryTarget the plugin depends on). ld only links MH_OBJECT/MH_DYLIB, so any extension that is an MH_BUNDLE aborts the iOS-sim app build with "Unsupported mach-o filetype (only MH_OBJECT and MH_DYLIB can be linked) in <pkg>.framework/<pkg>". iOS CPython's sysconfig sets LDSHARED='...-dynamiclib -F . -framework Python', which setuptools/Cython/meson/maturin honor (-> MH_DYLIB), but CMake/scikit-build recipes link a Python MODULE with Apple's default -bundle (-> MH_BUNDLE), ignoring LDSHARED. So the failing set is exactly the CMake ones: opencv (cv2), ncnn, coolprop, faiss (_swigfaiss); the setuptools/Cython/meson ones already ship MH_DYLIB and pass. (llama-cpp-python is a DIFFERENT bug — ctypes interdependent SHARED dylibs, sp #223 — not MH_BUNDLE; this is a no-op there.) fix_wheel's new iOS branch mirrors the Android block: for each MH_BUNDLE .so it injects an LC_ID_DYLIB into the header's free padding and flips the filetype to MH_DYLIB, then ad-hoc re-signs (the edit invalidates the linker signature). dlopen (the import path) works on both filetypes, so nothing regresses; serious_python's own `install_name_tool -id` overwrites the placeholder id. Validated: converting cv2.abi3.so + ncnn.so makes `ld` go from the exact error to a clean link (exit 0); a real forge build of ncnn (iphonesimulator arm64) runs the new branch (MH_BUNDLE -> MH_DYLIB, codesign ok) and the recipe-tester iOS-sim app build gets past the cv2.cv2.framework link (Packaged Python app OK).
…=-dynamiclib; forge fix_wheel converts)
…hed copy -> runtime error (opaque->pysodium); prebuild the dep recipe
…e (Flet 0.86) [skip ci]
The Flet 0.86 cv2 loader fast-path only handled Android (find_spec('cv2.cv2')
origin is a real .so from the soref finder). On iOS the native is framework-ized:
serious_python leaves a cv2/cv2.fwork marker whose text is the app-relative path
to the framework binary, so find_spec's origin ends in .fwork, the old fast-path
fell through, and the stock loader recursed ("recursion is detected during
loading of cv2 binary extensions").
Unify the fast-path: when the resolved origin ends in .fwork, read it and join
with dirname(sys.executable) to get the framework binary (exactly as CPython's
AppleFrameworkLoader.create_module does), then load it under the required
top-level name "cv2" via ExtensionFileLoader. Android (.so origin) is unchanged.
Verified: Android arm64 4/4 (unchanged path); iOS bundle->dylib link + framework
packaging validated locally, loader .fwork resolution mirrors AppleFrameworkLoader
— CI-confirmed next to the coolprop iOS pass. Build 2 -> 3.
… local gotchas (per-arch staging ignores find-links; device_info_plus old-Xcode override) [skip ci]
…dylib [skip ci]
pyobjus iOS: Flet 0.86 compiles every bundled .py -> .pyc, and pyobjus ships
Python-2 example scripts (share/pyobjus-examples/*.py) whose `print` statements
raise SyntaxError, failing the iOS app build. Drop the `examples` data tree from
setup.py (demos, no runtime role). Build 1 -> 2.
pyarrow iOS: its _json/_pyarrow_cpp_tests/lib extensions are CMake MODULE
MH_BUNDLE binaries that Xcode refused to link ("Unsupported mach-o filetype")
— the same class the forge fix_wheel MH_BUNDLE->MH_DYLIB conversion now handles.
Bump build 1 -> 2 so the forge-fixed rebuild wins over any published wheel.
…n-build gap; re-enable in build_ios.py [skip ci]
… patch [skip ci]
The patch guarded `from multiprocessing.connection import ...` in
sklearn/callback/_transport.py on the (false) premise that iOS disables
_multiprocessing. It's dead + ineffective:
- python-build re-enables _multiprocessing on iOS (build_ios.py), so the
guarded import succeeds and the except branch never fires;
- the actual sklearn iOS failure is sklearn/utils/validation ->
multiprocessing/resource_tracker -> `import _posixshmem` (a DIFFERENT module,
imported earlier), which this patch never touched — the CI run failed with it
applied.
The real fix is re-enabling _posixshmem in python-build (build_ios.py). Build 1 -> 2.
…oader [skip ci]
On iOS serious-python relocates each bundled dylib (libggml-base/libggml-cpu/
libggml/libllama) into a code-signed <name>.framework and leaves a lib<name>.fwork
TEXT marker whose content is the app-relative framework-binary path. The loader
was trying to `ctypes.CDLL("lib<name>.fwork")` — i.e. dlopen the text marker —
which fails, so the ggml chain never preloaded and libllama's sibling @rpath refs
went unresolved (app hung/crashed at import; empty console.log, 900s timeout).
Now the loader RESOLVES the .fwork marker to the real framework binary (as
CPython's AppleFrameworkLoader does) before CDLL, and preloads the chain in
dependency order with RTLD_GLOBAL. serious-python leaves the dylibs' install-ids
unchanged (@rpath/lib<name>.dylib), so each preloaded framework satisfies the next
lib's sibling reference by install-id (the pyarrow shim pattern). Build 2 -> 3.
NOTE: if the app still crashes at LAUNCH (before Python), the frameworks are being
linked with the un-rewritten sibling @rpath refs — that's the serious-python #223
gap and needs an SP-side install-name rewrite, not a recipe fix.
…ependent dylibs) — reconcile framework install-names; + local-test-unreleased-sp recipe (local flet-cli, --python-version 3.12, SERIOUS_PYTHON_APP) [skip ci]
…ix branch in recipe-tester template - build-wheels-version.yml: bump the python_build_run_id fallback 29193662574 -> 29242819654 (flet-dev/python-build run re-enabling _posixshmem for iOS multiprocessing/resource_tracker — unblocks scikit-learn iOS and any multiprocessing user). - recipe-tester/pyproject.toml.tpl: add a dependency_overrides block pulling serious_python from flet-dev/serious-python@fix/soref-package-init so recipe tests validate against the unreleased iOS fixes (#223 framework relocation + _SorefFinder) before they ship. TEMPORARY — remove once sp is released. Lets recipes be validated end-to-end with all pending fixes prior to PR. [skip ci]
… (git override in recipe-tester template) + python-build (run-id fallback bump); python_build_run_id + empty mobile_test_pythons dispatch rows [skip ci]
…BUNDLE->MH_DYLIB reship These 3 CMake recipes are fixed by forge fix_wheel's iOS MH_BUNDLE->MH_DYLIB conversion (06f4f71, this branch), not by any recipe-file change. On push to main the detect job only builds recipes whose recipes/** paths changed, so without a bump they would neither rebuild (unchanged) nor reship (a same-tag rebuild is a 409 duplicate-skip on gemfury) — the fixed wheels would never reach users. Comment-only bump mirrors pyarrow's. ncnn 1->2, coolprop 10->11, faiss-cpu 1->2. [skip ci]
…rk's remote HEAD, not local commits; verify run headSha == local HEAD [skip ci]
…d bump (3->2) Each was bumped twice within this branch (once for the Android fix, once for iOS), but nothing publishes until merge and the build number only needs to end up > the last-published (main = build 1). Collapse to a single 1->2 bump. Harmless either way; just tidier. [skip ci]
…thon override
1. Android app build: pass `--arch x86_64` to `flet build apk` (the on-device
test runs on an x86_64 emulator). Fixes recipes with excluded_arches
(pyarrow drops armeabi-v7a) whose all-ABI app build failed pip resolution
('No matching distribution'), and cuts build time.
2. New `serious_python_ref` workflow input (build-wheels.yml + build-wheels-version.yml,
default 'fix/soref-package-init'). stage_recipe.sh now appends the
serious_python git dependency_overrides from $SERIOUS_PYTHON_REF instead of
the template hardcoding them; empty ref -> published serious_python. The
pyproject template no longer hardcodes the branch. Pass -f serious_python_ref=''
to test against released sp. [skip ci]
…d dlopen On Flet 0.86 Android, site-packages ship in sitepackages.zip. llama's native libs live in llama_cpp/lib/ (CMAKE_INSTALL_LIBDIR), which is neither an opt/ tree (so copyOpt never relocates them to jniLibs) nor a set of PyInit extensions (so the .soref relocation skips them) — they stay INSIDE the zip, where _ctypes_extensions.py's ctypes.CDLL(base_path / libllama.so) fails with 'dlopen ... library not found'. The CI failure confirmed the lib was still at sitepackages.zip/llama_cpp/lib/libllama.so (not jniLibs), proving flet does not relocate these ctypes libs. Extracting llama_cpp to disk makes base_path a real filesystem dir so the preloaded ggml chain + libllama load. App-level config, so the wheel is unchanged (build stays 2); Android-only (iOS uses .fwork, already green). [skip ci]
… (Android) -> extract_packages; distinct from PyInit ABI-tag; llama-cpp-python case [skip ci]
…x Android dlopen" This reverts commit aaa268b.
…e on Android (real fix) The build-3 loader raised RuntimeError on the FIRST (disk/zip) candidate's miss, never reaching its own bare-soname jniLibs fallback right below. Under Flet 0.86 Android the .so IS relocated into jniLibs (verified: unzip -l APK shows lib/arm64-v8a/libllama.so + libggml*), so the disk path always misses and only the soname load works. Change the main-load except to fall through (pass) instead of raising. Build 2->3 (wheel change). extract_packages was a RED HERRING (reverted): local on-device test showed it moved the .pyc to disk but the .so still went to jniLibs, so the raise still fired. Verified on-device (Android arm64, test_native_lib_callable PASS) with the loader fall-through. Skill entry corrected. [skip ci]
…ader (py3.13)
Python 3.13+ reports sys.platform == 'android' (PEP 738) where 3.12 reported
'linux'. The mobile.patch loader's _candidate_paths only matched linux/freebsd/
darwin/ios/win32, so on py3.13 Android it hit the else branch and raised
'Unsupported platform' at import — before the bare-soname jniLibs fallback (added
in build 3) could run. Add 'android' to the linux branch so it returns lib<name>.so,
which misses on the Flet 0.86 zip and falls through to the jniLibs load, exactly as
on py3.12. Build 3 -> 4. (General py3.12->3.13 tell: any loader gating on
sys.platform.startswith('linux') breaks on 3.13 Android.)
…tell (PEP 738) — llama case [skip ci]
…PyInit Two Android fix_wheel gaps surfaced by pymongo and onnxruntime on py3.13: - pymongo: a plain-setuptools sdist builds extensions in-place into a source tree forge reuses across ABI slices, so bdist_wheel globbed the prior slice's arch-tagged .so into the next slice's wheel (x86_64 wheel carried the arm64 _cbson/_cmessage byte-for-byte). serious_python strips the arch off the SOABI tag when writing .soref -> two arches collide on one <name>.soref -> Gradle 'duplicate entry'. Fix: drop any .so whose ABI-tag triplet != this target's platform_triplet (a per-arch wheel must be arch-pure). - onnxruntime: its pybind module exports the init symbol versioned via a linker version script -> 'llvm-nm -D' prints PyInit_..._state@@VERS_1.0, so the exact 'PyInit_<module>' membership test missed it, the .so shipped bare on BOTH arches, no .soref, ModuleNotFoundError on device. Fix: match the base name tolerating a trailing @Version suffix. Both are generic fix_wheel changes; no recipe edits. Catalogue updated. [skip ci]
matplotlib's ft2font extension crashed at iOS app LAUNCH with SIGSEGV, before Py_Initialize() — 0-byte console.log, faulting in dyld runInitializers -> _GLOBAL__sub_I_ft2font_wrapper.cpp -> pybind11::gil_scoped_acquire -> PyGILState_Ensure. Root cause: matplotlib's vendored pybind11 enum helper (src/_enums.h) declares each enum via P11X_DECLARE_ENUM, which expands to a namespace-scope global whose initializer acquires the GIL + calls pybind11::cast — i.e. touches the Python C-API from a C++ static initializer. Harmless on desktop/Android (ext dlopen'd at import, interpreter up), fatal on iOS: serious_python links each ext as a framework whose static initializers dyld runs at app launch, before the interpreter exists. Fix (mobile.patch on _enums.h): record each enum spec as plain C++ data at static-init (no GIL, no pybind11::cast) and defer all Python work to bind_enums(), which already runs inside PYBIND11_MODULE (at import). Build 1 -> 2 (also republishes the iOS wheel with forge's MH_BUNDLE->MH_DYLIB conversion; the published build-1 iOS wheel was a stale MH_BUNDLE). Verified on-device: patched wheel on the iOS simulator launches, test_png PASSED, EXIT 0; disassembly confirms the rebuilt static initializer calls only plain C++. Catalogue updated (forge-error-catalogue: iOS launch SIGSEGV in runInitializers). [skip ci]
- build-wheels(.yml/-version.yml): new `publish` input (boolean, default false). The publish step now fires when a caller opts in (publish: true — e.g. a deploy workflow_dispatch with a pinned python_build_run_id) OR on the existing automatic path (push to main with an empty python_build_run_id). Lets us deploy wheels via dispatch with a specific python-build run + sp ref. - pymongo 1->2, onnxruntime 1->2: their build-1 wheels are already published, so a rebuild would 409-skip; bump so forge fix_wheel's foreign-arch drop (pymongo) and symbol-versioned-PyInit ABI-tag (onnxruntime) actually reach pypi.flet.dev. [skip ci]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flet 0.86 changed the mobile packaging model — site-packages ship as a compressed
sitepackages.zip(zipimport); native.soare relocated tojniLibs/<abi>/with.sorefmarkers on Android; and on iOS each extension is framework-ized and linked into the app at
launch. That broke a cluster of recipes on-device even though their wheels build cleanly.
This branch fixes the regressions across three layers — forge core, the recipes, and the
CI / recipe-tester — and adds a deploy-via-dispatch path so fixed wheels can be published to
pypi.flet.dev without a push to
main.forge core —
src/forge/build.py(fix_wheel),schema/meta-schema.yamlThree generic, recipe-agnostic packaging fixes:
MH_BUNDLE→MH_DYLIB(06f4f71). CMake/scikit-build extensions link asMH_BUNDLE(they ignore CPython's
LDSHARED=-dynamiclib); serious_python framework-izes and links them,and
ldrejects a bundle (Unsupported mach-o filetype (only MH_OBJECT and MH_DYLIB can be linked)).fix_wheelinjects anLC_ID_DYLIB, flips the filetype, and re-signs ad-hoc..sodrop (385430f). A plain-setuptools sdist that buildsbuild_ext --inplaceinto forge's reused source tree leaks a prior ABI slice's arch-tagged.sointo the next slice's wheel (pymongo shipped the arm64
_cbson/_cmessagebyte-for-byte inthe x86_64 wheel → serious_python strips the arch off the SOABI tag → two arches collide on one
<name>.soref→ Gradleduplicate entry).fix_wheelnow drops any.sowhose ABI-tag triplet≠ the target's.
PyInitABI-tag (385430f). forge ABI-tags a bareNAME.soexporting
PyInit_<name>so serious_python writes its.soref. onnxruntime's pybind moduleexports the init symbol version-scripted (
PyInit_..._state@@VERS_1.0), which the exact-matchtest missed → the
.soshipped bare on both arches → on-deviceModuleNotFoundError. The matchnow tolerates a trailing
@versionsuffix.Plus a new
extract_packagesmeta field (schema + recipe-tester wiring): Android-only list ofpackages to ship extracted to disk rather than inside
sitepackages.zip, for "path-hungry"packages that read bundled data through a real
__file__path (NotADirectoryErrorinside a zip).Recipe fixes
Flet 0.86 loader / packaging:
jniLibsbare-soname on Android, accepts
sys.platform == "android"(PEP 738, py3.13+), and resolves iOS.fworkframework markers.cv2under the top-level name (Android) + resolve the iOSframework-ized native.
magic.mgcin-wheel and load it from memory (magic_load_buffers);Flet 0.86's
copyOptdrops the flet-libopt/data file.importlib.modestnamespace import._enums.hP11X_DECLARE_ENUM) acquired the Python GIL in a C++ static initializer, which dyld runs atapp launch before
Py_Initialize()under serious_python's link-at-launch iOS model. Deferredthe GIL/pybind work to
bind_enums()(runs insidePYBIND11_MODULE, at import).ld-l:NAME.avs Appleld64 -lX);flet-libjqkeeps its static archives (libjq.a/libonig.a)._posixshmemnow).Build-number bumps (republish with the fixes above): pymongo→2, onnxruntime→2, matplotlib→2,
llama-cpp-python→2, opencv-python→2, ncnn→2, faiss-cpu→2, pyarrow→2, pyobjus→2, opaque→2,
pycryptodome→2, pycryptodomex→2, python-magic→2, scikit-learn→2, jq→2, coolprop→11, pysodium→11,
selectolax→11, flet-libjq→11.
Minor (no wheel change): astropy/spacy/thinc gain
extract_packages(data via__file__);flet-lib{crc32c,freetype,jpeg,png,pyjni,yaml,geos} version jinja-templating.
CI / recipe-tester (
.github/workflows/*,tests/recipe-tester/*,setup.sh)publishinput — publish to pypi.flet.dev viaworkflow_dispatch(publish: true), inaddition to the automatic push-to-main path. Lets a deploy dispatch publish with a pinned
python_build_run_id+serious_python_ref.serious_python_ref+python_build_run_idinputs — validate recipes against anunreleased serious_python branch / a specific python-build run before release; the recipe-tester
template appends the sp git-override when set.
excluded_archespip-resolution failures (e.g. pyarrow/onnxruntime dropping armeabi-v7a).stage_recipe.shderives[tool.flet.android] target_archfrom the built wheels and wiresextract_packages+test.requires.setup.shpins python-build release20260714(includesdc76612's_pyreplprune +mimalloc seccomp
open()fix — required for genuine 3.13/3.14 on-device tests).Validation
pyzmq, selectolax, xxhash, llama-cpp-python, jq, opaque, ncnn (+opencv-python), python-magic, …
SIGSYS" timeouts and py3.14 iOS_pyreplfailures wereflet/python-build infra bugs, fixed by python-build
dc76612(Fix Python 3.13/3.14 on-device crashes: mimalloc seccomp open() + pruned _pyrepl python-build#29) — not recipe bugs.main.Consumer notes
The following packages now build and run on-device under Flet 0.86 (Android + iOS), published to
pypi.flet.dev. Add them under[tool.flet.dependencies](or your app's deps) as usual.savefig(..., format="png")verified on device. (iOSpreviously crashed at launch — fixed.)
.sorefcollision).InferenceSessionloads and runs (the pybind module now imports on device).import cv2works on Android and iOS; heavy.so, expect a larger app.magicDB: the compiledmagic.mgcships in the wheeland is loaded from memory; no
[tool.flet]data config needed.flet-libjq/pysodiumrespectively (published alongside).core functionality verified on device.
Path-hungry packages (astropy, spacy, thinc, matplotlib) set
extract_packagesin their recipe, sotheir bundled data resolves on Android automatically — no app-side config required.