feat(build): generate per-exchange Python packages from python/ccxt

[szemyd]

TL;DR

import ccxt runs one __init__.py that imports all 104 exchanges. This PR adds a generator that mechanically splits python/ccxt into one PyPI distribution per exchange plus a shared core, a verifier that proves each package imports in isolation and behaves identically to upstream, and CI that runs both. No exchange code is hand-edited and the monolithic package keeps shipping unchanged.

                    python/ccxt  (unchanged, still ships as `ccxt`)
                         │
          python/split/split_packages.py   ← reads the tree, rewrites imports
                         │
   ┌─────────────────────┼─────────────────────────────┐
   ▼                     ▼                             ▼
 ccxt-core        ccxt-binance, ccxt-okx, …    ccxt-core-starknet,
 → ccxt_core      → ccxt_binance, …            ccxt-core-msgpack, …
 base/ errors/    one exchange each, in the    the vendored third-party
 Precise, ws      flavours upstream ships it   trees ccxt-core imports
 client.          (sync / async_support /      lazily — 1.7 MB that most
 No exchanges,     pro / prediction)           exchanges never touch
 no vendored code
   ▲                     │                             ▲
   └── depended on by ───┴──── depends on what its ────┘
                               call graph reaches

        ccxt-binanceus ──depends on──▶ ccxt-binance ──▶ ccxt-core
        (subclasses it, so it declares a dependency
         instead of vendoring a second copy)

        ccxt-paradex ──depends on──▶ ccxt-core-starknet ──▶ ccxt-core-keccak
        (calls starknet_sign, so it gets the 768 KB bundle;
         the other 82 exchanges get none of it)

import ccxt_binance

exchange = ccxt_binance.binance()
exchange.fetch_ticker('BTC/USDT')

116 distributions generated, all 108 exchange packages verified byte-identical in behaviour to upstream, every wheel/sdist passes twine check.

---

Why

	import ccxt	import ccxt_bit2c
Python modules loaded	603	359
Resident memory	67 MB	38 MB
Import time	0.226 s	0.145 s
Installed source	36 MB	794 KB (core + exchange)
Lines of exchange code on disk	751 047	44 011

Across all 108 exchanges the median single-exchange install is 1023 KB, down from 36 MB for the monolith and from 2167 KB before the vendored trees were split out.

import ccxt.pro goes from 888 modules / 111 MB to 504 / 74 MB for ccxt_binance.pro. The wins that matter most are the ones that don't fit in a table: an audit of a trading service no longer has to cover 103 exchange modules it never calls, and a CVE in one exchange's vendored dependency stops being everyone's problem.

What is in the PR

File
python/split/split_packages.py	the generator
python/split/verify_packages.py	isolation + upstream-equivalence checks
python/split/tests/test_split_packages.py	27 tests (subset by default, full sweep under CCXT_SPLIT_FULL=1)
python/split/README.md	how it works, how to run it, how to publish
.github/workflows/python-split-packages.yml	generate → build → verify → install-a-wheel-and-use-it
.gitignore, CLAUDE.md	ignore python/split-dist/, one line in the repo map
.github/workflows/{js,python,php,cs,go-app,java}.yml	run the six build jobs on ubuntu-latest instead of self-hosted runners this fork does not have

Nothing under ts/src/, python/ccxt/, or any other generated tree is touched.

How the split works

Every input is read out of the source tree, so the same command works on the next release with no edits:

1. Discovery — the exchange list for each flavour comes from that flavour's __init__.py exchanges list; the version from __version__.
2. Rewriting — ccxt.… module paths are rewritten over the token stream, not with a text regex. That is what keeps the ~6000 ccxt.com URLs in docstrings and describe() blocks untouched while still rewriting class binanceus(ccxt.async_support.binanceus). ccxt.base.*, ccxt.static_dependencies.*, ccxt.protobuf.*, ccxt.async_support.base.* → ccxt_core; ccxt.<id> and its flavour/abstract variants → ccxt_<id>.
3. Entry points — each generated __init__.py is the upstream one with the per-exchange import lines it does not own removed and exchanges shrunk. Licence header, __version__, error re-exports and __all__ carry over verbatim.
4. Inheritance — the 11 exchanges that subclass another (binanceus→binance, bequant→hitbtc, kucoinfutures→kucoin, …) depend on the parent distribution rather than vendoring it, discovered from the imports rather than a hard-coded table. isinstance and except therefore still work across packages, and because the error hierarchy lives in ccxt_core, except ccxt_core.NetworkError catches failures from every installed exchange.
5. Vendored code — see below.
6. Metadata — licence, authors, classifiers and requires-python are inherited from the root pyproject.toml; ccxt-core inherits the pinned runtime dependencies; exchange packages depend only on ccxt-core plus any parent and vendored bundle.

$ python python/split/split_packages.py --out python/split-dist
generated 116 packages into python/split-dist          # 26s

Useful flags: --only binance,okx (parents pulled in automatically), --build (wheels + sdists), --dist-prefix/--module-prefix if the PyPI names need to differ, --vendored core to put the vendored trees back inside ccxt-core, --source to split a different checkout.

The vendored trees

static_dependencies and protobuf are 1.7 MB of third-party code that base/exchange.py imports from inside the few methods that need it. In ccxt-core it landed on every install, and 82 of the 108 exchanges never reach any of it:

distribution	ships	reached by
ccxt-core-starknet	starknet, starkware, marshmallow, marshmallow_oneofschema, lark	2 exchanges
ccxt-core-dydx-v4-client	dydx_v4_client	1
ccxt-core-msgpack	msgpack	1
ccxt-core-protobuf	protobuf/mexc	1
ccxt-core-keccak	keccak	25
ccxt-core-ethabi	ethabi	12
ccxt-core-lighter-client	lighter_client	1

The grouping is derived, not declared. Each vendored directory gets a signature — the set of base/exchange.py entry points that reach it through the vendored import graph — and directories sharing a signature ship together. That is why lark (428 KB, and used only to parse Cairo ABIs) travels with starknet, while keccak, reachable on its own as well as through ethabi and starknet, stays separate with both depending on it. Which exchange needs what comes from the base-method call graph, closed over self.… calls. It over-approximates where unsure — anything calling hash() gets ccxt-core-keccak (7 KB) whether or not it passes 'keccak' — because an extra dependency is harmless and a missing one is not.

Cross-bundle relative imports inside the vendored code are rewritten to absolute ones: starknet/hash/utils.py said from ... import keccak and now says from ccxt_core_keccak import keccak.

Result:

install	before	after
ccxt-core alone	1849 KB	695 KB
ccxt-bit2c	1951 KB	794 KB
ccxt-kraken	2248 KB	1102 KB
ccxt-binance	3584 KB	2427 KB
ccxt-paradex (needs starknet)	2180 KB	1842 KB
median across all 108	2167 KB	1023 KB

Verification

verify_packages.py imports each package in a fresh interpreter, instantiates the exchange in every flavour it ships, and asserts the only ccxt_* modules left in sys.modules are the package itself, ccxt_core, and the parents it declares. A leak shows up as an extra module name rather than as an import time nobody measures.

$ python python/split/verify_packages.py --out python/split-dist --compare-upstream
verified 108 exchange packages

--compare-upstream additionally loads the monolithic package alongside and asserts that describe(), the full attribute surface and the MRO of every split class match upstream exactly. That is the strongest claim here: for all 108 exchanges across all their flavours, the split class is the upstream class.

Because the vendored imports are lazy, importing a package proves nothing about them — a wrong dependency would stay invisible until a user signed an order. So the verifier executes them: for each exchange it collects the vendored import statements from the base methods that package actually calls, and runs them in a child interpreter that can see only that package's declared dependencies. Deleting ccxt-core-starknet from ccxt-paradex's dependencies makes it fail, as it should:

ccxt-paradex: vendored code it calls is not installable:
  from ccxt_core_starknet.starknet.hash.utils import message_signature -> No module named 'ccxt_core_starknet'

Ran locally on this branch:

• pytest python/split/tests — 24 passed (rewriter unit tests + subset trees)
• CCXT_SPLIT_FULL=1 pytest python/split/tests — 27 passed, full sweep over every exchange
• split_packages.py --build — 116 distributions, 232 artifacts
• twine check — all PASSED
• wheel install into a clean venv → ccxt_binanceus.binanceus() works, loads exactly ['ccxt_binance', 'ccxt_binanceus', 'ccxt_core']
• wheel install of ccxt-paradex → retrieve_stark_account(...) derives a real Stark key from the lazily-imported vendored starknet code
• ruff check python/split/ (repo config) — clean

The subset used by the fast tests is chosen for coverage rather than convenience: binance (largest, all three flavours), binanceus (cross-package inheritance), hyperliquid (the only id present in all four flavours), kalshi (prediction-only, no sync class), bit2c (the ordinary case), paradex (heaviest vendored bundle).

The §6.5 checklist in CLAUDE.md does not apply — this PR contains no ts/src/ change and so nothing to transpile.

Publishing

Build output is gitignored; this PR does not publish anything and adds no release automation.

$ python python/split/split_packages.py --out python/split-dist --build
$ twine upload python/split-dist/dist/* -u __token__ -p "$PYPI_TOKEN"

ccxt-core and the ccxt-core-* bundles have to land on the index first, since every exchange package pins them by exact version.

Notes / decisions worth a second opinion

• Names. Defaults are ccxt-core / ccxt-<id> / ccxt-core-<lib> on PyPI. Several may already be taken — --dist-prefix/--module-prefix exist precisely so this is a flag, not a rewrite.
• The vendored mapping is static analysis, so a base method reached by a name the analysis cannot see would produce a ModuleNotFoundError at call time. Three things bound that risk: the analysis over-approximates, the verifier executes every lazy import against the declared dependencies only, and ccxt-core exposes each bundle as an extra (pip install ccxt-core[starknet], or [all]) with --vendored core as a full escape hatch.
• google.protobuf is missing from ccxt's dependencies upstream, so mexc's protobuf decoding and dydx's transaction encoding fail on a plain pip install ccxt today. The verifier reports it separately rather than failing; fixing it is out of scope for this PR, but ccxt-core-protobuf and ccxt-core-dydx-v4-client would be the natural place to declare it.
• Shared core rather than 109 self-contained packages. Vendoring base/ into every package would make each one standalone, but two installed exchanges would then have two unrelated NetworkError classes and except would silently stop working. The shared core is what keeps multi-exchange installs coherent.
• python/ccxt/test is deliberately not packaged — the upstream harness imports the whole library by design.
• The six language workflows now run on ubuntu-latest (they targeted self-hosted runners this fork does not have, so every check sat queued). All six build jobs pass. Their live-tests jobs fail against real venues from a GitHub-hosted IP — e.g. bitrue WS: "44 succeeded, 1 failed" — which is unrelated to this change and left alone rather than skipped.

🤖 Generated with Claude Code