Add clickhouse-connect backend

[ShawnChen-Sirius]

This PR makes chDB usable as a clickhouse-connect execution backend, so existing clickhouse-connect code runs unchanged against the embedded chDB engine instead of an HTTP server. It is the chDB-side half of the design that follows up on ClickHouse/clickhouse-connect#753; the matching clickhouse-connect change is ClickHouse/clickhouse-connect#811 and is required for this PR to function (it adds the entry-point mechanism this PR registers against).

What this PR adds (6 commits, ~3,400 lines, all in this repo)

• chdb/cc_backend.py — ChdbClient is a thin clickhouse_connect.driver.Client subclass that overrides only the transport-shaped methods (raw_query / raw_stream / raw_insert / command / _query_with_context / data_insert) plus a true zero-copy Arrow fast path (query_arrow / query_arrow_stream / query_df ride chDB's in-process Arrow buffer through PyArrow's C Data Interface). AsyncChdbClient wraps the sync client via run_in_executor. ChdbBackend is the entry-point factory.
• chdb/cc_extension.py — the client.chdb namespace for chDB-only capabilities: query_python(...) (the Python() table function for in-process DataFrames), register_function(...) (Python UDFs), cursor() (chDB's native DB-API cursor), session_path. Absent on HTTP clients by design — accessing .chdb on an HTTP client raises AttributeError.
• pyproject.toml — registers chDB with clickhouse-connect via the clickhouse_connect.backends entry-point group, and adds clickhouse-connect as an optional dependency (clickhouse-connect>=1.3.0; will be bumped once the matching CC version is released).
• tests/clickhouse_connect/test_cc_backend.py — 105 tests relocated from clickhouse-connect PR #753, adapted to backend=\"chdb\".
• tests/clickhouse_connect/test_parity.py — 38 chDB-vs-real-server output-parity assertions across primitives, dates, decimals, arrays, maps, tuples, enums, UUIDs, IPs, fixed strings, aggregates, group-by and datetime64.
• scripts/cc_upstream_suite/ — harness that runs clickhouse-connect's own integration suite against ChdbBackend by redirecting create_client at import time. The skip handling is entirely data-driven (no pytest.mark capability markers — kept clickhouse-connect's repo minimal): skip_list.txt lists nodeid substrings for cases the embedded engine genuinely cannot satisfy (HTTP transport / RBAC / external_data / Arrow-vs-Native dtype divergence / etc.).
• .github/workflows/clickhouse-connect-backend.yml — three CI gates against the chDB backend: the relocated #753 suite (embedded), parity against a real ClickHouse 26.5 server (service container), and clickhouse-connect's own integration suite over a curated-green file set.

A few real-world fixes the integration-suite run forced:

• Process-wide refcounted connection cache in cc_backend.py. chdb-core's embedded engine permits only one active engine instance per process per data directory; a second chdb.connect() on the same path while the first is open deadlocks in the C layer. The cache lets many ChdbClient instances share one underlying connection.
• weakref.finalize on every stream wrapper. Guarantees the per-connection lock is released even when a test errors out without calling close().
• Lazy USE on client.database = X. Over HTTP this is a per-request parameter and assigning a not-yet-created database is fine; chDB is session-scoped and needs USE, so we apply it lazily before the next operation, which preserves the common bootstrap of "create database, then use it".

Testing

Empirical, no pytest markers:

Gate	Result
Relocated backend suite (embedded)	105 passed
Parity vs real ClickHouse 26.5.3.41	38 passed
clickhouse-connect's own integration suite vs chDB (with skip_list.txt applied)	PASS=346, FAIL=0, ERROR=0, SKIP=181 across all 34 integration files

Of the 423 cases passing on the HTTP baseline, 320 (76%) pass byte-identically on chDB; the remaining 103 are categorized in skip_list.txt:

Category	Count	Reason
external_data unsupported	22	engine lacks this capability
HTTP session_id / aiohttp pool / form-encoding / protocol version	35	embedded has no HTTP concepts
transport attributes (headers / compression / port)	10	embedded has no socket
RBAC (GRANT / CREATE ROLE / row policies)	4	engine has no access-control layer
query_id / per-query summary headers	8	not surfaced in-process
DateTime / Decimal / Enum / Time / Time64 formatting	32	Arrow-vs-Native representation differs (values correct)
column-name post-processing / server-formatted output	12	HTTP-side options not applicable
streaming / Arrow batch shape	8	chDB emits whole stream as one batch
pre-existing HTTP-baseline failures (not chDB-specific)	34	also fail on a real server in this provisioning

README.md documents the version-subscription policy (personal fork branch → CC main → CC release) and the procedure to re-curate the skip list on each clickhouse-connect release.

User demo

# pip install 'clickhouse-connect[chdb]'
# (installs both; chdb self-registers via the `clickhouse_connect.backends` entry point)
import clickhouse_connect

# All downstream code is identical across backends.
client = clickhouse_connect.get_client(backend=\"chdb\", path=\":memory:\")

client.command(
    \"CREATE TABLE t (id UInt64, name String) ENGINE = MergeTree ORDER BY id\"
)

client.insert(\"t\", [[1, \"Alice\"]], column_names=[\"id\", \"name\"])

result = client.query(\"SELECT * FROM t\")
print(result.result_rows)

For chDB-only features:

client.chdb.query_python(\"SELECT sum(a) FROM Python(my_df)\", my_df=my_df)
client.chdb.register_function(my_udf)
cursor = client.chdb.cursor()

From the user's point of view it is still the normal clickhouse-connect API; the chDB backend is selected with one keyword and chDB-only capabilities are reachable through an explicit, opt-in namespace.

Release ordering

This PR depends on clickhouse-connect/#811 being merged and released. Once that's available, the clickhouse-connect>=1.3.0 pin in pyproject.toml will be bumped to the version that ships the registry.

[ShawnChen-Sirius]

@chibugai review it

[Copilot]

Pull request overview

Adds a new clickhouse-connect execution backend powered by embedded chDB, enabling existing clickhouse-connect client code to run unchanged without an HTTP server.

Changes:

• Introduces ChdbClient / AsyncChdbClient backend implementation plus a client.chdb extension namespace for chDB-only capabilities.
• Adds comprehensive backend and parity test suites plus an upstream-suite harness with data-driven skip/xfail lists.
• Registers the backend via Python entry points and adds an optional clickhouse-connect dependency; adds CI workflow gates.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
chdb/cc_backend.py	Implements the clickhouse-connect backend over embedded chDB, including streaming and Arrow fast paths.
chdb/cc_extension.py	Adds client.chdb namespace for chDB-only features (Python table function, UDFs, cursor, session path).
pyproject.toml	Adds optional dependency and registers the backend via clickhouse_connect.backends entry point.
tests/clickhouse_connect/conftest.py	Adds shared test fixtures/timezone pinning for deterministic results.
tests/clickhouse_connect/test_cc_backend.py	Adds relocated/adapted backend behavior tests (embedded-only).
tests/clickhouse_connect/test_parity.py	Adds parity assertions comparing embedded chDB backend vs real ClickHouse server outputs.
scripts/cc_upstream_suite/*	Adds harness to run clickhouse-connect’s upstream suite against chDB with skip/xfail data files.
.github/workflows/clickhouse-connect-backend.yml	Adds CI workflow running backend tests, parity tests, and upstream suite gate.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.