diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml index cad0f7b2..98a63415 100644 --- a/.github/workflows/examples.yml +++ b/.github/workflows/examples.yml @@ -8,7 +8,7 @@ on: schedule: - cron: '0 0 1 * *' # Runs at 00:00 UTC on the 1st day of every month -# One live run per branch — the README examples run real queries against the +# One live run per branch — the example queries run against the # production VFB backend, so superseded runs are cancelled rather than left to # pile up alongside the other live-backend workflows. concurrency: @@ -49,21 +49,10 @@ jobs: f.write('SOLR_AVAILABLE=false\n') exit(1) " - - name: Run examples from README.md + - name: Run the example queries against the live backend run: | - cat README.md | grep -e '```python' -e '```' -e '^[^`]*$' | sed -e '/^```python/,/^```/!d' -e '/^```/d' -e 's/\(vfb\.[^(]*([^)]*)\)/print(\1)/g' > test_examples.py - cat test_examples.py export VFBQUERY_CACHE_ENABLED=false - python test_examples.py - - name: Parse README.md and generate test files - run: | - python -m src.test.readme_parser - env: - PYTHONPATH: ${{ github.workspace }} - if: env.SOLR_AVAILABLE == 'true' - - name: Run examples from README.md and validate structure - run: | - python -m src.test.test_examples_diff + python -m pytest -v src/test/test_example_queries.py env: PYTHONPATH: ${{ github.workspace }} if: env.SOLR_AVAILABLE == 'true' diff --git a/.github/workflows/python-test.yml b/.github/workflows/python-test.yml index 5413d8d5..2efa2b90 100644 --- a/.github/workflows/python-test.yml +++ b/.github/workflows/python-test.yml @@ -66,10 +66,10 @@ jobs: # step turns any skips into a PR-visible warning. # Excludes: test_query_performance.py — wall-clock threshold # assertions that flap under parallel load, already gated by the - # dedicated "Performance Test" workflow; and test_examples_diff.py / - # test_examples_code.py — README-example scripts (no pytest tests, - # pull in deepdiff/colorama) run by the "Test VFBquery examples" - # workflow instead. + # dedicated "Performance Test" workflow; and + # test_example_queries.py — the canonical worked examples, run + # by the "Test VFBquery examples" workflow instead so this job + # does not double the live load they put on production. # -n 4 rather than -n auto: an explicit cap on how many concurrent # query streams one run points at production. `auto` happens to be 4 # on today's GitHub-hosted ubuntu runner, so this is not a slowdown — @@ -77,8 +77,7 @@ jobs: # runner spec grows. pytest -v -ra -n 4 --dist loadscope \ --ignore=src/test/test_query_performance.py \ - --ignore=src/test/test_examples_diff.py \ - --ignore=src/test/test_examples_code.py \ + --ignore=src/test/test_example_queries.py \ src/test tests 2>&1 | tee pytest_output.log - name: Flag skipped tests (backend unavailable) if: always() diff --git a/README.md b/README.md index 7b361a02..ed294283 100644 --- a/README.md +++ b/README.md @@ -1,2858 +1,114 @@ # VFBquery -to setup requirements: -```bash -pip install --upgrade vfbquery -``` - -📖 **Full documentation: [vfbquery.readthedocs.io](https://vfbquery.readthedocs.io/en/stable/)** — the -HTTP API, the `/combine` set-algebra reference, the lightweight `vfbquery-client` package, and the -query catalogue, all rendered and cross-linked. This README is the quick start. +[![PyPI](https://img.shields.io/pypi/v/vfbquery)](https://pypi.org/project/vfbquery/) +[![Documentation](https://readthedocs.org/projects/vfbquery/badge/?version=stable)](https://vfbquery.readthedocs.io/en/stable/) -🧪 **Adding or changing a test?** Read **[TESTING.md](TESTING.md)** first. The suite runs live -queries against the VFB backend, so tests must assert real content (never suppress an empty result, -never swallow errors, always verify fixtures return data). Those rules exist because a batch of -silently-passing tests was found and fixed — the doc is how we keep them fixed. +VFBquery is the query layer of [Virtual Fly Brain](https://virtualflybrain.org) +(VFB) — the *Drosophila* nervous-system atlas. It turns the identifiers VFB +users work with (`FBbt_...` anatomy classes, `VFB_...` individuals) into the +rich, cross-referenced reports the VFB website shows: term metadata, aligned +images, connectivity, transcriptomics, similar neurons, stocks and +publications. It is both a Python package and the HTTP backend that serves +those results to the VFB site and to third-party tools. -## 🚀 Performance & Caching +## How it works -VFBquery includes intelligent SOLR-based caching for optimal performance: +VFBquery sits in front of the VFB data stores and does the fan-out for you: -- **Instant Results** for repeated queries -- **NBLAST similarity queries**: 10+ seconds → <0.1 seconds (cached) -- **Zero configuration** - works automatically -- **Persistent cache** survives restarts -- **3-month TTL** matches VFB_connect behavior - -```python -import vfbquery as vfb +- **Solr** holds pre-built `term_info` documents and powers free-text search — + most lookups are answered from here in milliseconds. +- **Neo4j** (the VFB knowledge graph) answers the relational questions: + connectivity, cross-references, images by template, dataset membership. +- **Owlery** (an OWL reasoner) answers the ontology queries that need + reasoning rather than lookup. +- A **Solr-backed result cache** stores computed results with a three-month + TTL and version-based invalidation, so repeated queries are effectively + instant — see [CACHING.md](CACHING.md). -# First query builds cache (~1-2 seconds) -result1 = vfb.get_term_info('FBbt_00003748') +The same functions are exposed two ways: imported as a Python package, or +served over HTTP by the bundled high-availability server (`vfbquery.ha_api`), +which adds request coalescing, queueing, backpressure and its own short-lived +result cache. The VFB website's term-information panels are drawn from this +service. -# Subsequent queries served from cache (<0.1 seconds) -result2 = vfb.get_term_info('FBbt_00003748') # 54,000x faster! +## Install -# Similarity queries also cached -similar = vfb.get_similar_neurons('VFB_jrchk00s') # Fast after first run ``` - -📚 See [CACHING.md](CACHING.md) for cache configuration, the `VFBQUERY_CACHE_ENABLED` -bypass (used by the tests), and version-based invalidation; and -[RELEASING.md](RELEASING.md) for how the single-source version (`_version.py`) is -bumped from the release tag. - -## 🕸️ CATMAID pass-through - -VFB hosts public, read-only CATMAID servers for several connectomics datasets -(FAFB, FANC, L1EM, ...; registry at -). `vfbquery` exposes their -query API directly, and anywhere a command takes skeleton ids you can pass -CATMAID skids, VFB ids (`VFB_xxxxxxxx`) or a mixed list — VFB ids are converted -through the knowledge graph's cross-references before the request is made. - -```python -import vfbquery as vfb -from vfbquery import catmaid - -vfb.list_catmaid_instances() # hosted instances + tokens + projects -vfb.list_catmaid_commands() # the curated read-only command registry - -fafb = catmaid('fafb') # optionally catmaid('fanc', project=2) -fafb.commands() # {command: doc} - -# Mixed VFB ids and skids; the envelope carries the id mapping both ways: -fafb.connectivity(ids=['VFB_001011rj', 10603863], boolean_op='OR') -fafb.neuron_names(ids=['VFB_001011rj']) -fafb.swc(id='VFB_001011rj') # single-id commands take id= -fafb.swc_alignments(id='VFB_001011rj') # spaces an SWC is available in -fafb.swc(id='VFB_001011rj', aligned='JRC2018Unisex') # VFB's template-registered copy -# (aligned=