diff --git a/docs/_static/img/ci-artifacts.png b/docs/_static/img/ci-artifacts.png new file mode 100644 index 0000000000..bdc96a1488 Binary files /dev/null and b/docs/_static/img/ci-artifacts.png differ diff --git a/docs/_static/img/ci-workflow.png b/docs/_static/img/ci-workflow.png new file mode 100644 index 0000000000..7dff67ab45 Binary files /dev/null and b/docs/_static/img/ci-workflow.png differ diff --git a/docs/_static/img/ci_plot-view_attachment-tab.png b/docs/_static/img/ci_plot-view_attachment-tab.png deleted file mode 100644 index 59354dc98d..0000000000 Binary files a/docs/_static/img/ci_plot-view_attachment-tab.png and /dev/null differ diff --git a/docs/_static/img/ci_plot-view_select-test.png b/docs/_static/img/ci_plot-view_select-test.png deleted file mode 100644 index f243bcdacd..0000000000 Binary files a/docs/_static/img/ci_plot-view_select-test.png and /dev/null differ diff --git a/docs/_static/img/ci_plot-view_tests-tab.png b/docs/_static/img/ci_plot-view_tests-tab.png deleted file mode 100644 index cc6e3f23af..0000000000 Binary files a/docs/_static/img/ci_plot-view_tests-tab.png and /dev/null differ diff --git a/docs/dev/ci.md b/docs/dev/ci.md index 013d5d1836..b90a8077f9 100644 --- a/docs/dev/ci.md +++ b/docs/dev/ci.md @@ -5,28 +5,39 @@ A frequent frustration in testing is the reproducibility of the plots and `matplotlib`'s behaviour in different environments. We have some tooling to help with this. -### Viewing plots from failed tests on Azure pipelines +(viewing-plots-from-failed-tests)= -The fixtures `check_same_image` and `plot_cmp` upload plots from failing tests so you can view them from the azure pipelines test viewer. -To find these, navigate to the tests tab for your build +### Viewing plots from failed tests -```{image} ../_static/img/ci_plot-view_tests-tab.png -:width: 750px +When a plot comparison fails, the `check_same_image` and `plot_cmp` fixtures copy the expected, actual, and diff images into pytest’s cache directory (`.pytest_cache/d/debug`). +Locally, the assertion message links to the originals directly, so you can just click the paths in the test output. + +On CI, that directory is uploaded as an artifact named `debug-data-{environment}` (one per test environment in the matrix). +To get at it, open the {guilabel}`Checks` tab of your PR and select the {guilabel}`CI` workflow run: + +```{figure} ../_static/img/ci-workflow.png +:alt: A PR’s Checks tab, listing the CI workflow run. + +The {guilabel}`CI` workflow run in the {guilabel}`Checks` tab of a PR. ``` -Select your failing test +Then scroll to the {guilabel}`Artifacts` section at the bottom of the run summary and download the artifact for the environment whose test failed: + +```{figure} ../_static/img/ci-artifacts.png +:alt: The Artifacts section of a workflow run summary, listing debug-data artifacts. -```{image} ../_static/img/ci_plot-view_select-test.png -:width: 750px +The {guilabel}`Artifacts` section, with one `debug-data-{environment}` entry per test environment. ``` -And open the attachments tab +The downloaded archive mirrors the layout of the reference image directory, so a failing test shows up as -```{image} ../_static/img/ci_plot-view_attachment-tab.png -:width: 750px +```text +{matplotlib-version}/{test-name}/expected.png +{matplotlib-version}/{test-name}/actual.png +{matplotlib-version}/{test-name}/actual-failed-diff.png ``` -From here you can view and download the images which were compared, as well as a diff between them. +If the actual image is the correct one, you can copy it over the reference image (see {ref}`plotting-tests`). ### Misc diff --git a/docs/dev/code.md b/docs/dev/code.md index 3ca393c8f7..9485f790e0 100644 --- a/docs/dev/code.md +++ b/docs/dev/code.md @@ -19,6 +19,9 @@ To learn how to ignore checks per line please read about [ignoring errors][]. Additionally, we use Scanpy’s [EditorConfig][], so using an editor/IDE with support for both is helpful. +These checks run via our {ref}`commit hooks `, and in CI. +You can also apply them to the whole repo with `hatch check fmt --fix` (formatting) and `hatch check code --fix` (linting). + [Ruff]: https://docs.astral.sh/ruff/ [ignoring errors]: https://docs.astral.sh/ruff/tutorial/#ignoring-errors [EditorConfig]: https://github.com/scverse/scanpy/blob/main/.editorconfig diff --git a/docs/dev/getting-set-up.md b/docs/dev/getting-set-up.md index 20c6cba63a..acf5db9ad6 100644 --- a/docs/dev/getting-set-up.md +++ b/docs/dev/getting-set-up.md @@ -21,16 +21,16 @@ To get the code, and be able to push changes back to the main project, you'll ne This is very straight forward if you're using [GitHub's CLI][]: ```console -$ gh repo fork scverse/scanpy --clone --remote +$ gh repo fork scverse/scanpy --clone --remote -- --filter=blob:none ``` This will fork the repo to your github account, create a clone of the repo on your current machine, add our repository as a remote, and set the `main` development branch to track our repository. -To do this manually, first make a fork of the repository by clicking the "fork" button on our main github package. Then, on your machine, run: +To do this manually, first make a fork of the repository by clicking the {guilabel}`Fork` button on our main github page. Then, on your machine, run: ```console $ # Clone your fork of the repository (substitute in your username) -$ git clone https://github.com/{your-username}/scanpy.git +$ git clone --filter=blob:none https://github.com/{your-username}/scanpy.git $ # Enter the cloned repository $ cd scanpy $ # Add our repository as a remote @@ -40,28 +40,31 @@ $ # git branch --set-upstream-to "upstream/main" [GitHub's CLI]: https://cli.github.com -### `pre-commit` +(prek)= +(pre-commit)= -We use [pre-commit][] to run some styling checks in an automated way. +### Commit hooks + +We use [prek][] to run some styling checks in an automated way. We also test against these checks, so make sure you follow them! -You can install pre-commit with: +You can install `prek` with: ```console -$ pip install pre-commit +$ uv tool install prek ``` You can then install it to run while developing here with: ```console -$ pre-commit install +$ prek install ``` From the root of the repo. -If you choose not to run the hooks on each commit, you can run them manually with `pre-commit run --files={your files}`. +If you choose not to run the hooks on each commit, you can run them manually with `prek run --files={your files}`. -[pre-commit]: https://pre-commit.com +[prek]: https://prek.j178.dev/ (creating-a-branch)= @@ -90,7 +93,7 @@ $ # After that, just use $ git push ``` -And open a pull request by going to the main repo and clicking *New pull request*. +And open a pull request by going to the main repo and clicking {guilabel}`New pull request`. GitHub is also pretty good about prompting you to open PRs for recently pushed branches. We'll try and get back to you soon! diff --git a/docs/dev/release.md b/docs/dev/release.md index f93b73eaa1..44515ce6bf 100644 --- a/docs/dev/release.md +++ b/docs/dev/release.md @@ -6,18 +6,18 @@ That page also explains concepts like *pre-releases* and applications thereof. ## Preparing the release 1. Switch to the `main` branch for a major/minor release and the respective release series branch for a *patch* release (e.g. `1.8.x` when releasing version 1.8.4). -2. Run `hatch towncrier:build` to generate a PR that creates a new release notes file. Wait for the PR to be auto-merged. +2. Run `hatch run towncrier:build` to generate a PR that creates a new release notes file. Wait for the PR to be auto-merged. 3. If it is a *patch* release, merge the backport PR (see {ref}`versioning-tooling`) into the `main` branch. ## Actually making the release 1. Go to GitHub’s [releases][] page. -2. Click the “Draft a new release” button. -3. Open the “Choose a tag” dropdown and type the version of the tag you want to release, such as `1.9.6`. -4. Select the dropdown entry “**+ Create new tag: 1.\.\** on publish”. -5. In the second dropdown “Target:”, select the base branch i.e. `main` for a minor/major release, +2. Click the {guilabel}`Draft a new release` button. +3. Open the {guilabel}`Choose a tag` dropdown and type the version of the tag you want to release, such as `1.9.6`. +4. Select the dropdown entry {guilabel}`+ Create new tag: 1.. on publish`. +5. In the second dropdown {guilabel}`Target:`, select the base branch i.e. `main` for a minor/major release, and e.g. `1.9.x` for our example patch release `1.9.6`. -6. If the version is a *pre-release* version, such as `1.7.0rc1` or `1.10.0a1`, tick the “Set as a pre-release” checkbox. +6. If the version is a *pre-release* version, such as `1.7.0rc1` or `1.10.0a1`, tick the {guilabel}`Set as a pre-release` checkbox. [releases]: https://github.com/scverse/scanpy/releases diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 11e7e6643b..2a7c2c29a6 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -31,10 +31,10 @@ If you haven't written tests before, Software Carpentry has an [in-depth testing We highly recommend using [Test-Driven Development][] when contributing code. This not only ensures you have tests written, it often makes implementation easier since you start out with a specification for your function. -Consider parameterizing your tests using the `pytest.mark.parameterize` and `pytest.fixture` decorators. +Consider parametrizing your tests using the `pytest.mark.parametrize` and `pytest.fixture` decorators. You can read more about [fixtures][] in pytest’s documentation, but we’d also recommend searching our test suite for existing usage. -[existing test suite]: https://github.com/scverse/scanpy/tree/main/scanpy/tests +[existing test suite]: https://github.com/scverse/scanpy/tree/main/tests [in-depth testing guide]: https://katyhuff.github.io/2016-07-11-scipy/testing/ [test-driven development]: https://en.wikipedia.org/wiki/Test-driven_development [fixtures]: https://docs.pytest.org/en/stable/fixture.html @@ -67,17 +67,21 @@ Some approaches to this include: [scoped test fixture]: https://docs.pytest.org/en/stable/fixture.html#sharing-test-data +(plotting-tests)= + ### Plotting tests While computational functions will return arrays and values, it can be harder to work with the output of plotting functions. To make this easier, we use the `plot_cmp` fixture for comparing plotting results (search the test suite for example usage). This is used to check that generated plots look the same as they did previously. -Reference images (the expected output) are stored as `expected.png` to relevant tests directory under `scanpy/tests/_images`. -When run, the test suite will generate `actual.png` files for each check. -These files are compared, and if the `actual` plot differs from the reference plot, a `diff` of the images is also generated. -Paths for all these files will be reported when a test fails, and images for failed plots can be viewed via the :doc:`CI interface `. +Reference images (the expected output) are stored as `expected.png` in a per-test directory under `tests/plotting/legacy/_images/{matplotlib-version}`, +where the version segment is the minor version of `matplotlib` the images were generated with (e.g. `3.10`). +When run, the test suite will generate an `actual.png` file next to the reference image for each check. +These files are compared, and if the `actual` plot differs from the reference plot, an `actual-failed-diff.png` is also generated. +Paths for all these files will be reported when a test fails, and images for failed plots on CI can be retrieved as described in {ref}`viewing-plots-from-failed-tests`. -A common gotcha here is that plots often change slightly on different machines/ OSs. +A common gotcha here is that plots often change slightly on different machines/ OSs and between `matplotlib` versions. `scanpy`'s test suite sets a number of environment variables to ensure as similar of plots as possible. -When adding new reference plots, the recommended workflow is to write the test as though an expected result already exists, run it once to generate the output, then move that output to the reference directory. +When adding new reference plots, the recommended workflow is to write the test as though an expected result already exists, +run it once to generate the `actual.png`, then rename that file to `expected.png` if it looks correct. diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index 748b3d2e2c..1980275a1c 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -44,4 +44,4 @@ given as a comment on the PR, or a label or milestone description: In our case, these commands are part of the milestone description, which causes the merge of a PR assigned to a milestone to trigger the bot. -[meseeks bot]: https://meeseeksbox.github.io +[meeseeks bot]: https://meeseeksbox.github.io