Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis)

[eduralph]

This is part of the answer to the Issue 9393, essentially using the "X framebuffer" as described in Option 1 of the Feature Request.

Introduce GitHub Actions CI workflow running inside a shared Docker image on ghcr.io, with a native Windows runner for cross-platform unit-test coverage. Add a shared pytest harness with Gramps-backed fixtures and repo-wide integration tests that verify every addon registers, loads, and exposes valid plugin metadata. I chose to fix lint and structural issues across 29 addons so the pipeline goes green - the changes made should not impact functionality but might require some testing.

CI infrastructure

• .github/docker/gramps-ci/Dockerfile — Python 3.12 + Gramps 6.0 (pip)
  • PyGObject + GTK typelibs + xvfb/xauth + ruff, pytest, dbf, intltool, gettext, git. GTK lives in the base so addon modules that do from gi.repository import Gtk at load time are importable; xvfb/xauth are bundled for tests that actually render.
• .github/workflows/docker-build.yml — rebuilds the image on .github/docker/** changes or via workflow_dispatch.
• .github/workflows/ci.yml — seven jobs: lint (ruff E9/F63/F7/F82 + trailing whitespace), addon-structure (every addon has po/template.pot), compile-check (py_compile on every .py), unit-test-linux (container), unit-test-windows (native, conda+pip), integration-test (container with --init so xvfb-run doesn't hang), build (make.py gramps60 build all).
• .github/environment.yml — hybrid conda+pip env for Windows. Gramps isn't on conda-forge, so pygobject/gtk3 come from conda and gramps/orjson/pytest/dbf come from pip.

Shared test harness

• tests/conftest.py — fixtures: gramps_user, gramps_plugin_manager, gramps_plugin_registry, gramps_db (fresh in-memory per test), gramps_db_session (shared).
• tests/test_plugin_registration.py — registers every addon, subprocess-isolates module loading (crash-safe), verifies gramps_target_version=6.0 and valid id/name/version, smoke-tests import/export entry functions.
• conftest.py (repo root) — registers the gui marker and auto-skips @pytest.mark.gui tests when GTK is unimportable.
• pytest.ini — declares the gui marker.

Lint fixes (79 → 0 ruff errors across 29 files)

• Missing imports added: WindowActiveError, ErrorDialog, DbTxn, EditDate, ReportError, EventType, Surname, display_help, reduce, time, sys, etc.
• 3× os.name is 'nt' → ==; tuple-in-if bug in LifeLineChartView.
• Py2 leftovers: except E(msg) → except E as msg, unicode() → str(), dropped basestring/reload branches.
• Real bugs: missing value param in libaccess lambda, wrong var in JSONImport LOG.warn branch, displayer.display → name_displayer.display in QuiltView, missing paren in QueryQuickview, stray parent=self.uistate.window at module level in lxmlGramplet.
• Dead-code cleanup: AttachSourceTool, SourceIndex/index.py, DynamicWeb/run_dynamicweb.py.
• Renamed SurnameMappingGramplet.grp.py → .gpr.py (typo — Gramps never loaded this file).

Addon structure

• Added po/template.pot stubs for AnniversariesGramplet, ArchiveAssist, GrampsChat, GrampyScript (had no po/ dir at all).

TMGimporter tests

• test_libtmg.py: 58 pure-logic tests (strip codes, date parsing, repo type / URL inference). Runs on Linux + Windows.
• test_integration.py: 117 DB-backed tests (E2E pipeline + 12 function-level classes). Linux-only — in-memory Gramps SQLite hangs on Windows under pip-Gramps + conda-forge GTK, so the Windows job skips it via --ignore-glob='**/test_integration*.py'.

Misc

• .gitignore: debug.log (was accidentally committed once); CLAUDE.md (fork-local AI guidance file).

[GaryGriffin]

Added Note to https://gramps-project.org/bugs/view.php?id=9393 requesting testing and feedback of this PR.

[eduralph]

Corrected to conform to the agents.md guidelines, including using unittest instead of pytest

[GaryGriffin]

Can you provide the commands to test the CI workflow. And a usage of the test harness.

[kulath]

How did the code work without the imports you have added

[kulath]

As per comment on another PR, the CLAUDE.md file should not be here as it just duplicates Agents.md in the main Gramps repository meaning any changes etc. will have to be made in two place and it will be hard to keep the files in step.

[kulath]

Thanks for fixing so many bugs and problems, but there are too many completely different changes in a single commit. Should they not be made in several commits(or even separate PRs).

[eduralph]

Yeah, I wanted more at one go, I can see that it might not be the right approach.

@kulath , there are many Lint issues and I chose to fix them them instead of dropping the Lint. I thought this might get some pushback. We can do it the other way around - I'll remove the Linting and we can make issues out of them as preperation for activating them.

[eduralph]

I've significantly reduced the scope of this PR based on @kulath's feedback about "too many completely different changes." It's now strictly CI infrastructure — 7 files, one commit.

What's in the PR

• .github/docker/gramps-ci/Dockerfile — container image for Linux CI jobs
• .github/environment.yml — conda environment for Windows CI
• .github/workflows/ci.yml — the CI workflow itself (lint, structure, compile, unit tests on Linux+Windows, integration, build)
• .github/workflows/docker-build.yml — publishes the CI image to GHCR
• tests/init.py
• tests/gramps_test_env.py — shared GrampsTestCase / GrampsDbTestCase base classes
• tests/test_plugin_registration.py — sanity checks that every registered plugin imports

Everything else from the earlier revision (lint fixes across ~29 addons, po/template.pot stubs for the four addons missing them, TMGimporter tests, the SurnameMappingGramplet.grp.py → .gpr.py rename, CLAUDE.md) has been removed and will be submitted as separate PRs.

Gate Policy

All seven jobs run on every push and PR. To avoid showering innocent contributors with red CI for pre-existing tree state, four jobs are marked continue-on-error: true so they surface issues without blocking merges:

lint — non-blocking (~79 pre-existing ruff E9/F63/F7/F82 errors)
addon-structure — non-blocking (4 addons missing po/template.pot)
unit-test-linux — non-blocking (some addon test modules fail to import today)
unit-test-windows — non-blocking (same)
compile-check — blocking (green today)
integration-test — blocking (green today)
build — blocking (green today)

In order to activate all gates, there will be some rework of existing addons needed

[eduralph]

@GaryGriffin — here are the commands for each CI job, lifted straight from ci.yml. Run from a checkout of addons-source with Gramps importable (either PYTHONPATH / GRAMPS_RESOURCES pointing at a Gramps checkout, or inside the CI container image).

Lint — syntax/import errors plus trailing whitespace

ruff check --select=E9,F63,F7,F82 --no-fix --exclude='.gpr.py' .
git --no-pager grep -n --full-name '[ \t]$' -- '.py' && echo FAIL || echo OK

Addon structure — every addon must have po/template.pot

for gpr in /.gpr.py; do
d="$(dirname "$gpr")"
[ -f "$d/po/template.pot" ] || echo "MISSING: $d"
done

Compile check — py_compile on every .py

find . -name '.py' ! -name '.gpr.py' ! -path './.git/' ! -path '/pycache/*'
-exec python3 -m py_compile {} +

Per-addon unit tests — matches both the Linux and Windows jobs

export PYTHONPATH=.
modules=""
for f in /tests/test_.py; do
[ -f "$f" ] || continue
case "$(basename "$f")" in test_integration*) continue ;; esac
case "$f" in Sqlite/tests/test_sqlite.py) continue ;; esac
mod="${f%.py}"; modules="$modules ${mod////.}"
done
python3 -m unittest -v $modules

Integration tests — plugin registration plus per-addon integration

export PYTHONPATH=.
python3 -m unittest discover -s tests -p 'test_*.py' -t . -v

per-addon integration uses the same loop as the unit block above
but matching test_integration.py instead*

[GaryGriffin]

here are the commands for each CI job, lifted straight from ci.yml.

Sorry, what I meant was, is there a way to invoke the CI process (not the commands that are invoked by the CI process) manually to test the github actions.

I did try the invoked commands:

Lint: I dont have ruff on my Mac. How would I invoke the CI lint process within github as an action.

Addon Structure: if I manually invoke the commands on my Mac that is in the comment, it fails (due to escaping special chars needed for comments). I need to use:

for gpr in */*.gpr.py; do
d="$(dirname "$gpr")"
[ -f "$d/po/template.pot" ] || echo "MISSING: $d"
done

Results: 4 issues, as you stated. I do not know if it is a requirement that the template.pot exists. If there are no translated strings, then there shouldn't need to be one, I think. Maybe I am wrong.

Compile Check: same issue with escaping special chars. I think the gpr.py should actually be included. I need to use:

find . -name *.py -exec python3 -m py_compile {} \;

Results: 4 issues - Themes, Query, HouseTimelineGramplet, and lxml

Integration Tests: failed with error ImportError: Start directory is not importable: 'tests' .

[eduralph]

Can you tell me what you are trying to do or understand? I'm not sure if I'm giving you the right answers.

The tests run in containers stored on ghcr.io containing all the necessary tools. It's optimized for steady-state but is kind of a pain for this specific PR, because you can't really test it in this environment. The upside is that the image is stable across hundreds of normal PRs. It only churns when someone explicitly wants the CI environment to change.

• The Dockerfile changes rarely. Maybe once per Gramps minor, or when a system dependency is added. Most weeks, nobody touches .github/docker/**.
• The path filter ensures docker-build doesn't fire on every push — only when the Dockerfile actually changes. So in steady state, ci.yml pulls a stable, cached image and runs fast.
• The "default branch only" scope ensures one canonical image per supported branch (:gramps60). PR-built images would proliferate (:gramps60-pr123, :gramps60-pr124...) and clutter GHCR with throwaway tags that need lifecycle management.
• GHCR write access is gated to default-branch context. Fork PRs have read-only GITHUB_TOKEN, so a pull_request: trigger on docker-build would silently no-op for external contributors anyway. Even internal PRs publishing to GHCR would mean unmerged code can write to the registry — a small but real trust-boundary violation.

You will have to make the images publically available for the PRs to work, but I don't think that's an issue. There will be some maintainance work needed to be done once you flip over to branch61 as default, I'd see what can be done to make it a bit easier then, assuming you like the current process.

In order for you to verify how it works, take a poke at my forked repo where I've merged the PR. This is how you can do it

1. Fork eduralph/addons-source to your account using GitHub's "Fork" button: https://github.com/eduralph/addons-source/fork.
2. Create a branch off maintenance/gramps60 on your new fork — e.g., test/ci-tryout. The CI workflow files come along automatically.
3. Make any change you want to exercise the pipeline, or none at all if you just want to see a clean run. Some ideas:
   • Add a deliberate import nonexistent_module to any .py file → trips the Lint job.
   • Add a stray syntax error → trips the Compile Check job..
   • Modify a real addon you care about → exercises the Build job and unit/integration tests.
4. Commit, push, and open a PR from your branch targeting eduralph/addons-source:maintenance/gramps60. (Not gramps-project/addons-source — point the PR base at my fork.)
5. Wait for approve the first workflow run. Because you'll be a first-time contributor to my fork, GitHub will hold the run for manual approval — click the "Approve and run workflows" button on the PR. Subsequent runs go automatically.
6. Watch the Checks tab on your PR. You'll see 7 jobs fire:
   • 5 Linux jobs (Lint, Addon Structure, Compile Check, Unit Tests Linux, Integration Tests, Build) running in the public container.
   • 1 Windows job (Unit Tests Windows) on a GitHub-hosted Windows runner with a conda environment.
   • The advisory ones (Lint, Addon Structure, Unit Tests Linux/Windows) will report red without blocking your PR; the blocking ones (Compile Check, Integration Tests, Build) gate the merge.

[GaryGriffin]

Can you tell me what you are trying to do or understand? I'm not sure if I'm giving you the right answers.

I should have been more explicit. I am trying to test the implementation before merging/publishing this PR. Understanding the resources needed and the frequency of activity is part of that. And making sure that it functions completely and as expected.

I was hoping for something like a github command to activate the action so I can see it work. Given your complete description (much appreciated), I think I am going to have to yield to @Nick-Hall to review/merge/publish this one since it impacts the repo actions in ways far above my knowledge of github. For instance, I dont know if these are the right blocking/non-blocking decisions. Or if the scope should just be the Addon impacted by the PR rather than all Addons.

[eduralph]

@GaryGriffin - I thought of adding a manual button for everything, but that only works if it's merged with the default branch, lol

[eduralph]

I just reworked the CI approach a bit to be able to differentiate between Base, Linux & Windows test specifically. Also added automated dependency loading for future unit tests. The Unit Tests currently fail because of an issue in Websearch, but that needs to be adressed seperately.