LAMMPS Test Status Website

Aggregates the results of the automated LAMMPS test runs into a static website and a rolling GitHub status issue.

How it works

• GitHub Actions workflows in lammps/lammps upload JUnit XML test results as artifacts for post-merge runs on the develop branch (regression tests: merged run.json + JUnit XML; unit tests: one junit-<config> artifact per platform/configuration; the example input check: the JUnit XML of the test harness as it is, see below).
• The full regression tests are no longer run in GitHub Actions but on a dedicated machine with a much more complete LAMMPS configuration, and published on download.lammps.org (tools/fetch_regression.py, see below). The same input decks are run in four configurations: serial (one MPI task), parallel (4 MPI tasks), openmp (2 MPI tasks with 2 OpenMP threads each, through the OPENMP package), and kokkos (the same through KOKKOS/OpenMP).
• The same machine runs the unit tests in its native GCC build of x86_64 Linux and publishes them as a JUnit XML file next to the coverage report (tools/fetch_unittest.py, see below). That build has a far more complete package selection than the GitHub Actions runners compile, so it covers several hundred tests more than any of the configurations ingested from there.
• The update workflow in this repository ingests new artifacts (tools/ingest_actions.py) and the latest published regression and unit test results (tools/fetch_regression.py, tools/fetch_unittest.py), archives one run.json per run under data/<suite>/<runid>/, rebuilds the website (generator/build_site.py), deploys it to GitHub Pages, and updates the rolling status issue (tools/update_issue.py). It runs twice a day.
• Summaries of the server-side reports (code coverage, static analysis) can be ingested as data/external/*.json; the state of the Coverity Scan is collected from two sources, the analysis metrics of the project overview page and the summary of the build that was submitted for scanning (tools/fetch_coverity.py, see below).
• The status of the automated manual builds is collected from the status.json files published with the three manual variants (tools/fetch_docs.py, see below), including the words the spellchecker flagged in the development version.
• The monthly GitHub activity of the members of the core team is swept from the GitHub API (tools/fetch_team.py, see below) and reported on a page of its own, linked from the repository activity card.

Notifications

The status issue body is rewritten in place on every update, which does not notify anybody. A comment is posted only when new failures appear or known failures are fixed; comments notify issue subscribers. Subscribe to the pinned status issue to get emails about regressions - and nothing else.

The same applies to the manual builds: a comment goes out when one of the three variants starts failing or falls behind its branch, and again when it recovers. Each of those is announced once, not on every update, which is tracked per manual with a hidden marker in the comment rather than with the run id used for the test suites.

Local use

Everything only needs the Python standard library (plus the gh CLI for the scripts that talk to GitHub):

python3 generator/build_site.py             # data/ -> _site/
python3 tools/ingest_actions.py --dry-run   # what would be ingested
python3 tools/fetch_regression.py --dry-run # latest regression results
python3 tools/fetch_docs.py                 # manual build status
python3 tools/fetch_team.py                 # core developer activity
python3 tools/update_issue.py --repo <owner/repo> --site-url <url> --dry-run

run.json files can also be produced manually from local test runs with tools/regression-tests/merge_results.py (regression tests) in lammps/lammps or tools/junit_to_json.py (any JUnit XML file, e.g. from ctest --output-junit).

Data layout

data/<suite>/<runid>/run.json

<suite> is quick-regression, kokkos-regression, check-examples, full-regression/<config>, or unit-tests/<config>: a suite that is run in more than one configuration keeps them in subdirectories and appears once per configuration. <runid> is <ISO timestamp>_<short sha> and sorts chronologically. The run.json format is documented in tools/rundata.py; a test is recorded there as passed, failed, error, runtest, or skipped, and the website classifies some of them further as it reads them: the errors that hit the time limit of the harness as timeouts, and the skipped tests of the runs archived before the harness had the word runtest as what they are (see below).

Full regression tests

tools/fetch_regression.py archives the results published as https://download.lammps.org/coverage/serial.json and the corresponding parallel.json, openmp.json, and kokkos.json (the -summary.md and -regression.xml files next to them show the same data and are not ingested, since the JSON is a superset of both). The file name is what identifies the configuration: the config_file property does not, three of the four share config.yaml, and only the title of the run spells the difference out - which is why the status issue carries it alongside the suite name where it says more than the name does (rundata.config_label()).

That title is a shorthand, though (MPI+OpenMP, KOKKOS/OpenMP), and it says nothing about the decomposition. The website therefore does not repeat it on the dashboard cards but spells out what each configuration runs on the run page and on the comparison page (rundata.CONFIG_DETAILS), which is also where the order the configurations are listed in comes from (rundata.CONFIG_ORDER: serial, parallel, openmp, kokkos - by what each adds to the one before, not alphabetically).

Only the most recent run is published, so a run that is not picked up before the next one replaces it is lost; the runs are gated by changes in the monitored branch, though, so unchanged results simply stay in place. Since the published files are rewritten even when no new test run happened, ingestion deduplicates on the generation time and commit recorded in the file rather than on its modification time.

Beyond that, the archive keeps one run per commit, since every archived run is a bar of the trend on the dashboard. The test machine only runs when the monitored branch has changed, so a commit that is published twice was run again while the test scripts themselves were being worked on: those results replace the run archived for that commit (rundata.archived_with_commit(), the new run is written before the old one is removed), and where they repeat its every verdict as well they are not archived at all - that is a re-publication rather than a run.

The commit and the branch are read from the commit and branch metadata fields, and recovered from the git_info property where those are missing; that property is also the source of the git describe string kept as version. The website and the status issue read the commit as sha.

The run id is stamped with the generated time where that carries a time zone, and with the publication time from the Last-Modified header where it does not: a generated field without a zone is in the local time of the test machine and cannot be compared with the UTC stamps of the runs ingested from GitHub Actions.

Tests that run out of time

A test that hits the time limit of the test harness is reported as an error like any other, with a message ending in timeout (<n>s expired). Whether it expires depends on the limit in force (180 s serial, 60 s for the others), on how many tests run beside it, and on the machine - so it says nothing about the code. Those runs are classified as timeout (rundata.status_of()), counted apart from the errors, and left out of the broken count that drives the last all OK run and the notification comments. They are not swept under the carpet: they have their own tile, their own band in the trend bars of a card, their own filter and column, a run-to-run comparison lists them as newly out of time, and a test that starts hanging because of a code change shows up there. The limit itself is read back from the messages (rundata.time_limits()), since the run data does not record it.

Runs that complete without a check

Since lammps/lammps#5144 the harness reports a test that ran to completion but could not be checked against anything as runtest: an input without a reference log file (run either shortened to a few steps as a crash test, or in full), a reference log the harness cannot parse, a run without thermo output, or a --preflight-only run in which every input is only parsed and taken one step with -skiprun. Only the run itself was tested, so it is neither passed nor skipped, and the website keeps it apart from both: its own tile and band on the cards, its own column in the status issue, its own filter on the run pages. It is the bulk of what the regression suites report that is not a verdict - 184 of the 811 inputs of the first serial run that carried the word.

Before that, the harness reported the same outcomes as skipped, with a message that starts with completed. rundata.status_of() reads those as runtest as well, so that the archive is one vocabulary from end to end: the band on the trend bars runs through unbroken where the harness changed its wording, and a re-publication of a commit that was archived before the change is not mistaken for a run with hundreds of moved verdicts. In JUnit XML, which has no element for the outcome, the harness writes a runtest as <skipped> with a status="runtest" attribute on the test case, which tools/junit_to_json.py reads back apart.

A test that was broken and now completes as a runtest counts as fixed, in the run-to-run comparison and as the last OK run of a test that breaks again later: the crash is gone, whatever became of the reference log file. This is what merge_results.py upstream reports as fixed too.

What a dashboard card shows

The numbers of the latest run as tiles, the last TREND_RUNS (25) archived runs as one stacked bar each, what changed since the run before, and which branch, commit and time the numbers are of. A bar is as tall as the number of tests of that run and is stacked from the baseline up in the order failed, errors, timed out, skipped, runtest, passed: the outcomes worth watching sit on the baseline, where a change in one of them changes the height of that band rather than shifting everything above it, and the tests that passed float on top, with the runs that completed unchecked right below them, so the top edge of a bar stays the number of tests and the two bands together are what did not break. The bars keep their pitch while the archive fills, with the newest run at the right edge, and they carry the color each outcome has everywhere on the site - which is what makes the tiles above them the legend of the chart.

Reading a regression result

The example inputs were not written to be tests, so a plain pass/fail count is misleading and the run pages group the results the way tools/regression-tests/REPORTING.md in lammps/lammps describes:

• Needs a fix in the examples tree - every test whose attention field names a problem with the input script itself, grouped by kind (rundata.attention_groups()). This is a work list against the repository, not against the code, and it is set independently of the verdict, so a test that passes can carry one. It is also the majority of what the regression suites report: reference log files that match no input, inputs that run a production number of steps, and inputs whose initial velocities depend on the number of MPI processes.
• Worth investigating - the remaining failures, sorted by how early the run deviates from its reference log (rundata.divergence()). A classical MD trajectory is chaotic, so a difference that first appears after a thousand steps says nothing about the code, while one that is there in the very first thermo output cannot be rounding. The late ones are folded away.
• Not really tested - the statuses that are not verdicts, counted per kind, since each implies different work, and grouped by what the harness made of them: the runtests, which ran and only lack a check (no reference log file, a log file the harness cannot parse, no thermo output), the skipped inputs, which were never run (needs a multi-partition run, excluded by the test configuration, a style the binary does not have), and the errors that are not about the code either (package not installed). A wording the site does not know is counted as other rather than dropped (rundata.NOT_TESTED_KINDS), so that a new kind cannot go unnoticed.

compare.html puts the configurations of one commit side by side. It is reached from the run pages of that commit rather than from the dashboard, since it says nothing about a run of any other commit. A test is only counted there where every configuration reaches a verdict on it: inputs that need a fix and inputs that ran out of time are left out, because most of the former cannot match a reference log file that was written with a different number of MPI processes, and they bury everything else.

Because a timeout is the absence of a verdict rather than one, a comparison against a run in which a test timed out falls back to the most recent run before it that did judge that test (rundata.compare_runs() reads older runs lazily, only as far as it needs them). Otherwise a test that keeps failing but flaps through a timeout would be announced as a new failure every time it came back - which is exactly what the archived parallel runs did on 2026-07-27.

Incomplete ingestion

An ingest pass never fails the job it runs in. What it could not take in is written to data/external/ingest.json instead, and the dashboard leads with a warning that says so. A failed job would publish nothing at all and leave the previous page standing, which hides a gap in the data behind results that look current; the point of the report is that a short pass still publishes what it did get and says what is missing.

Three things are reported. Runs that did not come in for a reason another pass may not hit - an artifact that did not download, a zip without the file it should hold, an XML that does not parse - are listed under pending and retried on the following passes by run id, which does not depend on the run listing at all: a run that has meanwhile scrolled out of the window is still reachable that way. After ingest_actions.MAX_ATTEMPTS tries the run is reported as a problem rather than queued forever. Runs that carry nothing this repository can read are listed under problems, but only where they would have advanced a suite: a run older than what the suite already holds is not a gap in its history, which keeps the tail of runs left behind by a corrected workflow out of the report.

The third is the run listing itself. The window is the newest --max-runs completed runs on develop, and lammps/lammps produces some 55 of those a day, so the window reaches back about three and a half days. A listing that comes back out of order - as one did on 2026-08-28, during the GitHub Actions outage of that week - need not reach the newest runs at all, and the pass then finds nothing new and looks exactly like an idle poll, while the results it was meant to pick up quietly scroll out of reach. Each pass therefore checks that the listing is newest-first and that its newest run is not older than the newest run already archived. Neither check stops the pass: they set recheck, which makes the next pass examine a window RECHECK_FACTOR times wider, so that whatever the bad listing skipped is picked up on the next round.

generator/build_site.py reads the report back (ingest_state). The dashboard carries a banner naming what was missed and what is queued, the suites named in it carry a warning sign, and api/summary.json carries the whole report, so that whoever gates a nightly run on the snapshot can tell a suite that reported nothing from one whose result never arrived. A report that has not been rewritten in INGEST_STALE_HOURS is itself a warning: the collectors are polled twice a day, and an ingester that stopped leaves the same data behind as one that found nothing.

Example input check

check-examples.yml in lammps/lammps (Check example inputs /w -skiprun, added with the same pull request as the runtest status) runs every input script of the examples tree with -skiprun appended, on 2 MPI tasks of a GitHub Actions build: the whole script is parsed, every style instantiated, the system set up, and one step taken - the loops of all run and minimize commands end after one step. That finds inputs that no longer parse or crash in setup at a small fraction of the cost of the regression runs, and on every push rather than once a day; it checks nothing numerically. The workflow's own verdict is the failure.yaml it stops on; what is archived here is the JUnit XML it uploads in check-examples-results, converted by tools/junit_to_json.py into data/check-examples/<runid>/run.json (ingest_actions.JUNIT_WORKFLOWS; the commit, the branch and the time are those of the workflow run, since the git_info the harness records is the detached HEAD of the checkout).

By design, every input that completes is a runtest (only a -skiprun check) and none passes, so the card of the suite shows zero passed and zero failed and the suite is all OK exactly when no input errors out. The inputs it cannot judge are skipped and counted per kind on the run page: the ones excluded in config_skiprun.yaml, the ones whose post-run analysis cannot work with a run cut short (cannot be checked with -skiprun, an error message that names a variable or a tally that needs the full run), the multi-partition inputs, and the ones that use a package or feature the Actions build lacks. The dashboard sorts the suite after the full regression cards (rundata.SUITE_ORDER), names it Example Input Check (rundata.SUITE_TITLES), and spells out on its run pages what it runs (rundata.SUITE_DETAILS), since the name of the suite says neither.

The same pull request adds unittest-fftw.yaml (Unittest for Linux, FFTW3 and KOKKOS OpenMP): the unit tests in the build configuration the example check uses - MPI enabled, FFTW3 instead of KISS FFT, the KOKKOS package with its OpenMP backend, and the default (smallbig) integer sizes, which no other Actions configuration has. Its results are ingested like those of the other unit test workflows, as unit-tests/linux-x86_64-fftw.

Published unit test run

tools/fetch_unittest.py archives https://download.lammps.org/coverage/junit.xml under unit-tests/linux-x86_64-gcc. This is the unit test suite run in the same pass as the coverage report, in the machine's native GCC build of x86_64 Linux: a package selection far more complete than the GitHub Actions runners compile, and the only configuration in the matrix that is not ingested from there. It is published in the ctest --output-junit format, which the converter for the GitHub Actions artifacts already reads (tools/junit_to_json.py).

Publication works as for the regression results - a single file rewritten in place - so the same rules apply, and they are implemented by the same helpers in tools/rundata.py: deduplication on the generation time and commit recorded in the file, and one run per commit.

What the JUnit format does not carry is the commit and the branch; it stamps the run in the local time of the test machine, which cannot be compared with the UTC stamps of the other suites, and it says nothing about the build beyond the host name. All of that comes from https://download.lammps.org/coverage/summary.json instead - the second set of data published by this same run, already fetched for the coverage numbers on the dashboard (tools/fetch_external.py). It records the commit in full, the branch, the date of the run as UTC (which is what the run id is stamped with), and the compiler and operating system, kept as properties of the run under the names the full regression runs report the same two in (fetch_unittest.SUMMARY_PROPERTIES).

The two files are the two halves of one run only once it has finished publishing: a run wipes the webroot and fills it again, so both are absent for the length of that (which is why all four regression results can read as 404 for a while), and a summary fetched in between can still be the one of the run before. The abbreviated commit in the test output is of the binary those very tests ran, so a summary that disagrees with it is not the other half of this run: that read is dropped with a warning and retried on the next poll, since what is published stays in place until the next run replaces it.

The git describe string kept as version is read from a version field of the summary where it carries one, and otherwise from the output of the tests themselves: lmp -h prints a Git info (<branch> / <describe>) banner, and the JUnit file quotes what each test printed (fetch_unittest.git_info_of()). That banner also names the branch and the abbreviated commit, which is what carries a run whose summary could not be fetched - with the Last-Modified header of the JUnit file for a stamp, so that a fetch that fails on the summary alone still archives the run instead of dropping it.

That fallback is thin, and deliberately not relied on for anything else: ctest cuts the output it quotes off at 1024 bytes per test (719 of the 991 tests of the run archived on 2026-07-28 are truncated), so the banner survives only because it is printed near the top of the help text. The compiler and the operating system, printed at the end of the same help text, do not survive at all - which is why the summary is the only source for those two.

The name of the ctest suite (e.g. Linux-g++-15) is kept as a property beside them: it names the compiler in short, and it is the only such record for a run archived before the summary reported one.

Documentation build status

The three published variants of the manual - develop (https://docs.lammps.org/latest/), release (https://docs.lammps.org/), and stable (https://docs.lammps.org/stable/) - each carry a status.json in their document root with the documented commit, the build time, and the outcome and duration of the build steps (html, pdf, publish, and, on develop, spelling; see below). tools/fetch_docs.py collects those into data/external/docs.json, together with the current head commit of each branch as queried from GitHub.

The manual is rebuilt hourly but only once per commit hash, so an unchanged status.json normally means there was nothing to do and its age says nothing about the health of the build machine. Freshness is therefore judged by comparing the documented commit against the branch head: a newer head commit shows as pending and, once it has gone unbuilt for more than STALE_HOURS (6 h), as stale. A status.json that cannot be fetched leaves the last known values in place; if it stays unreachable for longer than the same interval, the entry is shown as unknown rather than as a stale success.

Those rules and the docs.json layout live in tools/docsdata.py (the counterpart of tools/rundata.py), so the website and the status issue judge a build the same way and report the same state.

Spellchecker

The build of the development version also runs the spellchecker (make spelling in the doc directory) and lists every word it did not recognize in status.json, one line of <file>:<line>: (<word>) <context> per hit, with their number in the spelling build step. The other two manuals do not run it: a typo can only be fixed on develop.

The list is carried into docs.json as it stands and rendered on a page of its own, spelling.html, which links each hit to the documentation source on GitHub at the commit that was built. The dashboard card and the status issue only say how many words were flagged and link to that page.

A flagged word is not necessarily a misspelling - technical terms, author names, and syntax the checker cannot know belong in doc/utils/sphinx-config/false_positives.txt in the LAMMPS repository - so the number is reported but does not enter the verdict on the build. What the spelling step reports is whether the checker ran, the same as for the html, pdf, and publish steps.

Coverity Scan

tools/fetch_coverity.py collects data/external/coverity.json from the two ends of the scan, which know different things about it:

• metrics - the publicly visible Analysis Metrics of https://scan.coverity.com/projects/lammps-lammps, scraped from the project overview page: outstanding, newly detected and fixed defects, defect density, lines analyzed, and the day of the last analysis. This is the outcome of an analysis, and the page does not say which state of the source tree it was run on.
• build - https://download.lammps.org/analysis/coverity.json, written by the script that builds and submits LAMMPS for scanning (coverity.sh in the lammps-analyze repository, run twice a week and only when the monitored branch has changed). It records the branch, commit, and version that were built, when, and with which compiler on which operating system - the input of an analysis, and none of it visible on the Coverity side.

The two halves are not two views of one run and are not merged into one: the build is submitted from the LAMMPS side, the analysis runs on the Coverity servers and can be delayed considerably, so the metrics usually describe an earlier submission than the last one recorded. The dashboard card states them as two things, and where the last submission postdates the day of the last analysis - which is as precisely as the project page dates it - it is marked analysis pending rather than compared against numbers it cannot be part of.

Whichever half cannot be read keeps the values it had, so a Coverity page that refuses the scrape does not take the recorded submission down with it, and an unreachable summary file does not blank out the metrics.

Core developer activity

tools/fetch_team.py collects data/external/team.json: seven counts per member of the core team and calendar month, over the last 13 months. The site renders them on team.html, linked from the repository activity card of the dashboard - a table of the sums over the twelve completed months, and one line chart per member and count below it. The thirteenth month is the one still running: it is drawn, with its last point left hollow, but left out of every sum, so that a partial month cannot read as a drop in activity.

GitHub has no single activity endpoint, so each count is swept separately and bucketed here:

count	where it comes from
commits	the GraphQL commit history of the default branch
PRs opened, issues opened	one GraphQL search per month, by creation date
PRs merged	a second search per month; there is no merged-by: qualifier, so the merged pull requests are listed and their mergedBy counted
reviews, approvals	the reviews themselves, read off the pull requests each member reviewed
comments	the two repository-wide comment listings, conversation and inline

Three of those need care when reading them:

• commits count for whoever wrote them, PRs merged for whoever merged them. In a repository where one maintainer does most of the merging, the merge column says so and is not a measure of the others' activity.
• approvals are the subset of reviews that approved; a review that requested changes or only commented is in the first number and not in the second. The summary text of a review counts as a review, not as a comment.
• the repository, all contributors row of the table has no entry for reviews and approvals: those are collected one member at a time, from that member's own contributions, so there is no repository-wide number to put beside them.

Every count is bucketed by the UTC month of the event itself rather than read off an aggregate endpoint. That is deliberate, and it is worth saying why: the two aggregates GitHub offers for this are both wrong for a monthly breakdown, neither of them noisily.

• The totals of a GraphQL contributions collection are bucketed by day in the contributor's own profile timezone, and a query window snaps out to whole days in that timezone. For a US/Eastern member, one boundary day alone moved 39 commits across a month queried as UTC.
• Its pullRequestReviewContributions report one contribution per pull request rather than one per review, deduplicated across the whole window asked for. A pull request reviewed in two months is credited to one of them, so the count of a fixed month falls as the surrounding window grows. One member's November came out as 14 reviews read from the pull requests, 10 from a one-month collection, and 7 from an eleven-month one.

Neither is used. Every number is counted from the individual events and their own timestamps, so the same month gives the same answer however wide a window it was asked for - which is checked by sweeping a short window and a long one and comparing the months they share.

A full sweep is about 150 requests and takes a couple of minutes, well inside both rate budgets; the REST search API and its 30 requests per minute are avoided throughout in favour of GraphQL search. Nothing is collected incrementally - the whole window is swept each time, so a month that gains a late edit or loses a deleted comment stays correct.

Reading team membership needs a token with the read:org scope, which the repository GITHUB_TOKEN of Actions does not have; the workflow takes one from the TEAM_TOKEN secret (falling back to ISSUE_TOKEN). As in tools/fetch_activity.py, an incomplete sweep is never published: any failure warns and leaves an existing file untouched, so the page keeps showing the last complete set of counts rather than one in which some members silently dropped to zero. Without the file the page is not built and the dashboard does not link it.