NMS-20109: Active/passive HA for OpenNMS core - #8700
Open
cgorantla wants to merge 22 commits into
Open
Conversation
(cherry picked from commit 99ab08c)
(cherry picked from commit 203bf32)
(cherry picked from commit 7b39e32)
… on the fly (cherry picked from commit d04e0a9)
(cherry picked from commit a98638e)
(cherry picked from commit 77b60c3)
(cherry picked from commit e7bc80f)
…down (cherry picked from commit 5f54bea)
… became active when, and expose that value in both karaf shell and in ha rest api. (cherry picked from commit 726f4ca)
(cherry picked from commit 2b0ef35)
- Create ha_instance_status from the coordinator at startup (advisory-lock serialized, includes agent_last_seen) instead of the core Liquibase changelog - Write only last_heartbeat from the heartbeat cycle; state columns change on transitions only - Add <mode>heartbeat-only</mode>: an external supervisor owns the state machine and OpenNMS only publishes liveness, never gating startup - Move the HA REST endpoints out of opennms-webapp-rest into the ha-rest OSGi whiteboard bundle; enforce authorization via Spring Security intercept-urls - Replace the filesystem-API config sync with a binary manifest transfer (sha256 diff, atomic apply, deletion propagation, sync-excludes); the hasync account no longer needs ROLE_FILESYSTEM_EDITOR
cgorantla
force-pushed
the
features/ha-management
branch
from
July 30, 2026 16:24
016437e to
949bb1b
Compare
Fail-closed startup Partner check before claiming ACTIVE No assume-inactive-on-error
- Abort startup instead of proceeding standalone when the HA config is
unreadable, required fields are missing, or coordination throws;
coordinator mode now requires a distinct partner-instance-id (enforced
at load, REST update, and reload)
- Clamp failover-threshold to at least 2x the heartbeat interval and
enforce a minimum sync interval before scheduling
- Suppress config sync in heartbeat-only mode, including config reloads
- Halt the JVM when failover cannot stop services after the step-down
is advertised, instead of running on undetectable
- Config sync: support the ${scv:alias:attr|default} fallback syntax and
resolve credentials via the stock vault (honors the configured keystore
type); advertise the serving node's exclusions in the manifest and use
the union of both nodes' lists for deletion propagation; canonicalize
paths before exclusion matching; reject symlink escapes from etc/ and
skip symlinks when building manifests; re-check HA state before every
file write and deletion so a mid-cycle promotion aborts the pass
- Copy mode and sync-excludes in the ha-config shell command
- Use subtree intercept-url patterns for /rest/ha so trailing-slash
variants keep requiring ROLE_ADMIN
- Document featuresBoot.d feature activation in the config template
- Defer the STANDBY/FAILED row write until the service Invoker has finished stopping, so the partner can never promote alongside a node that is still draining; a hung drain falls back to heartbeat staleness at the failover threshold. The split-brain yield keeps its immediate write since the halt is instantaneous. - Arbitrate split-brain only against a partner whose heartbeat is within the failover threshold: an ACTIVE row with a dead heartbeat means the partner stopped while holding the role, so the survivor continues as the sole active instance instead of halting in deference to it. - Replace the reflective HA hooks in Starter and Manager with a StartupLifecycleHook SPI discovered via ServiceLoader; ha-daemon registers the implementation, and its absence is a no-op. - Clarify the sync-excludes example in the config template.
Contributor
Author
|
Lot more tests done. Should be ready for review. Need docs. |
cgorantla
requested review from
christianpape,
dino2gnt,
indigo423,
marshallmassengill,
mershad-manesh and
synqotik
July 31, 2026 20:04
cgorantla
marked this pull request as ready for review
August 3, 2026 16:32
Contributor
Author
Updated docs. Tested locally on Mac with two instances with one Postgres. Seems to work fine. |
dino2gnt
requested changes
Aug 3, 2026
| = High Availability | ||
| :description: Configure an active/passive {page-component-title} pair coordinated through the shared PostgreSQL database. | ||
|
|
||
| ifndef::opennms-prime[] |
Contributor
There was a problem hiding this comment.
Is this the right gate?
I think the most common pattern is:
ifeval::["{page-component-title}" == "..."]
Should also add the Karaf shell commands to the shell reference page.
…drain - register the SIGUSR1 status handler before the startup gate can block, so "opennms status" no longer kills a waiting standby; an empty status file is written when no services exist yet - keep the heartbeat publishing liveness until the terminal state is written post-drain: the partner promotes on the state write, never on staleness beside a still-draining node (a hung drain is escalated by the service manager's stop timeout) - promote when the partner row stays missing beyond the failover threshold instead of waiting forever - serialize config-sync passes so a reschedule cannot overlap a still-running pass
marshallmassengill
requested changes
Aug 3, 2026
Contributor
There was a problem hiding this comment.
These are minor but I think we want to get them documented/sorted:
The enable steps don't have running the install script.... A staged standby has no etc/configured and bin/opennms start refuses outright without it.... I kind of think we may want to consider an option for the install script that does this "install -standby" or something that preps a standby host.
We don't document a procedure for creating the hasync account. Not a big deal but something we should probably do in these docs to be explicit with customers.
"DEGRADED" means two things in one ha-status table. STATE=DEGRADED (restarted primary awaiting failback) versus a DEGRADED=YES column (stale heartbeat, or SECONDARY holding ACTIVE). After a normal successful failover both rows read DEGRADED=YES, which reads as total failure.
We need to document an approximate failover time length and latency requirements. I'm going to do further testing on this.
Some other nits that claude caught:
- nav.adoc:257 still gates on ifdef::opennms-prime[]
- Page :description: (line 2) sits outside the conditional, so the Horizon build emits metadata advertising HA even though the body is a "not included" stub.
- only sees CLI properties, so if the Meridian pipeline sets it as a pom property neither profile activates and HA silently doesn't ship.
- Defaults disagree in a couple of spots: failover-threshold-seconds: class 30, shipped XML 60, javadoc 30. sync-interval-seconds: class 300, shipped XML 60. The page states no defaults, so omitting an element gives 300s sync, not the 60 commented in the file.
- Silent clamping undocumented. clampConfig forces heartbeat ≥5, threshold ≥20 and ≥2× heartbeat, sync ≥5. The 2× floor silently overrides a configured threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add built-in active/passive high availability for a pair of OpenNMS core instances, coordinated through the shared PostgreSQL database.
Architecture
(Core idea and initial implementation credits to @dino2gnt !)
Two instances share one coordination table (ha_instance_status). All timing decisions use the database clock. Each node is configured in etc/ha-configuration.xml with a static role (PRIMARY/SECONDARY) and one of two supervision modes:
coordinator (default): OpenNMS runs the HA state machine itself — startup gating on the standby, heartbeat monitoring, automatic promotion with anti-flap verification, DEGRADED/manual-failback handling, split-brain resolution via active_since arbitration, and config sync.
heartbeat-only: an external supervisor (HA agent or third-party tooling) owns promotion/fencing; OpenNMS only publishes its liveness heartbeat and never gates startup
External References