vmupdate: add qvm-template-upgrade by nihalxkumar · Pull Request #213 · QubesOS/qubes-core-admin-linux

nihalxkumar · 2026-05-25T19:46:03Z

This PR introduces the qvm-template-upgrade dom0 command-line utility, that performs an in-place N -> N+1 distribution upgrade of Debian and Fedora TemplateVM or StandaloneVM

fixes: QubesOS/qubes-issues#8605
GSoC 2026 project: Automate Template Version Upgrade

ben-grande

Yay, started. As I've done a review now, I will receive a notification every time you commit. Let me know when you need another look or have doubts by mentioning me.

codecov-commenter · 2026-06-01T12:32:28Z

Codecov Report

❌ Patch coverage is 86.27544% with 290 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.09%. Comparing base (a571230) to head (721ebef).

Files with missing lines	Patch %	Lines
vmupdate/tests/test_version_upgrade_agent.py	83.51%	123 Missing ⚠️
vmupdate/agent/source/dnf/dnf5_api.py	2.15%	91 Missing ⚠️
vmupdate/agent/source/apt/apt_api.py	2.50%	39 Missing ⚠️
vmupdate/agent/source/apt/apt_cli.py	89.18%	12 Missing ⚠️
vmupdate/template_upgrade.py	98.05%	6 Missing ⚠️
vmupdate/agent/source/common/package_manager.py	90.56%	5 Missing ⚠️
vmupdate/agent/source/dnf/dnf_api.py	89.79%	5 Missing ⚠️
vmupdate/agent/source/common/progress_reporter.py	92.59%	4 Missing ⚠️
vmupdate/agent/source/dnf/dnf_cli.py	92.30%	3 Missing ⚠️
vmupdate/tests/test_template_upgrade.py	99.82%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
- Coverage   71.36%   68.09%   -3.28%     
==========================================
  Files          12       34      +22     
  Lines        1383     4604    +3221     
==========================================
+ Hits          987     3135    +2148     
- Misses        396     1469    +1073

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ben-grande · 2026-06-05T06:48:10Z

PipelineRetryFailed

nihalxkumar · 2026-06-05T09:27:26Z

It's showing a successful run

ben-grande · 2026-06-05T09:58:23Z

Some tests are not enabled on this repo, such as mypy, black and pylint. See this as an example: https://github.com/QubesOS/qubes-core-admin/blob/main/.gitlab-ci.yml. Can you create a separate PR to enable those checks?

nihalxkumar · 2026-06-05T10:56:30Z

Sure, will do

nihalxkumar · 2026-06-09T07:21:00Z

We can squash here if it's looks fine. Will also have to rebase as this is 24 commits behind.

ben-grande · 2026-06-09T08:03:45Z

Looks fine to squash.

marmarek · 2026-06-10T10:51:56Z

Oh no, commit message in the other PR closed it ...

marmarek · 2026-06-10T10:54:06Z

I see a conflict here - CI will not run until it's resolved.

ben-grande

I have taken a look through the new code. Thanks for the progress, I like to see it evolving. Just minor review though

ben-grande · 2026-06-24T13:02:47Z

https://github.com/QubesOS/qubes-core-admin-linux/pull/213/changes/a0289a754c4936f215ac7f6920d678b4821ec213..a90eee15a3ce76eb66ca5d87eea4bf923789955e

Can you share more details of what fails on cleanup?

nihalxkumar · 2026-06-24T22:13:44Z

When I tried upgrading fedora 41 -> 42 there was a cleanup failure which happened after the successful version upgrade.

logs as shared privately on tuesday:

https://gist.github.com/nihalxkumar/01fb990deab3960cb28d680773bb1089#file-upgrade42-log-L48-L55

we can see Complete! in the above log.

https://gist.github.com/nihalxkumar/0e4edccb4f4409b1a4daf5c44b6f576f

After version_upgrade() returns success, the agent still runs cleanup. The old code used the cleanup exit code when deciding the final agent exit code.
For dnf this includes DNFCLI.clean() (dnf clean packages), and the transport also removes /run/qubes-update/.

a90eee1 (last commit) is based on that failure boundary. After the release transaction has succeeded, a later cleanup failure should not cause rollback of the upgraded clone.

ben-grande · 2026-06-25T04:47:36Z

But we can't ignore all cleanup failures. We don't know what is causing it, so this needs to be investigated.

parulin

I have some suggestion for better integration with the rest of the docs. If you have any question, feel free to ask.

This file is in doc/tools instead of doc/manpages, why?

parulin · 2026-07-21T12:30:59Z

As I understand, this tool should also be promoted in qubes-doc, in the user/templates/*/*-upgrade.rst files for fedora and debian.

nihalxkumar · 2026-07-22T17:35:49Z

@parulin you are mistaking this repository as qubes-core-admin-client

nihalxkumar · 2026-07-23T21:13:32Z

Debian upgrade end-to-end successful

https://gist.github.com/nihalxkumar/3f64e3a056e9ce367e9787d5bd94b9df
https://gist.github.com/nihalxkumar/212e885811589b85fdd23dafdfddf887

deeplow · 2026-07-30T09:08:49Z

+AUTHORS
+=======
+| Nihal Kumar <nihalxkumar at tutamail dot com>
+| Ben Grande <ben at invisiblethingslab dot com>


Not sure how credit works here, but it could be worth also adding @kennethrrosen as the creator of the tool that lead to this. 🤔

Kenneth's script only served as a conceptual inspiration. I have no problems mentioning this but how should I do it?

Exactly. That's in the sense I meant. Maybe:

Suggested change

| Ben Grande <ben at invisiblethingslab dot com>

| Ben Grande <ben at invisiblethingslab dot com>

| Kenneth R. Rosen <kennethrrosen at proton.me>

How about this:

https://github.com/QubesOS/qubes-core-admin-linux/pull/213/changes#diff-3acefc53177842eb0ded6639f194362822f6d512bf9fca09f96fd24d09e158cbR96-R102

ben-grande · 2026-08-10T09:02:58Z

Hi. Could you please rebase your PR from the main branch to fix the conflicts?

Add a --version-upgrade mode to the in-VM agent so a qube can be moved to the next major distribution release. The dnf backend distro-syncs to the target releasever; the apt backend updates, rewrites source codenames, and dist-upgrades. Both confirm os-release before reporting success.

The agent emits a final explicit 100% even when callback progress already reached it, causing a bogus "err: 100.00" line. Drop the duplicate 100 milestone after progress has finished.

Upgrade a clone of a TemplateVM or StandaloneVM to the next distribution release, leaving the original qube untouched.

Document the options, the clone name derivation, and which template-* features the tool rewrites, including why the inherited epoch, version and release make qvm-template list the upgraded clone as upgradeable.

Progress hit 100% before post-transaction scriptlets finished, making the bar appear frozen for the remaining work. Cap package progress at 97% and advance asymptotically through post-transaction scriptlets. Report 100% after qubes.PostInstall so the bar stays responsive until everything is done.

During release upgrades the root volume briefly holds both releases, which can trigger a spurious disk-space warning toast. Suppress disk-space-not-notify on the clone during the upgrade and restore it afterwards.

Each Debian release-upgrade step drove the progress bar to 100%, silencing later steps. Give each step its own weighted slice of the bar so progress stays proportional throughout the upgrade.

qubesos-bot · 2026-08-24T20:28:20Z

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026082500-devel&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026050504-devel&flavor=update

system_tests_network_updates
- TemplateUpgrade_fedora-43-xfce: test_100_full_release_upgrade (failure)
  AssertionError: qvm-template-upgrade returned unexpected code: 1 no...

Failed tests

1 failures

system_tests_network_updates
- TemplateUpgrade_fedora-43-xfce: test_100_full_release_upgrade (failure)
  AssertionError: qvm-template-upgrade returned unexpected code: 1 no...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/176874#dependencies
Nothing fixed

Unstable tests

Details

system_tests_network_updates
system_tests/Fail (2/5 times with errors)
- job 192061 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
- job 192160 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
system_tests/Fail (1/5 times with errors)
- job 192568 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
system_tests/Fail (1/5 times with errors)
- job 191227 Tests qubes.tests.integ.vm_update failed (exit code 1), details rep...
system_tests/Failed (3/5 times with errors)
- job 191227 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
- job 192061 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
- job 192160 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
system_tests/Failed (1/5 times with errors)
- job 192568 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
TC_00_Dom0Upgrade_whonix-gateway-18/test_000_update_salt (1/5 times with errors)
- job 192568 ?[0;31mDuration: 204145.11 ms?[0;0m... ?[0;31m Changes: ?[0;0m
TC_00_Dom0Upgrade_whonix-gateway-18/test_005_update_flag_clear (1/5 times with errors)
- job 192160 Error: Failed to download metadata for repo 'test': Cannot download...
TC_00_Dom0Upgrade_whonix-gateway-18/test_006_update_flag_clear (1/5 times with errors)
- job 192160 Error: Failed to download metadata for repo 'test': Cannot download...
TC_00_Dom0Upgrade_whonix-gateway-18/test_010_instal (1/5 times with errors)
- job 192061 Error: Failed to download metadata for repo 'test': Cannot download...
VmUpdates_fedora-44-xfce/test_011_pkcon_via_proxy (1/5 times with errors)
- job 191227 AssertionError: 7 not found in [0, 100] : pkcon install -y test-pkg...
VmUpdates_debian-13-xfce/test_110_update_via_proxy_qubes_vm_update (1/5 times with errors)
- job 191227 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

Performance Tests

Performance degradation:

No issues

Remaining performance tests:

No remaining performance tests

ben-grande suggested changes May 25, 2026

View reviewed changes

ben-grande reviewed May 27, 2026

View reviewed changes

Comment thread vmupdate/template_upgrade.py Outdated

nihalxkumar marked this pull request as ready for review June 2, 2026 05:29

ben-grande suggested changes Jun 7, 2026

View reviewed changes

nihalxkumar force-pushed the qvm-template-upgrade branch from 71dee9e to 6c3d431 Compare June 9, 2026 11:55

nihalxkumar mentioned this pull request Jun 9, 2026

Add CI coverage for vmupdate formatting and pylint checks. #215

Merged

marmarek closed this in 497d467 Jun 10, 2026

marmarek reopened this Jun 10, 2026

nihalxkumar force-pushed the qvm-template-upgrade branch 6 times, most recently from 16b99f5 to f1f8fc1 Compare June 14, 2026 16:29

nihalxkumar changed the title ~~qvm-template-upgrade: add CLI skeleton and orchestration flow~~ qvm-template-upgrade: add orchestration workflow and in-VM agent Jun 14, 2026

ben-grande reviewed Jun 15, 2026

View reviewed changes

Comment thread vmupdate/agent/source/common/package_manager.py

Comment thread vmupdate/agent/source/common/package_manager.py

Comment thread vmupdate/agent/source/dnf/dnf_cli.py Outdated

Comment thread vmupdate/agent/source/dnf/dnf_cli.py Outdated

nihalxkumar force-pushed the qvm-template-upgrade branch from ad97dca to a0289a7 Compare June 15, 2026 17:11

ben-grande suggested changes Jul 17, 2026

View reviewed changes

Comment thread vmupdate/agent/source/apt/apt_cli.py Outdated

Comment thread vmupdate/agent/source/apt/apt_cli.py Outdated

Comment thread vmupdate/agent/source/apt/apt_cli.py Outdated

Comment thread vmupdate/agent/source/apt/apt_cli.py Outdated

nihalxkumar mentioned this pull request Jul 21, 2026

Document qvm-template-upgrade in tools reference and template upgrade pages QubesOS/qubes-doc#1737

Open

nihalxkumar force-pushed the qvm-template-upgrade branch from 0a8f8ac to 80e3d99 Compare July 21, 2026 11:01

parulin reviewed Jul 21, 2026

View reviewed changes

nihalxkumar force-pushed the qvm-template-upgrade branch from 80e3d99 to c3f550e Compare July 23, 2026 21:04

ben-grande mentioned this pull request Jul 24, 2026

Lint with mypy on CI #219

Merged

deeplow reviewed Jul 30, 2026

View reviewed changes

nihalxkumar added 8 commits August 24, 2026 12:14

vmupdate: ignore the agent's duplicate final progress milestone

270e377

The agent emits a final explicit 100% even when callback progress already reached it, causing a bogus "err: 100.00" line. Drop the duplicate 100 milestone after progress has finished.

vmupdate: add qvm-template-upgrade

33383cd

Upgrade a clone of a TemplateVM or StandaloneVM to the next distribution release, leaving the original qube untouched.

doc: add qvm-template-upgrade manpage

7016d19

Document the options, the clone name derivation, and which template-* features the tool rewrites, including why the inherited epoch, version and release make qvm-template list the upgraded clone as upgradeable.

vmupdate: suppress qui-disk-space toast during template-upgrade runs

dcf464b

During release upgrades the root volume briefly holds both releases, which can trigger a spurious disk-space warning toast. Suppress disk-space-not-notify on the clone during the upgrade and restore it afterwards.

vmupdate: give each Debian release-upgrade step its own slice of the bar

706ef26

Each Debian release-upgrade step drove the progress bar to 100%, silencing later steps. Give each step its own weighted slice of the bar so progress stays proportional throughout the upgrade.

vmupdate: fix Optional annotations and add missing tests

5a78c27

nihalxkumar force-pushed the qvm-template-upgrade branch from c3f550e to 5a78c27 Compare August 24, 2026 12:39

nihalxkumar mentioned this pull request Aug 24, 2026

Add qvm-template-upgrade-integtration tests QubesOS/qubes-core-admin#875

Open

nihalxkumar changed the title ~~qvm-template-upgrade: add orchestration workflow and in-VM agent~~ vmupdate: add qvm-template-upgrade Aug 24, 2026

nihalxkumar force-pushed the qvm-template-upgrade branch from 4c27ce0 to 3b2de76 Compare August 24, 2026 13:22

vmupdate: apply black formatting and fix dnf5 estimate test

721ebef

nihalxkumar force-pushed the qvm-template-upgrade branch from 3b2de76 to 721ebef Compare August 24, 2026 13:39

marmarek added the openqa-group-1 label Aug 24, 2026

qubesos-qa Bot temporarily deployed to qa August 24, 2026 17:11 Inactive

qubesos-bot mentioned this pull request Aug 24, 2026

DONOTMERGE - Enable qmemman debug log QubesOS/qubes-core-admin#783

Open

marmarek removed the openqa-group-1 label Aug 27, 2026

	\| Ben Grande <ben at invisiblethingslab dot com>
	\| Ben Grande <ben at invisiblethingslab dot com>
	\| Kenneth R. Rosen <kennethrrosen at proton.me>

Uh oh!

Uh oh!

Conversation

nihalxkumar commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ben-grande left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ben-grande commented Jun 5, 2026

Uh oh!

nihalxkumar commented Jun 5, 2026

Uh oh!

ben-grande commented Jun 5, 2026

Uh oh!

nihalxkumar commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nihalxkumar commented Jun 9, 2026

Uh oh!

ben-grande commented Jun 9, 2026

Uh oh!

marmarek commented Jun 10, 2026

Uh oh!

marmarek commented Jun 10, 2026

Uh oh!

ben-grande left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ben-grande commented Jun 24, 2026

Uh oh!

nihalxkumar commented Jun 24, 2026

Uh oh!

ben-grande commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parulin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

parulin commented Jul 21, 2026

Uh oh!

nihalxkumar commented Jul 22, 2026

Uh oh!

nihalxkumar commented Jul 23, 2026

Uh oh!

deeplow Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

nihalxkumar commented May 25, 2026 •

edited

Loading

codecov-commenter commented Jun 1, 2026 •

edited

Loading

qubesos-bot commented Aug 24, 2026 •

edited

Loading