Skip to content

Add qvm-template-upgrade-integtration tests - #875

Open
nihalxkumar wants to merge 1 commit into
QubesOS:mainfrom
nihalxkumar:add-vm-template-upgrade-integ-tests
Open

Add qvm-template-upgrade-integtration tests#875
nihalxkumar wants to merge 1 commit into
QubesOS:mainfrom
nihalxkumar:add-vm-template-upgrade-integ-tests

Conversation

@nihalxkumar

Copy link
Copy Markdown

@nihalxkumar
nihalxkumar force-pushed the add-vm-template-upgrade-integ-tests branch from ef0af26 to b35c376 Compare August 24, 2026 13:22
@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.70%. Comparing base (1ad5f9b) to head (b35c376).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #875      +/-   ##
==========================================
- Coverage   70.72%   70.70%   -0.03%     
==========================================
  Files          61       61              
  Lines       14315    14315              
==========================================
- Hits        10124    10121       -3     
- Misses       4191     4194       +3     
Flag Coverage Δ
unittests 70.70% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@marmarek

Copy link
Copy Markdown
Member

openQArun PR_LABEL=openqa-group-1 TEST=system_tests_network_updates

@qubesos-bot

qubesos-bot commented Aug 24, 2026

Copy link
Copy Markdown

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026083119-devel&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026050504-devel&flavor=update

Failed tests

No failures!

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/176874#dependencies
Nothing fixed

Unstable tests

Details
  • system_tests_network_updates

    system_tests/Fail (1/5 times with errors)
    • job 193004 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
    system_tests/Fail (1/5 times with errors)
    • job 193841 Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...
    system_tests/Fail (1/5 times with errors)
    • job 193757 Tests qubes.tests.integ.vm_template_upgrade failed (exit code 1), d...
    system_tests/Fail (1/5 times with errors)
    • job 193841 Tests qubes.tests.integ.vm_template_upgrade failed (exit code 1), d...
    system_tests/Failed (1/5 times with errors)
    • job 193004 # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...
    system_tests/Failed (1/5 times with errors)
    • job 193757 # Test died: command 'sudo python3 split_logs.py --junit-xml=nose2-...
    system_tests/Failed (1/5 times with errors)
    • job 193841 # Test died: command 'sudo python3 split_logs.py --junit-xml=nose2-...
    TC_00_Dom0Upgrade_whonix-gateway-18/test_000_update_qubes_vm_update (1/5 times with errors)
    • job 193841 ~~~~~~~~~^^^^^^^^^^^^^^^^^^... AssertionError: qubes-vm-update failed:
    TC_00_Dom0Upgrade_whonix-gateway-18/test_010_instal (1/5 times with errors)
    • job 193004 Error: Failed to download metadata for repo 'test': Cannot download...
    TC_11_QvmTemplateMgmtVM_fedora-44-xfce/test_010_template_install (1/5 times with errors)
    • job 193004 AssertionError: qvm-template failed: [Qrexec] ERROR: dnf command is...
    integ/vm_template_upgrade (2/5 times with errors)
    • job 193757 ModuleNotFoundError: No module named 'qubes.tests.integ.vm_template...
    • job 193841 ModuleNotFoundError: No module named 'qubes.tests.integ.vm_template...

Performance Tests

Performance degradation:

No issues

Remaining performance tests:

No remaining performance tests

@marmarek

Copy link
Copy Markdown
Member

The final test didn't run, because QUBES_TEST_RELEASE_UPGRADE wasn't set... let me try again.

source.features.update(tpl.features)
self.loop.run_until_complete(source.clone_disk_files(tpl))
else:
self.loop.run_until_complete(source.create_on_disk())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This, for Standalone, will create an empty qube. Is that intentional? While there are tests below that don't care really, this function could be simpler if it has only the with_disk=True case (unless I'm missing some actual usage).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, with_disk=False was intentional for the CLI validation tests to skip the disk clone overhead since those tests only check dom0 argument validation and never boot the VM.
Should I remove it and always clone?

@marmarek

Copy link
Copy Markdown
Member

Both tests failed, but it might be expected. For Debian, it complains about missing codename for 14 (forky):

2026-08-24 18:26:56,478 [Agent] Run entrypoint with args: Namespace(log='DEBUG', no_refresh=False, force_upgrade=False, no_cleanup=False, leave_obsolete=False, download_only=False, version_upgrade='14', show_output=False, quiet=False, no_progress=False, just_print_progress=False)
2026-08-24 18:26:56,483 [Agent] Selecting package manager.
2026-08-24 18:27:01,160 [Agent] Running distribution version upgrade to 14.
2026-08-24 18:27:01,164 [Agent] Refusing version upgrade: no Debian codename for release '14' in /usr/share/distro-info/debian.csv.
2026-08-24 18:27:01,165 [Agent] version-upgrade err: Refusing version upgrade: no Debian codename for release '14' in /usr/share/distro-info/debian.csv.
2026-08-24 18:27:01,165 [Agent] Notify dom0 about upgrades.
2026-08-24 18:27:06,311 [Agent] run command: apt-get clean
2026-08-24 18:27:06,346 [Agent] command exit code: 0

I'm not sure when new codename is added there, but it is still testing at this point.

For Fedora:

2026-08-24 18:28:15,178 [Agent] Using dnf5.
2026-08-24 18:28:16,833 [Agent] Refusing version upgrade: in-qube release '44' can only move to 45 (single step), not '999'.
2026-08-24 18:28:16,834 [Agent] version-upgrade err: Refusing version upgrade: in-qube release '44' can only move to 45 (single step), not '999'.

Is it also because Fedora 45 isn't released yet? But here, I'm not sure where 999 came from, Fedora rawhide seems to report version 46 now, so not there...

@marmarek

Copy link
Copy Markdown
Member

I retried it with Debian 12 and Fedora 43. Debian worked, but Fedora failed:

# timestamp 2026-08-25T04:20:56.699195
Traceback (most recent call last):
  File "/usr/lib64/python3.13/contextlib.py", line 85, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.13/site-packages/qubes/tests/integ/vm_template_upgrade.py", line 298, in test_100_full_release_upgrade
    self.run_template_upgrade(
    ~~~~~~~~~~~~~~~~~~~~~~~~~^
        "--template",
        ^^^^^^^^^^^^^
    ...<5 lines>...
        timeout=7200,
        ^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/qubes/tests/integ/vm_template_upgrade.py", line 92, in run_template_upgrade
    self.fail(
    ~~~~~~~~~^
        "qvm-template-upgrade returned unexpected code: "
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        )
        ^
    )
    ^
AssertionError: qvm-template-upgrade returned unexpected code: 1 not in (0,)
Plan: upgrade fedora-43-xfce (fedora 43) -> clone test-inst-upgraded (fedora 44)
Cloning fedora-43-xfce -> test-inst-upgraded
test-inst-upgraded: Cloning private volume
test-inst-upgraded: Cloning root volume
Running version-upgrade agent in test-inst-upgraded (-> 44)
test-inst-upgraded:out: Preparing distribution upgrade; dependency calculation may take some time...
test-inst-upgraded:out: Fetching 6 packages [0.00 B]
test-inst-upgraded:out: Fetching Qubes VM Repository (devel testing) [3.38 KiB]
test-inst-upgraded:out: Fetching Fedora 44 - x86_64 [19.78 KiB]
test-inst-upgraded:out: Fetching Fedora 44 openh264 (From Cisco) - x86_64 [986.00 B]
test-inst-upgraded:out: Fetching Fedora 44 - x86_64 - Updates [10.27 KiB]
test-inst-upgraded:out: Fetching Qubes OS Repository for VM (updates) [1.97 KiB]
test-inst-upgraded:out: Fetching qubes testing [1.97 KiB]
test-inst-upgraded:out: Fetching package failure (Fedora 44 - x86_64) Status code: 404 for http://mirror.de.leaseweb.net/fedora/linux/releases/44/Everything/x86_64/os/repodata/repomd.xml (IP: 127.0.0.1)
test-inst-upgraded:out: repomd.xml GPG signature verification error: Signing key not found
test-inst-upgraded:out: repomd.xml GPG signature verification error: Signing key not found
test-inst-upgraded:out: Fetching package failure (Fedora 44 - x86_64) Status code: 404 for http://mirror.de.leaseweb.net/fedora/linux/releases/44/Everything/x86_64/os/repodata/c48e47563bbf65b996c95caf4a608223f982c314cab637e6ab87dd1df67b9d26-primary.xml.zst (IP: 127.0.0.1)
test-inst-upgraded:out: Fetching package failure (Fedora 44 - x86_64) Status code: 404 for http://mirror.de.leaseweb.net/fedora/linux/releases/44/Everything/x86_64/os/repodata/6a37d36914955dc65cf6dc037c37dcf00c64ded464dc34164a674bf65fa16927-comps-Everything.x86_64.xml.zst (IP: 127.0.0.1)
test-inst-upgraded:out: Fetching Qubes OS Repository for VM (updates) [1.97 KiB]
test-inst-upgraded:out: Fetching qubes testing [1.97 KiB]
test-inst-upgraded:out: Calculating package changes...
Upgrade failed: in-VM version-upgrade agent failed for test-inst-upgraded (exit code 24); see /var/log/qubes/update-test-inst-upgraded.log
Removing failed clone test-inst-upgraded
error: in-VM version-upgrade agent failed for test-inst-upgraded (exit code 24); see /var/log/qubes/update-test-inst-upgraded.log

And the log says:

2026-08-25 00:19:50,094 [Agent] Using dnf5.
2026-08-25 00:19:51,971 [Agent] Refusing version upgrade: in-qube release '43' can only move to 44 (single step), not '999'.
2026-08-25 00:19:51,974 [Agent] version-upgrade err: Refusing version upgrade: in-qube release '43' can only move to 44 (single step), not '999'.
2026-08-25 00:19:59,228 Remove /run/qubes-update/
2026-08-25 00:19:59,854 Shutdown test-inst-upgraded
2026-08-25 00:19:59,879 agent exit code: 24
2026-08-25 00:20:17,934 Running update agent for test-inst-upgraded
2026-08-25 00:20:17,937 Transferring files to destination qube: test-inst-upgraded
2026-08-25 00:20:41,879 The agent is starting the task in qube: test-inst-upgraded
2026-08-25 00:20:42,224 [Agent] Using dnf5.
2026-08-25 00:20:43,971 [Agent] Refusing version upgrade: in-qube release '43' can only move to 44 (single step), not '999'.
2026-08-25 00:20:43,974 [Agent] version-upgrade err: Refusing version upgrade: in-qube release '43' can only move to 44 (single step), not '999'.
2026-08-25 00:20:50,844 Remove /run/qubes-update/
2026-08-25 00:20:51,028 Shutdown test-inst-upgraded
2026-08-25 00:20:51,051 agent exit code: 24
2026-08-25 00:21:02,595 Running update agent for test-inst-upgraded
2026-08-25 00:21:02,598 Transferring files to destination qube: test-inst-upgraded
2026-08-25 00:21:02,661 run command in test-inst-upgraded: mkdir -p /run/qubes-update/
2026-08-25 00:21:02,661 Wait for output
2026-08-25 00:21:25,672 run command: cat > /run/qubes-update/agent.tar.gz < /tmp/tmpbwbeg5dp/agent.tar.gz
2026-08-25 00:21:25,966 run command in test-inst-upgraded: tar -xzf /run/qubes-update/agent.tar.gz -C /run/qubes-update/
2026-08-25 00:21:25,966 Wait for output
2026-08-25 00:21:26,516 The agent is starting the task in qube: test-inst-upgraded
2026-08-25 00:21:26,516 /run/qubes-update/agent/entrypoint.py
2026-08-25 00:21:26,516 run command in test-inst-upgraded: /usr/bin/python3 /run/qubes-update/agent/entrypoint.py --log DEBUG --version-upgrade 44
2026-08-25 00:21:26,516 Progress reporting enabled.
2026-08-25 00:21:26,540 Fetching agent process stdout/stderr.
2026-08-25 00:22:30,683 Agent stdout closed.
2026-08-25 00:22:30,700 Agent stderr closed.
2026-08-25 00:22:30,701 Agent process finished.
2026-08-25 00:22:30,701 run command in test-inst-upgraded: cat /var/log/qubes/qubes-update/update-agent.log
2026-08-25 00:22:30,701 Wait for output
2026-08-25 00:21:26,964 [Agent] Run entrypoint with args: Namespace(log='DEBUG', no_refresh=False, force_upgrade=False, no_cleanup=False, leave_obsolete=False, download_only=False, version_upgrade='44', show_output=False, quiet=False, no_progress=False, just_print_progress=False)
2026-08-25 00:21:26,967 [Agent] Selecting package manager.
2026-08-25 00:21:26,978 [Agent] Add `deltarpm=False` to /etc/dnf/dnf.conf
2026-08-25 00:21:29,394 [Agent] Using dnf5.
2026-08-25 00:21:34,540 [Agent] Running distribution version upgrade to 44.
2026-08-25 00:21:34,545 [Agent] run command: dnf clean all
2026-08-25 00:21:35,950 [Agent] command exit code: 0
2026-08-25 00:22:01,870 [Agent] dnf5 repo load for release 44 took 25.908s
2026-08-25 00:22:07,079 [Agent] dnf5 dependency resolution for release 44 took 5.204s
2026-08-25 00:22:07,079 [Agent] Distro-sync found nothing to do.
2026-08-25 00:22:07,136 [Agent] release upgrade did not reach 44; os-release reports '43'.
2026-08-25 00:22:07,136 [Agent] version-upgrade out: Removed 38 files, 24 directories (total of 85 MiB). 0 errors occurred.
2026-08-25 00:22:07,137 [Agent] version-upgrade out: 
2026-08-25 00:22:07,137 [Agent] version-upgrade err: release upgrade did not reach 44; os-release reports '43'.
2026-08-25 00:22:07,137 [Agent] Notify dom0 about upgrades.
2026-08-25 00:22:29,216 [Agent] run command: dnf clean packages
2026-08-25 00:22:29,874 [Agent] command exit code: 0
2026-08-25 00:22:32,634 Remove /run/qubes-update/
2026-08-25 00:22:32,634 run command in test-inst-upgraded: rm -r /run/qubes-update/
2026-08-25 00:22:32,634 Wait for output
2026-08-25 00:22:33,673 Shutdown test-inst-upgraded
2026-08-25 00:22:33,697 agent output: 
2026-08-25 00:22:33,698 agent output: 
2026-08-25 00:22:33,698 agent exit code: 24

@nihalxkumar

Copy link
Copy Markdown
Author

test_020 and test_021 test failure rollbacks using a bogus version 998 for +1. The agent correctly rejects it against /etc/os-release (Refusing version upgrade: in-qube release '43' can only move to 44...). Because all test methods in the test class share /var/log/qubes/qubes-update/update-agent.log, those 999 lines in the log were from test_020/test_021 running earlier in the suite before test_100.

Fedora 43 -> 44
This was an external mirror issue during openQA: the log shows a 404 fetching Fedora 44 repodata. Because repo loading failed, dnf5 resolved 0 packages. The post-upgrade guard detected /etc/os-release was still 43 and exited with error 24, triggering a clean dom0 rollback. The rollback and verification safety net worked as intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants