Skip to content

Updating behaviour on concurrent access to a pending VM shutdown oper… - #876

Draft
ekdama wants to merge 3 commits into
QubesOS:mainfrom
ekdama:patch-shutdown
Draft

Updating behaviour on concurrent access to a pending VM shutdown oper…#876
ekdama wants to merge 3 commits into
QubesOS:mainfrom
ekdama:patch-shutdown

Conversation

@ekdama

@ekdama ekdama commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Closes QubesOS/qubes-issues#11076

Now, the behaviours that could make a concurrent call (like the Halted check), waiting for a shutdown of the VM through the qvm-shutdown --wait command will also test if there is a waiter present in the VM.
The waiter job is to act as a synchronization primitive, when it is finally consumed all the concurrent shutdown request will exit, if the wait flag is enabled.

Is the event firing handled correctly? I only throw the pre-shutdown event when there is not a pending operation.

@marmarek marmarek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like kill() could use a similar change (if you're up for it)

Comment thread qubes/vm/qubesvm.py Outdated
"""

if self.is_halted():
pending = self.is_halted() and not self.__waiter is None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could use a comment explaining what case it handles (in-progress shutdown that is already at the storage stopping stage). Or if not a comment, at least include it in the commit message.

@marmarek

marmarek commented Aug 26, 2026

Copy link
Copy Markdown
Member

Is the event firing handled correctly? I only throw the pre-shutdown event when there is not a pending operation.

Yes, it makes sense.

@ben-grande

Copy link
Copy Markdown
Contributor

I think domain-shutdown is fired only once. Could you write an integration test that register domain-shutdown event handler and check how many times it is called? It should probably be in qubes/tests/integ/basic.py.

@ekdama

ekdama commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Sorry, I had some personal issues this week and had to be AFK. I believe that I can get the requested changes done in the following days.

Comment thread qubes/vm/qubesvm.py
raise qubes.exc.QubesVMNotStartedError(self)

try:
await self.fire_event_async(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a second shutdown is called on domain-pre-shutdown? Can you write a test that does that? I'd like to test if pending on the second shutdown request will be False, in case domain is not halted yet.

@marmarek

marmarek commented Sep 2, 2026

Copy link
Copy Markdown
Member

IMO domain-pre-shutdown can be fired multiple times, if qube is still shutting down (this allows retrying if it ignored the first request for some reason). But when it's already cleaning up (storage stop etc), domain-pre-shutdown should not be fired anymore.
As for domain-shutdown, it should be fired once per actually completed shutdown, regardless how many calls triggered that shutdown.

I think that's the current behavior, but some test for it could be useful.

@ekdama

ekdama commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

With the pending condition, the integration test looped firing domain-pre-shutdown events.
I fixed it by checking the __waiter as the first thing in the try block.

About kill, how should I handle it? I made these small changes, but I don't see a need since it does not have a wait option.

Comment thread qubes/vm/qubesvm.py Outdated
"""The best option is to fire the domain-pre-shutdown after self.__waiter check and assignment, to avoid race conditions and undesired synchronization behaviour."""
if self.__waiter is None:
self.__waiter = asyncio.get_running_loop().create_future()
await self.fire_event_async(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the domain-pre-shutdown fails (is rejected), the self.__waiter will continue to be None

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I clean self.__waiter in the except: block? This would avoid creating a __waiter in an unsuccesful operation.

Or else would be better creating a try except block wrapping 1681 to 1684?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#876 (comment)

Let's actually allow domain-pre-shutdown to repeat depending on the stage of shutdown.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weakening the pending condition by just checking self.__waiter could make domain-pre-shutdown repeatable. Could it work?

@@ -0,0 +1 @@
2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some files included in tests-data by mistake?

@ekdama ekdama Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I am correcting this and linting mistakes as of now. This also broke some test. I will eventually squash all commits.

Comment thread qubes/vm/qubesvm.py
if timeout is None:
timeout = self.shutdown_timeout
try:
await asyncio.wait_for(waiter, timeout=timeout)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One side-effect of calling this function again, is that it doesn't connect to the previous timeout, thus if it is hangs here, it will timeout later, possibly raising timeout twice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's check pending also there?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

        elif not pending:
       elif is_running:

We actually want to wait for the full shutdown, independent if it is report as running or it is the second call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes sorry, I was confused. is_running and not pending are not aimed to the timeout call but to avoid calling libvirt_domain.destroy() or other domain specific code more than once, as I believe is the expected behaviour.

Then the waiter is acquired, not concerned by the current VM state (except the initial is_halted check).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qvm-shutdown --wait succeeds too soon if VM shutdown was already in progress and volumes were stopping

3 participants