Skip to content

fix(x86_64): serialize full KVM custom MSR range for snapshot#5738

Open
joy-allen wants to merge 1 commit intofirecracker-microvm:mainfrom
virt-pvm:junchuan/fix-msr
Open

fix(x86_64): serialize full KVM custom MSR range for snapshot#5738
joy-allen wants to merge 1 commit intofirecracker-microvm:mainfrom
virt-pvm:junchuan/fix-msr

Conversation

@joy-allen
Copy link

@joy-allen joy-allen commented Mar 6, 2026

Changes

Replace the individual MSR entries with a single range covering the full KVM custom MSR space so that all KVM-defined MSRs are included during snapshot serialization.

Reason

KVM reserves the range 0x4b564d00-0x4b564dff for custom MSRs. The current snapshot implementation only serializes a small subset of these MSRs, which risks missing newly introduced KVM features. Some MSRs are already absent from the list (e.g. MSR_KVM_ASYNC_PF_INT and MSR_KVM_ASYNC_PF_ACK).

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkbuild --all to verify that the PR passes
    build checks on all supported architectures.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.

  • This functionality cannot be added in rust-vmm.

@joy-allen
Copy link
Author

Hi @Manciukic @zulinx86 ,
Could you please take a look at this PR when you have time? I'd appreciate any feedback.
Thanks!

@Manciukic
Copy link
Contributor

Hey @joy-allen, thanks for the contribution! We will take a deeper look in the next days. In the meantime, it would be helpful if you could provide some additional information.
Is there a specific problem or issue you're solving with this PR? Or is it just an improvement for the future?
Also, I'm wondering what's the impact of serialising MSRs like APF INT/ACK. Will the state in KVM be different from the MSRs we restore? Will that be an issue? Will KVM even allow to set these?
Thanks!

KVM reserves the range 0x4b564d00-0x4b564dff for custom MSRs. The
current snapshot implementation only serializes a small subset of
these MSRs, which risks missing newly introduced KVM features. Some
MSRs are already absent from the list (e.g. MSR_KVM_ASYNC_PF_INT and
MSR_KVM_ASYNC_PF_ACK).

Replace the individual MSR entries with a single range covering the
full KVM custom MSR space so that all KVM-defined MSRs are included
during snapshot serialization.

Signed-off-by: Zhiheng Tao <junchuan.tzh@antgroup.com>
@joy-allen
Copy link
Author

Hey @joy-allen, thanks for the contribution! We will take a deeper look in the next days. In the meantime, it would be helpful if you could provide some additional information. Is there a specific problem or issue you're solving with this PR? Or is it just an improvement for the future? Also, I'm wondering what's the impact of serialising MSRs like APF INT/ACK. Will the state in KVM be different from the MSRs we restore? Will that be an issue? Will KVM even allow to set these? Thanks!

Hi @Manciukic , thanks for taking a look and for the questions.

This PR is motivated by two concrete issues I encountered with Dragonball/Firecracker save/restore:

  1. KVM Async Page Fault (Linux 5.10). If the guest enables APF, the MSRs MSR_KVM_ASYNC_PF_INT and MSR_KVM_ASYNC_PF_ACK become part of the vCPU state. Since these MSRs are not included (in Dragonball snapshot), restoring a VM may cause the guest to hang.
  2. PVM-specific MSRs. When running with PVM (Pagetable-based Virtual Machine) enabled, additional MSR_PVM_* registers are introduced. Since these are also not serialized today, PVM does not function correctly after restore.

While these two cases could be fixed by explicitly adding only the APF and PVM MSRs, this PR takes a slightly more general approach by serializing the MSRs exposed by KVM to the guest. The intention is to avoid similar issues when new virtualization features introduce additional MSRs. More generally, the goal is to keep snapshot/restore aligned with the principle that guest-visible CPU state should be preserved across save/restore.

Regarding MSRs such as APF INT/ACK: these are part of the vCPU state exposed by KVM, and KVM allows reading and writing them through the existing MSR APIs. In testing on Linux 5.10, restoring these values worked correctly and allowed the guest to resume.

Please let me know if more details or testing results would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants