Draft PR: Add Steal Time support in ARM

[DakshinD]

Opening this PR as a draft as we continue working on this issue to add steal time support for ARM. Please see below for the changes we have implemented so far.

Changes

• Added PVTime struct in src/vmm/src/arch/aarch64/pvtime/mod.rs that allocates and stores per-vCPU stolen_time regions

  • It offers a new constructor to create the device (allocating system memory and storing regions per vcpu), and register_vcpu which attempts to use kvm_set_device_attr to register the corresponding IPA for a vcpu

• Created a function setup_pv_time in src/vmm/src/builder.rs which is invoked in build_microvm_for_boot

  • Checks if PVTime is supported using kvm_has_device_attr
  • Creates a PVTime device
  • Iterates over vcpu fd's to register their IPAs for steal_time struct

• Started using debug/info logs while booting microvm for testing

Current problems:

Future Progress

• We hope to continue making progress on this PR by:
  • Getting feedback on our current design of the PVTime device and setup_pv_time
  • Support snapshotting
  • Any other things we need to address

Reason

Attempting to resolve issue #4866

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

• I have read and understand CONTRIBUTING.md.
• I have run tools/devtool checkstyle to verify that the PR passes the
  automated style checks.
• I have described what is done in these changes, why they are needed, and
  how they are solving the problem in a clear and encompassing way.
• I have updated any relevant documentation (both in code and in the docs)
  in the PR.
• I have mentioned all user-facing changes in CHANGELOG.md.
• If a specific issue led to this PR, this PR closes the issue.
• When making API changes, I have followed the
  Runbook for Firecracker API changes.
• I have tested all new and changed functionalities in unit tests and/or
  integration tests.
• I have linked an issue to every new TODO.

---

• This functionality cannot be added in rust-vmm.

[Manciukic]

Let's try making the proposed change to pass the right address and see if it works.
I haven't left any comment regarding style or design to focus on functionality first. Once we get something working we can see how to make it "look better".

[Manciukic]

I got confused by the KVM API as well! Passing a user pointer to the IPA to kvm is ok, but it needs to be 64B aligned.

The hint was in the KVM documentation:

-EINVAL Base address not 64 byte aligned

https://www.kernel.org/doc/html/v5.8/virt/kvm/devices/vcpu.html#attribute-kvm-arm-vcpu-pvtime-ipa

[DakshinD]

Thank you so much for your fast response! Totally missed the docs for 64 byte alignment, but that ended up fixing the bugs and I can successfully boot a microVM manually. If you think its a good time now, we can look to flesh out the implementation and "make it better".

Other than that, we believe the next step is adding unit/integration tests to make sure our device is working properly, as well as implementing persistence across snapshots. Any leads on how to do both of these things would be really helpful.

The lead we have been looking at for snapshotting is the Persist interface.

• We store the MicrovmState in save_snapshot, so we need to first store our PVTime device somewhere in the vmm
• Then we implement Persist for PVTime, and store the PVTimeState we get from save() somewhere in the MicrovmState (could we extend MicrovmState to include this? or add this state to one of the other states already existing inside of it?)
• Finally, would need to write comprehensive tests to make sure this works