Optimize data cluster format for VHD and QCOW#6895
Optimize data cluster format for VHD and QCOW#6895last-genius wants to merge 10 commits intoxapi-project:26.1-lcmfrom
Conversation
|
Opened the corresponding xs-opam PR: xapi-project/xs-opam#755 |
| | x :: y :: _ -> | ||
| (to_int x, to_int y) | ||
| | _ -> | ||
| raise (Invalid_argument "Invalid JSON") |
There was a problem hiding this comment.
You might want to report the json.
There was a problem hiding this comment.
It can be rather large and pollute the logs... This shouldn't happen unless you have a version incompatibility, really
…rmat Qcow_stream now uses Qcow_mapping to store information on allocated clusters, which offers .to_interval_seq, outputting a list of pairs representing intervals of allocated virtual clusters. Signed-off-by: Andrii Sultanov <[email protected]>
This allows to switch on the more efficient interval format later. (QCOW always uses the new format) Signed-off-by: Andrii Sultanov <[email protected]>
This is just an easy way to make sure the semantics are preserved in any future refactorings, without having to run full VHD exports. Signed-off-by: Andrii Sultanov <[email protected]>
This command returns a more efficient representation of allocated clusters (when compared to read_headers), utilizing a sparse interval format instead of returning every single allocated cluster. This is the more efficient option, decreasing the filesize and memory usage in vhd-tool, but it's currently under a feature flag, so it's added as a new command instead of replacing read_headers immediately. Cram test for read_headers is still passing, so this refactoring has preserved the legacy format. Signed-off-by: Andrii Sultanov <[email protected]>
Signed-off-by: Andrii Sultanov <[email protected]>
Since the runtime feature flag vhd_legacy_blocks_format determines which block format is used to describe allocated VHD clusters, this requires duplicate parse_header_interval functions for VHD and QCOW. The right functions are selected in stream_vdi based on the feature flag. Signed-off-by: Andrii Sultanov <[email protected]>
…er allocation Instead of using a set with every individual allocated cluster index as a member, use a sorted list of intervals to verify if cluster is allocated - this uses much less memory and directly follows from the JSON format qcow-stream-tool and vhd-tool output now. Signed-off-by: Andrii Sultanov <[email protected]>
…_clusters nonzero_clusters no longer contain every single allocated cluster and instead are intervals of allocated clusters. Signed-off-by: Andrii Sultanov <[email protected]>
… files Signed-off-by: Andrii Sultanov <[email protected]>
last-genius
force-pushed
the
asv/8.3-sparse-vhd-qcow
branch
from
d41347a to
bcf47a8
Compare
|
Changed the approach:
This way, XS (which only uses VHD) can keep using the legacy format until they can ensure the new format didn't accidentally break anything else. XCP-ng will flip the feature flag to make VHD consistent with QCOW - we will use the interval-based format. |
| $ dd if=/dev/zero of=test.raw bs=2097152 count=10 seek=3 conv=notrunc > /dev/null 2>&1 | ||
|
|
||
| Convert to .vhd | ||
| $ qemu-img convert -f raw -O vpc test.raw test.vhd |
There was a problem hiding this comment.
Does this dependency need to be declared in the opam metadata?
Signed-off-by: Andrii Sultanov <[email protected]>
| cohttp | ||
| cohttp-lwt | ||
| conf-libssl | ||
| (conf-qemu-img :with-test) |
There was a problem hiding this comment.
There's no conf package for qemu-img, even though for some reason I thought it existed.
Another way of configuring it is:
depexts: [
["qemu-img"] {with-test}
]
There was a problem hiding this comment.
I added it last year, seems to still be there: https://github.com/ocaml/opam-repository/blob/master/packages/conf-qemu-img/conf-qemu-img.1/opam
There was a problem hiding this comment.
I don't see it in xs-opam, though
There was a problem hiding this comment.
Is this waiting for a review from Edwin, then? I think it was him that raised concerns
3 participants
mirage/ocaml-qcow#134 changes the type of the data structure containing info on allocated data clusters, returning allocated intervals instead of all the virtual cluster addresses. Change
qcow2-to-stdoutto the new interval-based format.Add
vhd-tool read_headers_intervalcommand which also conforms to this new format, and change the parsing code instream_vdito accept both formats depending on a feature flag. Add cram tests verifying legacy format is preserved as-is.I've ran vm export and vdi integrity quicktests and tested this extensively locally. The PR will only build once the new ocaml-qcow version is packaged into xs-opam, so keeping this as draft for now.