Skip to content

Commit 2089a3b

Browse files
passarodannycjones
andauthored
Add support for CRC64-NVME checksum algorithm (#1235)
This change adds support for the CRC64-NVME checksum algorithm when using relevant operations in `mountpoint-s3-client` and when appending to existing objects through Mountpoint (using `--incremental-upload` mode). ### Does this change impact existing behavior? No. ### Does this change need a changelog entry? Does it require a version change? Yes. --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). --------- Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk> Signed-off-by: Daniel Carl Jones <djonesoa@amazon.com> Co-authored-by: Daniel Carl Jones <djonesoa@amazon.com>
1 parent b545964 commit 2089a3b

File tree

25 files changed

+377
-91
lines changed

25 files changed

+377
-91
lines changed

Cargo.lock

Lines changed: 130 additions & 27 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

mountpoint-s3-client/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
### Other changes
44

5+
* Add support for CRC64-NVME checksum algorithm. ([#1235](https://github.com/awslabs/mountpoint-s3/pull/1235))
56
* Add support for overriding the number of threads used by the Event Loop.
67
([#1240](https://github.com/awslabs/mountpoint-s3/pull/1240/))
78
* The ECS credentials provider now performs retries in the event of some failures. ([awslabs/aws-c-auth#259](https://github.com/awslabs/aws-c-auth/pull/259/))

mountpoint-s3-client/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ rand_chacha = { version = "0.3.1", optional = true }
3838
[dev-dependencies]
3939
aws-config = "1.5.10"
4040
aws-credential-types = "1.2.1"
41-
aws-sdk-s3 = "1.65.0"
41+
aws-sdk-s3 = "1.69.0"
4242
aws-smithy-runtime-api = "1.7.3"
4343
bytes = "1.9.0"
4444
clap = { version = "4.5.23", features = ["derive"] }

mountpoint-s3-client/src/checksums.rs

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,26 @@
11
//! Provides base64 encoding/decoding for various checksums.
22
pub use mountpoint_s3_crt::checksums::crc32::{self, Crc32};
33
pub use mountpoint_s3_crt::checksums::crc32c::{self, Crc32c};
4+
pub use mountpoint_s3_crt::checksums::crc64nvme::{self, Crc64nvme};
45
pub use mountpoint_s3_crt::checksums::sha1::{self, Sha1};
56
pub use mountpoint_s3_crt::checksums::sha256::{self, Sha256};
67

78
use base64ct::Base64;
89
use base64ct::Encoding;
910
use thiserror::Error;
1011

12+
/// The base64 encoding for this CRC64-NVME checksum value.
13+
pub fn crc64nvme_to_base64(checksum: &Crc64nvme) -> String {
14+
Base64::encode_string(&checksum.value().to_be_bytes())
15+
}
16+
17+
/// Create a CRC64-NVME checksum from a base64 encoding.
18+
pub fn crc64nvme_from_base64(base64_str: &str) -> Result<Crc64nvme, ParseError> {
19+
let mut dec_buf = [0u8; std::mem::size_of::<u64>()];
20+
let _ = Base64::decode(base64_str, &mut dec_buf)?;
21+
Ok(Crc64nvme::new(u64::from_be_bytes(dec_buf)))
22+
}
23+
1124
/// The base64 encoding for this CRC32C checksum value.
1225
pub fn crc32c_to_base64(checksum: &Crc32c) -> String {
1326
Base64::encode_string(&checksum.value().to_be_bytes())
@@ -55,6 +68,20 @@ mod tests {
5568
use super::*;
5669
use test_case::test_case;
5770

71+
#[test]
72+
fn test_crc64nvme_to_base64() {
73+
let crc = Crc64nvme::new(0xAE8B14860A799888);
74+
let base64 = crc64nvme_to_base64(&crc);
75+
assert_eq!(&base64, "rosUhgp5mIg=");
76+
}
77+
78+
#[test]
79+
fn test_crc64nvme_from_base64() {
80+
let base64 = "rosUhgp5mIg=";
81+
let crc = crc64nvme_from_base64(base64).expect("parsing should succeeed");
82+
assert_eq!(crc.value(), 0xAE8B14860A799888);
83+
}
84+
5885
#[test]
5986
fn test_crc32c_to_base64() {
6087
let crc = Crc32c::new(1234);

0 commit comments

Comments
 (0)