Verifying SHA-256 of a file

[mdodds-cns]

I'm trying to write my first Hurl (version 5.0.1) script to validate a proxy for a MaxMind API proxy.
The API works as follows:

• You request EDITION and a suffix (e.g. zip or zip.sha256)
• When you retrieve the 'zip' the 'filename' Content-Disposition indicates actual name
• When you retrieve the 'zip.sha256' the file contains the filename & sha256 separated by a space

What I want do to is retrieve the SHA256 file, saved the SHA and then request the actual file and compare checksums.
I wrote this little script:

GET https://{{DOMAIN}}/geoip/databases/GeoLite2-Country-CSV/download?suffix=zip.sha256
HTTP 200

# Extract both the hash and filename from the SHA256 file
[Captures]
expected_sha256: body regex "([a-f0-9]{64})\\s+[^\\s]+"
extracted_filename: body regex "[a-f0-9]{64}\\s+([^\\s]+)"

# Now download the ZIP file using the extracted filename
GET https://{{DOMAIN}}/geoip/databases/GeoLite2-Country-CSV/download?suffix=zip
HTTP 200
[Captures]
file_content: bytes

# Verify Content-Disposition header contains the correct filename
[Asserts]
header "Content-Disposition" contains "filename={{extracted_filename}}"
header "Content-Type" == "application/zip"
sha256 == {{expected_sha256}}

This fails as follows:

(devbox) mdodds@mdsv2:~/build/hurl-test$ hurl --variable DOMAIN=${MAXMIND_PROXY} -u ${MAXMIND_ID}:${MAXMIND_KEY} ~/scripts/test.hurl
error: Assert failure
  --> /home/mdodds/scripts/test.hurl:19:0
   |
   | GET https://{{DOMAIN}}/geoip/databases/GeoLite2-Country-CSV/download?suffix=zip
   | ...
19 | sha256 == {{expected_sha256}}
   |   actual:   byte array <1740e2f2223af1c90a1e91b0e96b04dcaa8523c3bdd0993b1afe27bab9479c4a>
   |   expected: string <1740e2f2223af1c90a1e91b0e96b04dcaa8523c3bdd0993b1afe27bab9479c4a>
   |

Basically I can't work out how to resolve this issue to convert the bytes into a string or vice-versa.

Thank you in advance for any help!

[jcamiel]

Hi @mdodds-cns

Unfortunately I don't think we have implemented what you need!

We have a decode filter that convert bytes to string given an encoding. But that's not exactly what you want, you want a filter that convert hexadecimal representation string to bytes.

What we could do either:

1. Add a new filter fromHex or fromHexString: this way you could do:

expected_sha256: body regex "([a-f0-9]{64})\\s+[^\\s]+" fromHex
...
sha256 == {{expected_sha256}}

2. Or implement toString filter on bytes to convert back to an hexadecimal string representation

expected_sha256: body regex "([a-f0-9]{64})\\s+[^\\s]+"
...
sha256 toString == {{expected_sha256}}

3. Or decode could accept the encoding hex so that we can convert bytes to a hexadecimal string:

expected_sha256: body regex "([a-f0-9]{64})\\s+[^\\s]+"
...
sha256 decode "hex" == {{expected_sha256}}

I quite like the third solution, as it fits into the decode filter that transform bytes to string...

As your use case is a rather common use case, I think we're going to address it soon.

@fabricereix @lepapareil any preferences?

In the meantime, you can still hardcode the sha256 value:

GET https://example.org/data.tar.gz
HTTP 200
[Asserts]
sha256 == hex,039058c6f2c0cb492c533b0a4d14ef77cc0f78abccced5287d84a1a2011cfb81;

[mdodds-cns]

Thanks for investigating this!

It does feel like this would be a useful enhancement for Hurl as this is a common pattern when downloading files.

[jcamiel]

Issue created here => #3963

[jcamiel]

Ho @mdodds-cns

The filter toHex will be available for the next Hurl version 7.0.0, you'll be able to wrote this kind of file:

GET https://{{DOMAIN}}/geoip/databases/GeoLite2-Country-CSV/download?suffix=zip
HTTP 200
[Asserts]
sha256 toHex == {{expected_sha256}}
sha256 toHex == "039058c6f2c0cb492c533b0a4d14ef77cc0f78abccced5287d84a1a2011cfb81"