Add build option for exposing self-test failure messages #433

WillChilds-Klein · 2025-02-17T16:44:54Z

Issue #, if available: ACCP-129

THIS FEATURE IS NOT INTENDED FOR PRODUCTION AND IS THUS LEFT UNDOCUMENTED IN OUR README

Notes

This PR builds on prior work from @amirhosv on the fips-experimentation branch to provide an alternative mode of handling AWS-LC self test failures. We call this (non-default) mode FIPS_SELF_TEST_SKIP_ABORT, and it is only usable when ACCP is built in FIPS mode.

By default, AWS-LC will abort() on self-test failures. However, as of AWS-LC v1.47, when built with the AWSLC_FIPS_FAILURE_CALLBACK build flag AWS-LC will call a weak symbol AWS_LC_fips_failure_callback function to handle self test failures instead of aborting. When ACCP is built with -DFIPS_SELF_TEST_SKIP_ABORT, ACCP defines AWS_LC_fips_failure_callback such that it appends to a queue of error strings in ACCP's native heap. To manage the accumulated error strings, we add a native std::vector wrapper ConcurrentStringVector providing a minimal, threadsafe API. Once the error queue is non-empty, all subsequent getInstance calls on an algorithm provided by ACCP will throw FipsStatusException.

We provide two functions for callers to query fips self test error state on an AmazonCorrettoCryptoProvider instance:

public boolean isFipsStatusOk()
public List<String> getFipsSelfTestFailures()

We could get away with only the latter function, but we provide isFipsStatusOk() to avoid performance costs of copying error strings over the JNI.

Testing

To adequately test the new mode, we need to build AWS-LC with FIPS_BREAK_TEST=TESTS to programmatically break AWS-LC's pairwise consistency tests (PCTs). We test against each available PCT breakage during key generation by setting the appropriate environment variable, indicating which PCT to break. Unfortunately, Java doesn't appear to have a standard utility for manipulating process environment variables at runtime, so we create our own TestUtil.setEnv that calls POSIX setenv/unsetenv over the JNI.

In addition to CI tests, I've also executed run_accp_basic_tests.sh with the new --fips-self-test-failure-no-abort flag, which will eventually be incorporated into our GitHub CI.

$ TEST_JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 ./tests/ci/run_accp_basic_tests.sh --fips-self-test-failure-no-abort
...
BUILD SUCCESSFUL in 40m 15s
18 actionable tasks: 13 executed, 5 up-to-date

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

* `AWS_LC_fips_failure_callback` is a method invoked by AWS-LC in case a FIPS related check fails * ACCP updates a global flag to false in case this callback is invoked * The value of the global flag is returned as the result of `isFipsStatusOk` * ACCP built in FIPS mode checks `isFipsStatusOk` whenever a crypto service is requested

Co-authored-by: Will Childs-Klein <[email protected]> Co-authored-by: Gerardo Ravago 🇵🇭 <[email protected]>

This reverts commit dea1db2.

geedo0 · 2025-03-06T16:28:58Z

src/com/amazon/corretto/crypto/provider/AmazonCorrettoCryptoProvider.java

+  private native int fipsStatusErrorCount();
+
+  /**
+   * @return true if and only if the underlying libcrypto library's FIPS related checks pass


How do you guarantee that there are in fact 0 errors rather than we simply haven't run the self-test yet? There's no logic here to guard against returning a false-positive from an untested module.

We run AWS-LC's self-tests when ACCP's INSTANCE is constructed in a java static block. Those self-tests get executed in a threadpool, so there's a very slight potential for a race, but we do run AWS-LC self-tests immediately on load.

~~Ooh, I guess we could checkTests() here, but that might be expensive.~~

EDIT: no, this is a function in ACCPService, not AmazonCorrettoCryptoProvider...

~~Oh, even better, I can just check testsPassed!~~

EDIT: no, this is a function in ACCPService, not AmazonCorrettoCryptoProvider...

I ended up going with getSelfTestStatus().

@geedo0 -- looking into this a little more, this behavioral change will break callers (and our tests) when SkipAbort is enabled. we now check isFIpsStatusOk() in every call to ACCPService.newInstance, so unless the caller busy-polls getSelfTestStatus() != SelfTestStatus.NOT_RUN before using the provider, calls to newInstance will throw an exception until the power-on self tests have completed.

we could busy-poll that status in this method, and maybe that's OK because this is a very specialized build option, but that seems like a recipe for strange behavior. but maybe it's favorable in order to preserve current semantics? removing this check basically changes the semantics of true in this method from "self-tests have affirmatively passed" to "no self-test failures have occurred".

i removed this check in 34faca36e, but perhaps busy-polling in isFipsStatusOk is the better approach.

please let me know what you think.

Talked offline with @geedo0 and we convinced ourselves that busy polling in isFipsStatusOk is the way to go so we can guarantee we don't provide any crypto algorithms until the self tests have affirmatively passed.

Implemented busy poll with a deadline in 6d9a71a0c.

A more general solution would be to block the main thread here, but there's a scary comment advising against that. I'll do some experiments out-of-band of this PR.

geedo0 · 2025-03-06T16:35:44Z

src/com/amazon/corretto/crypto/provider/Loader.java

@@ -157,6 +159,7 @@ static String getProperty(String propertyName, String def) {
    PROVIDER_VERSION = oldVersion;
    FIPS_BUILD = available && isFipsMode();
    EXPERIMENTAL_FIPS_BUILD = available && isExperimentalFipsMode();
+    FIPS_SELF_TEST_SKIP_ABORT = available && isFipsSelfTestFailureSkipAbort();


Logic here is confusing me and a bit concerning. Why should all these be be false if the module is unavailable? I worry that a library load failure might break our assertions and slip through the detection cracks of failing loudly. It's happened before with our customers.

I was pattern-matching here, so can't speak to original intent. I think it might be reasonable to throw a RuntimeException if library loading fails, but that seems out of scope for this change.

csrc/loader.cpp

csrc/fips_status.cpp

build.gradle

alexw91

Just a small nit, otherwise LGTM

src/com/amazon/corretto/crypto/provider/AmazonCorrettoCryptoProvider.java

@amirhosv

*Issue #, if available:* ACCP-129 _**THIS FEATURE IS NOT INTENDED FOR PRODUCTION AND IS THUS LEFT UNDOCUMENTED IN OUR README**_ ## Notes This PR builds on prior work from @amirhosv on the `fips-experimentation` branch to provide an alternative mode of handling AWS-LC self test failures. We call this (non-default) mode `FIPS_SELF_TEST_SKIP_ABORT`, and it is only usable when ACCP is built in FIPS mode. By default, AWS-LC will `abort()` on self-test failures. However, as of AWS-LC v1.47, when built with the `AWSLC_FIPS_FAILURE_CALLBACK` build flag AWS-LC will [call][1] a [weak symbol][2] `AWS_LC_fips_failure_callback` function to handle self test failures instead of aborting. When ACCP is built with `-DFIPS_SELF_TEST_SKIP_ABORT`, ACCP defines `AWS_LC_fips_failure_callback` such that it appends to a queue of error strings in ACCP's native heap. To manage the accumulated error strings, we add a native `std::vector` wrapper `ConcurrentStringVector` providing a minimal, threadsafe API. Once the error queue is non-empty, all subsequent `getInstance` calls on an algorithm provided by ACCP will throw `FipsStatusException`. We provide two functions for callers to query fips self test error state on an `AmazonCorrettoCryptoProvider` instance: - `public boolean isFipsStatusOk()` - `public List<String> getFipsSelfTestFailures()` We could get away with only the latter function, but we provide `isFipsStatusOk()` to avoid performance costs of copying error strings over the JNI. [1]: https://github.com/aws/aws-lc/blob/1d8b807ed1ae75c89beda6c73a4ae27c404fa46f/crypto/fipsmodule/bcm.c#L416 [2]: https://github.com/aws/aws-lc/blob/1d8b807ed1ae75c89beda6c73a4ae27c404fa46f/crypto/internal.h#L1427-L1432 ## Testing To adequately test the new mode, we need to build AWS-LC with `FIPS_BREAK_TEST=TESTS` to programmatically break AWS-LC's pairwise consistency tests (PCTs). We test against each available PCT breakage during key generation by setting the appropriate environment variable, indicating which PCT to break. Unfortunately, Java doesn't appear to have a standard utility for manipulating process environment variables at runtime, so we create our own `TestUtil.setEnv` that calls POSIX `setenv`/`unsetenv` over the JNI. In addition to CI tests, I've also executed `run_accp_basic_tests.sh` with the new `--fips-self-test-failure-no-abort` flag, which will eventually be incorporated into our GitHub CI. ``` $ TEST_JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 ./tests/ci/run_accp_basic_tests.sh --fips-self-test-failure-no-abort ... BUILD SUCCESSFUL in 40m 15s 18 actionable tasks: 13 executed, 5 up-to-date ``` --- By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Amir Vakili <[email protected]> Co-authored-by: Amir Vakili <[email protected]> Co-authored-by: Gerardo Ravago 🇵🇭 <[email protected]>

amirhosv and others added 14 commits March 31, 2023 15:36

Break self tests by modifying AWS-LC

71882bc

Updating fips-experimentation branch with main (corretto#294)

d16e616

Co-authored-by: Will Childs-Klein <[email protected]> Co-authored-by: Gerardo Ravago 🇵🇭 <[email protected]>

Revert "Only register LibCryptoRng by default in non-FIPS mode"

f274433

This reverts commit dea1db2.

Use FIPS approved APIs to generate RSA key pairs (corretto#301)

e006dee

Merge branch 'main' into fips-experimentation

355b0dd

Fix spotless

893b388

Remove obsolete MacOS CI jobs

6c7d677

Merge branch 'fips-experimentation'

7c3672a

Clean up merge

9bb0c6d

Merge remote-tracking branch 'aws/main' into self-test-failure

9a05c8e

Add getFipsStatusError method + JNI helpers

9be07aa

Cross-thread access caused JVM heap issues, use native error queue

72a0568

Duct tape for the temp AWS-LC dep hack

427ee31

WillChilds-Klein changed the title ~~[DRAFT] Self test failure~~ [DRAFT] Add build option for exposing self-test failure messages Feb 17, 2025

WillChilds-Klein force-pushed the self-test-failure branch from d3b69d5 to 3a8643d Compare February 18, 2025 00:11

Execute FipsStatusTest on main test thread

0323a8d

WillChilds-Klein force-pushed the self-test-failure branch from 3a8643d to 0323a8d Compare February 18, 2025 00:25

WillChilds-Klein added 12 commits February 18, 2025 10:39

Add explanatory note for ibid.

0c5e4b5

Update AWS-LC version

0cdac70

Merge remote-tracking branch 'aws/main' into self-test-failure

50c61fe

Temporarily skip ML-DSA tests

d536459

Appease spotless

599d1ae

Remove break-tests example

10187b1

Update README and CHANGELOG

5183a70

Put callback stuff behind ACCP build flag

f1e2e62

Appease spotless, check isFipsSelfTestFailureNoAbort in getInstance

2a40997

Update relevant test scripts tests/ci/run_accp_* t handle new flag

d2b5953

Specify ALLOW_FIPS_TEST_BREAK in CI test scripts

ea7c873

Fix AWS-LC callback build flag plumbing, test PWCT breakage

62ef27f

geedo0 reviewed Mar 6, 2025

View reviewed changes

PR feedback

10503bd

WillChilds-Klein requested review from geedo0 and andrewhop March 6, 2025 21:30

Include vector header in fips_status.cpp

1129dc3

geedo0 previously approved these changes Mar 6, 2025

View reviewed changes

csrc/fips_status.cpp Show resolved Hide resolved

Fix vector ctor call

4caf24b

WillChilds-Klein dismissed geedo0’s stale review via 4caf24b March 6, 2025 21:41

WillChilds-Klein requested a review from geedo0 March 6, 2025 23:51

geedo0 reviewed Mar 7, 2025

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

geedo0 previously approved these changes Mar 7, 2025

View reviewed changes

Disable EdDSA PWCT, block until self-tests finish

f3c1fd5

WillChilds-Klein dismissed geedo0’s stale review via f3c1fd5 March 7, 2025 16:13

WillChilds-Klein added 7 commits March 7, 2025 16:30

Remove self test status check from isFipsStatusOk()

b34faca

Fix non-FIPS

25eac43

Appease spotless

ef0ed12

Busy poll self test status in isFipsStatusOk

6d9a71a

Fix conflict

aedf5be

Appease spotless

ad531ff

Add back Ed25519 PWCT tests with temporary guards

ffd6bc7

WillChilds-Klein requested a review from geedo0 March 7, 2025 18:41

WillChilds-Klein enabled auto-merge (squash) March 7, 2025 20:55

alexw91 requested changes Mar 7, 2025

View reviewed changes

src/com/amazon/corretto/crypto/provider/AmazonCorrettoCryptoProvider.java Outdated Show resolved Hide resolved

Throw RuntimeCryptoException on self-test timeout

4ce2164

geedo0 approved these changes Mar 7, 2025

View reviewed changes

alexw91 approved these changes Mar 7, 2025

View reviewed changes

WillChilds-Klein merged commit 3abb411 into corretto:main Mar 7, 2025
11 checks passed

WillChilds-Klein deleted the self-test-failure branch March 7, 2025 22:23

WillChilds-Klein mentioned this pull request Mar 12, 2025

PR 433 follow ups #441

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add build option for exposing self-test failure messages #433

Add build option for exposing self-test failure messages #433

WillChilds-Klein commented Feb 17, 2025 •

edited

Loading

geedo0 Mar 6, 2025

WillChilds-Klein Mar 6, 2025

WillChilds-Klein Mar 6, 2025 •

edited

Loading

WillChilds-Klein Mar 6, 2025 •

edited

Loading

WillChilds-Klein Mar 6, 2025

WillChilds-Klein Mar 7, 2025

WillChilds-Klein Mar 7, 2025 •

edited

Loading

geedo0 Mar 6, 2025

WillChilds-Klein Mar 6, 2025

alexw91 left a comment

Add build option for exposing self-test failure messages #433

Add build option for exposing self-test failure messages #433

Conversation

WillChilds-Klein commented Feb 17, 2025 • edited Loading

Notes

Testing

geedo0 Mar 6, 2025

Choose a reason for hiding this comment

WillChilds-Klein Mar 6, 2025

Choose a reason for hiding this comment

WillChilds-Klein Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

WillChilds-Klein Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

WillChilds-Klein Mar 6, 2025

Choose a reason for hiding this comment

WillChilds-Klein Mar 7, 2025

Choose a reason for hiding this comment

WillChilds-Klein Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

geedo0 Mar 6, 2025

Choose a reason for hiding this comment

WillChilds-Klein Mar 6, 2025

Choose a reason for hiding this comment

alexw91 left a comment

Choose a reason for hiding this comment

WillChilds-Klein commented Feb 17, 2025 •

edited

Loading

WillChilds-Klein Mar 6, 2025 •

edited

Loading

WillChilds-Klein Mar 6, 2025 •

edited

Loading

WillChilds-Klein Mar 7, 2025 •

edited

Loading