Remote Attestation Add-On for OpenFL

[anshubit5516]

Motivation

Even though they can run in TEE-ready containers, in the current implementation of OpenFL, both collaborators and aggregators implicitly trust each other (based on their TLS certificates). This lack of explicit trust verification of the FL workloads leaves the system vulnerable to potential attacks by malicious users or entities that could compromise the privacy of the model or the data.

To address this vulnerability, attestation becomes a critical requirement. Attestation plays a critical role in federated learning by verifying the integrity and authenticity of the software and hardware involved. By ensuring that both aggregators and collaborators are trusted, we can protect sensitive data and models from malicious actors. This verification process is essential for maintaining a secure and trusted environment, which is crucial for the confidentiality and integrity of the federated learning workflow. Attestation significantly reduces the risk of attacks and ensures that all participants adhere to elevated trust and security standards.

Additional background:

Confidential Federated Learning with OpenFL

The proposal involves enhancing the existing OpenFL topology by incorporating a remote attestation process. This additional trust and security layer will further verify the integrity of the software and hardware of the participants involved in federated learning.

Components

Aggregators: Usually operated by the model owner, it collects model updates from other participants (collaborators) to build an aggregate model without directly accessing the collaborators’ private data.

Collaborators: Collaborator nodes are operated by data owners, to train the model on local data, sending intermediate results (weight updates) to the aggregator.

Intel Trust Authority (ITA): A remote attestation service operated by Intel, which can be used to confirm the integrity and trustworthiness of each participant’s TEE (software + hardware). ITA returns an independently verifiable attestation certificate if the enclave is valid against policies defined by the federation.

End to End Attestation flow:

The main methods for (remote) workload attestation include Passport Attestation and Background attestation. Each has its own advantages and disadvantages. We will look into both strategies and decide on which suits best for our case.

In the proposed design we extend the secured gRPC over mTLS connection to perform attestation between aggregators and collaborators. We assume that each participant is running in an SGX environment.

NB! The attestation flow feature will be controlled by FL plan flag : enable_remote_attestation, for example under the network section:

network:
  settings:
    # ...
    tls: true
    enable_remote_attestation: true

Option 1: Passport Model

In this approach, each node interacts with the attestation service to generate a remote attestation certificate (“passport”) based on their local SGX quote.

• Generate a Local SGX Quote:
  The collaborator generates its own SGX quote and obtains a token from the Intel Trust Authority (ITA).

• Share remote attestation Token with Aggregator (via secure gRPC over mTLS channel):

  The collaborator shares its ITA token with the aggregator and requests the aggregator's ITA token.
  Aggregator can check the token’s validity via the ITA verification service, or autonomously by validating the token against an ITA-issued public key
  If verification fails, aggregator terminates the connection, as it does not trust the collaborator
  Otherwise, the Aggregator gets its own enclave token from ITA and share with collaborators for mutual attestation.

• Verify the Aggregator's Token:

  Upon receiving the aggregator's ITA token, the collaborator verifies it (i.e. using the ITA verification service).

• Verification Outcome:
  If the verification fails, the collaborator terminates the gRPC connection and exits the network, as the aggregator cannot be trusted.
  If the verification succeeds, the collaborator continues with task execution.
  NB! At this stage the aggregator and the collaborator trust each other based on their remote attestation certificates.

Pros:

Simplicity:
Passport attestation is straightforward and easy to implement. It involves presenting a certificate or token that proves the entity's identity and trustworthiness.

Speed:
The process is typically faster since it relies on pre-issued certificates or tokens, reducing the need for real-time verification.

Offline Capability:
Passport attestation can be performed offline if the certificates or tokens are pre-issued and stored locally.

Reduced Load on Attestation Service:
Since the attestation is done using pre-issued credentials, it reduces the load on the attestation service, making it more scalable and reducing usage costs.

Cons:

Static Trust:
The trust is static and based on the validity of the certificate or token. If the certificate is compromised, the trust is also compromised until the certificate is revoked.

Revocation Challenges:
Revoking a compromised certificate or token can be challenging and may not be immediate, leaving a window of vulnerability. This can be mitigated by using the ITA token verification service.

Limited Flexibility:
Passport attestation may not provide real-time verification of the entity's current state or behaviour, limiting its flexibility in dynamic environments.

Option 2: Background Model (Recommended)

In this approach, participants share their local SGX quotes, whereas the relying party (other participants) interact with the attestation service to generate a remote attestation report.

Generate Self Quote:
The collaborator generates its own SGX quote and shares it with the aggregator over a secure gRPC channel, requesting the aggregator's SGX quote.

Receive and Verify Collaborators Quote:
Upon receiving the collaborator’s quote, aggregator verifies it with ITA. If verification fails, Aggregator terminates gRPC channel.

Verification of Aggregators Quote:
Upon receiving the aggregators quote, collaborator verifies with ITA.

Verification Outcome:
If the verification fails, the collaborator terminates the gRPC connection and exits the network.
If the verification succeeds, the collaborator continues with task execution.

Note: Currently in this solution for simplicity we added only ITA for verification of quote. User can also use dcap library to attest of sgx quote.

Pros:

Dynamic Trust:
Background attestation provides real-time verification of the entity's current state, ensuring that the entity is trustworthy at the time of interaction.
Enhanced Security:
By continuously verifying the entity's state, background attestation reduces the risk of compromised entities participating in the system.
Immediate Revocation:
If an entity is found to be compromised, it can be immediately revoked, reducing the window of vulnerability.
Flexibility:
Background attestation can adapt to changing conditions and provide more granular control over the trustworthiness of entities.

Cons:

Complexity:
Implementing background attestation is more complex and requires continuous monitoring and verification of entities.
Performance Overhead:
The continuous verification process can introduce performance overhead, potentially impacting the system's overall performance.
Dependency on Attestation Service:
Background attestation relies on the availability and reliability of the attestation service, making it a potential single point of failure.
Increased Load on Attestation Service:
The continuous verification process increases the load on the attestation service, requiring more resources and scalability considerations.

Technical Details

The Trust Authority (TA) provides two APIs for generating and verifying attestation tokens:
/appraisal/v1/attest:

• This endpoint is used to attest an SGX quote, verifying its authenticity and generating an attestation token. 
  

Example :

/faithful-verification/v1/token-audit-report:

• This endpoint is used to verify the attestation token, ensuring that it is valid and has not been tampered with. 
  

Error Handling

SGX Quote Generation Error:
Description:
If an error occurs during the generation of the SGX quote, it is critical to handle it immediately.
Action:
Throw an exception and exit the process. Without a valid SGX quote, the attestation process cannot proceed, and continuing would compromise the integrity of the system.

ITA Attestation Error:
Description:
Errors during the ITA (Intel Trust Authority) attestation process can occur due to various reasons, such as invalid quotes or ill formed quote.
Action:
Terminate the TLS channel and exit the federation. This ensures that no untrusted entities can participate in the federated learning process.

ITA Not Available:
Description:
If the ITA service is unavailable, the attestation process cannot proceed, which is crucial for maintaining the security of the federation.
Action:
Implement a retry mechanism to keep attempting the attestation. Without a valid attestation token, the flow cannot proceed, and retrying ensures that temporary issues do not halt the process permanently.

Alternatives Considered

In the previous design, attestation was performed using an existing mTLS gRPC channel to exchange quotes and attested tokens. While this approach was convenient, it had a significant drawback: the TLS keys have been generated outside of the enclave.

To address this issue, we propose the following solution:

1. On boot-up, aggregators and collaborators will generate its tls key pair.
2. Aggregator will start a REST server
3. Collaborators will generate its sgx quote and add its public tls key in user data section of SGX quote.
4. Collaborators will do a POST call to exchange its quotes and validates each other identity
5. Aggregator on receiving collaborators quote, will do attestation of the quote via ITA or dcap library. If validation succeeds, then it adds that collaborators public tls key.
6. Aggregator will start gRPC server with its keys
7. Collaborators on receiving aggregators quote will validate via ITA or dcap lib. If validation succeeds, then collaborator adds aggregators public tls key.
8. Collaborators who have successfully exchanged quotes can now request mTLS gRPC connection to aggregator.

Pros:
OpenFl participants only establishes connection if they have attested each other's identity.
Keys are generated inside enclave, hence more secured.

Cons:
Currently in OpenFL, no REST support is available.

Risks and Mitigations

In both, Passport/Background attestation model, ITA becomes a risk of being single point of failure.

Possible Mitigation:
Passport model: We can introduce ‘KID’ based attestation which can help in verifying the token.
Background model: Since we have the quote, we can try for DCAP attestation.

Appendices:

Data Center Attestation Primitives Attestation (DCAP):

DCAP attestation is a mechanism used in Intel SGX (Software Guard Extensions) to verify the integrity and trustworthiness of an enclave (a secure execution environment). Unlike EPID-based attestation, DCAP does not rely on Intel's Attestation Service (IAS). Instead, it allows enterprises to perform attestation locally using a Provisioning Certification Service (PCS) provided by Intel.

How DCAP Attestation Works

1. Enclave Generates a Quote:

• The enclave generates a quote (a signed structure) containing information about its identity, measurements, and other metadata.
• This quote is signed using a Provisioning Certification Key (PCK), which is tied to the platform's hardware.

2. Quote Verification:

• The quote is sent to a Verifier (e.g., an enterprise's attestation service). In case of SGX, intel provides libraries in various commonly used languages.
• The verifier checks the quote's validity by:
  1. Verifying the signature using the PCK.
  2. Ensuring the PCK is valid by checking its certificate chain against Intel's Provisioning Certification Service (PCS).

3. Certificate Chain Validation:

• The verifier fetches the required certificates (PCK Certificate, CRLs, and TCB Info) from Intel's PCS.
• It validates the certificate chain to ensure the platform is trusted and up to date with the latest security patches.

4. Trust Decision:

• If the quote and certificate chain are valid, the verifier concludes that the enclave is trustworthy.
• The verifier can then allow the enclave to perform sensitive operations.

Advantages of DCAP

Local Attestation:
No dependency on Intel's IAS for attestation. Though, it needs to download time to time latest PCK certificates, CRLs, TCB information. But in real time verification, mostly cached information is used, if not staled.

Scalability:
Suitable for large-scale deployments in data centers.

Flexibility:
Enterprises can implement their own attestation policies.

KID Verification:

KID (Key ID) Verification is a process used in Intel SGX attestation to ensure the authenticity and validity of a Provisioning Certification Key (PCK) certificate. The KID is a unique identifier associated with the PCK certificate, and it helps in verifying that the certificate belongs to a specific platform and is issued by Intel's Provisioning Certification Service (PCS).

How KID Verification Works

1. KID in the Quote:

• When an SGX enclave generates a quote, it includes the KID of the PCK certificate used to sign the quote.

2. Fetching the PCK Certificate:

• The verifier uses the KID to fetch the corresponding PCK certificate from Intel's PCS or a local cache.

3. Certificate Validation:

• The verifier validates the PCK certificate by:
  1. Checking its signature against Intel's root CA.
  2. Ensuring the certificate is not revoked (using CRLs).
  3. Verifying the certificate's validity period.

4. Quote Verification:

• Once the PCK certificate is validated, the verifier uses it to verify the signature on the enclave's quote.

Purpose of KID Verification
1. Ensures the PCK certificate used in attestation is authentic and issued by Intel.
2. Links the quote to a specific platform's hardware.
3. Prevents tampering or misuse of invalid certificates.

Key Benefits
Efficiency:
The KID allows quick lookup of the PCK certificate, reducing overhead.

Security:
Ensures only trusted certificates are used for attestation.

Scalability:
Simplifies certificate management in large-scale deployments.

[teoparvanov]

@anshubit5516, as discussed offline, please add more details on how the remote attestation feature will be enabled via the FL plan (f.e. via a remote-attestation boolean flag).

[anshubit5516]

I have added a FL plan flag : enable_remote_attestation and updated the sequence diagrams.

[rahulga1]

@anshubit5516 Thanks for design, overall looks good to me. Two small comments:

1. In the diagram shouldnt it be Model Owner instead of Model Builder. We dont have anything like builder in OpenFL as per my understanding.
2. how your design will be impacted if we move from gRPC to REST, as thats the plan in near future?

[anshubit5516]

@rahulga1

1. I used the similar image used for Teo in his blog, so used Model Builder rather than owner.
2. Overall attestation will happen in similar way, but sure some changes will be there. As I have shown in alternate design, we can easily exchange quotes using REST as well, and then only training process will start.

In REST, based on its design, lest say it maintains session, then before creating session, we must do attestation.
If its stateless, then it must maintain api-key authorization way, meaning, 1st collaborators needs to exchange its quote/token and each collaborators and aggregators will use lets say hash of sgx token received and use it as api-key for authorization.

So we can easily handle attestation if we moved from gRPC to REST.
please let me know if you need any clarification.

[payalcha]

@anshubit5516 where ITA details will be added, will it be added in any section of plan.yaml and circulated by model owner to all participants.
During restart of service due to any reason, similar attestations will be required?

[anshubit5516]

@payalcha
Can you please clarify about what details of ITA. if you mean ITA's url and api token, it can be set as env variable by the user which will be used to by collaborators as well as aggregators.

Yes, attestation will always occur whenever connection is established meaning start/reconnect/restart.

[teoparvanov]

We have reached the following agreement during the design review:

1. Take a semi-manual "passport" approach where an enclave private key and a remote attestation report are generated on each participant with capable SGX hardware, and shared out of band. They can then be independently verified by any relying party (such as a data owner or a model builder). Based on this manual verification, they can then decide to join or not the federation, but once they do - there will be no automatic attestation report verifications as suggested in the original design discussion. This offers a more decentralized approach, and also enables "hybrid" federations, where not all participants have TEE-capable hardware.
2. The enclave private key and the attestation report generation can be implemented as callbacks for reuse in downstream projects.
3. A document shall be added, describing the manual attestation verification process.

The semi-manual approach described in this comment has been approved for implementation in OpenFL 1.9.

CC: @psfoley

[teoparvanov]

Some additional details on the agg/coll startup process (if SGX is available on the host):

1. Within the enclave, a private key and a public cert are generated
2. The enclave fetches a local SGX quote
3. An attestation report is requested from ITA, based on the local SGX quote
4. The enclave's public cert and ITA report are saved in a bundle for out-of-band transmission to relying parties
5. A hook (callback) is invoked for post-processing of the attestation bundle in downstream components

Relying party verification (manual):

1. Retrieve the attestation bundle (enclave cert and ITA attestation report)
2. Get the reference MRENCLAVE value, calculated when building the workspace image - see here
3. Check the ITA attestation report's validity against a trusted ITA certificate
4. Decode the ITA report, and compare the MRENCLAVE from the ITA report to the expected one provided by the model builder in step 2.

Note: there is a limitation of this process where only the Aggregator's attestation can be practically verified before a collaborator starts and joins the federation. This is because when the Collaborator starts, it will automatically connect to the Aggregator and start processing tasks.