RANGER-5175: Functional Test Case Support for KMS API and HDFS Encryption #547
+3,188
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spolavarpau1
requested review from
dineshkumar-yadav,
mneethiraj,
rameeshm,
kulkabhay,
dhavalshah9131,
kumaab and
fateh288
…in 'test_keyOps.py'
There was a problem hiding this comment.
Pull Request Overview
This PR adds a comprehensive Pytest-based functional test suite for validating KMS REST API endpoints and HDFS encryption workflows, along with a Bash script to orchestrate Docker test environments.
- Introduce
test-pytest.shto automate Docker setup/teardown and run Pytest for both KMS and HDFS tests - Add
test_kms/suite to cover key creation, rollover, metadata, bulk operations, policies, blacklisting, and error-log capture - Add
test_hdfs/suite to cover HDFS encryption zone creation, file operations, cross-zone tests, and audit spooling
Reviewed Changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test-pytest.sh | Bash script to orchestrate Docker environment and run tests |
| PyTest-KMS-HDFS/test_kms/utils.py | Helper to fetch KMS container logs |
| PyTest-KMS-HDFS/test_kms/test_keys.py | Tests key creation, validation, and duplicate scenarios |
| PyTest-KMS-HDFS/test_kms/test_keys_02.py | Tests version increment, material matching, bulk ops |
| PyTest-KMS-HDFS/test_kms/test_keyOps.py | Tests key rollover, DEK/EDEK generation and decryption |
| PyTest-KMS-HDFS/test_kms/test_keyDetails.py | Tests retrieval of key names, metadata, and versions |
| PyTest-KMS-HDFS/test_kms/test_blacklisting.py | Tests blacklisting/unblacklisting policy enforcement |
| PyTest-KMS-HDFS/test_kms/conftest.py | Fixtures for test key setup and teardown |
| PyTest-KMS-HDFS/test_kms/readme.md | Documentation for KMS tests |
| PyTest-KMS-HDFS/test_hdfs/utils.py | Utils to run HDFS commands and capture logs |
| PyTest-KMS-HDFS/test_hdfs/test_encryption.py | End-to-end HDFS encryption zone lifecycle tests |
| PyTest-KMS-HDFS/test_hdfs/test_encryption02.py | Tests post-rollover and post-deletion file access |
| PyTest-KMS-HDFS/test_hdfs/test_encryption03.py | Cross-zone permission tests |
| PyTest-KMS-HDFS/test_hdfs/test_AuditSpooling.py | Tests audit spooling configuration and log files |
| PyTest-KMS-HDFS/test_hdfs/test_config.py | Central constants and HDFS/KMS command templates |
| PyTest-KMS-HDFS/test_hdfs/conftest.py | Fixture for HDFS container setup and config injection |
| PyTest-KMS-HDFS/test_hdfs/readme.md | Documentation for HDFS tests |
| PyTest-KMS-HDFS/pytest.ini | Custom Pytest markers registration |
| PyTest-KMS-HDFS/requirements.txt | Python dependencies for test suite |
| PyTest-KMS-HDFS/readme.md | Top-level project README |
Comments suppressed due to low confidence (4)
test-pytest.sh:5
- Remove the extra space before the comma:
no arguments passed, using default DB: postgres.
echo "no arguments passed , using default DB: postgres"
PyTest-KMS-HDFS/test_hdfs/test_AuditSpooling.py:1
- Filename
test_AuditSpooling.pyuses uppercase letters; rename totest_audit_spooling.pyto follow pytest and project naming conventions.
import pytest
test-pytest.sh:25
- Add spaces around the
[and]in the test:if [ ${#EXTRA_SERVICES[@]} -gt 0 ]; thento avoid syntax errors in Bash.
if [${#EXTRA_SERVICES[@]} -gt 0]; then
PyTest-KMS-HDFS/test_kms/test_keys.py:50
- This line is indented with a single space under the
if; it should be indented to match the block (e.g., 8 spaces) to avoid syntax errors.
requests.delete(f"{BASE_URL}/key/{name}", params=PARAMS)
| @@ -0,0 +1,14 @@ | |||
| #no scope mismatch due to utils | |||
There was a problem hiding this comment.
[nitpick] This comment is unclear—consider clarifying the intent or removing it if it isn't adding meaningful context.
Copilot uses AI. Check for mistakes.
| } | ||
|
|
||
| response = requests.post(BASE_URL_RANGER, auth=RANGER_AUTH, json=policy_data) | ||
| time.sleep(30) # Wait for policy propagation |
There was a problem hiding this comment.
Fixed sleep delays can slow down tests and add flakiness; consider polling the service for readiness instead of a blind 30s wait.
Copilot uses AI. Check for mistakes.
Co-authored-by: Copilot <[email protected]>
Duplicate definition of HDFS_USER has been removed Co-authored-by: Copilot <[email protected]>
Instead of caching all exceptions ,catching only the exceptions that can be expected (e.g subprocess.CalledProcessError) Co-authored-by: Copilot <[email protected]>
…t-pytest.sh TESTS_PATH was holding diff test folder name so changed it And changed Virtual environment creation logic before hitting test cases
… into RANGER-5175-pytest test-pytest.sh file has been changed
Disk space was getting exhausted when CI pipeline ran, so trying --no-cache-dir while pip install -r requirements.txt
…ges in test_hdfs/conftest.py to overcome 409 client error
…r creation before policy creation and increased timeout period in wait_for_hdfs method
…he timeout period
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR involves developing a functional test suite using Pytest to validate the Key Management Service (KMS) REST API endpoints and test HDFS encryption functionalities. The scope includes:
Automating API endpoint tests for key creation, deletion and further operation on keys.
Managing encryption zones in HDFS and validating key association.
Performing read/write operations on encrypted files.
Verifying access control and permissions within encrypted zones.
For further information please refer [PyTest-KMS-HDFS/readme.md]
[PyTest-KMS-HDFS/test_hdfs/readme.md]
[PyTest-KMS-HDFS/test_kms/readme.md]