NC | Lifecycle | GPFS ILM integration

[romayalon]

Describe the Problem

Today, NooBaa NC lifecycle worker is running a directory scan for both POSIX and GPFS file systems.
A method for optimizing the scan on GPFS file system is to generate and apply GPFS ILM policies based on the configured lifecycle configuration.

Explain the Changes

Note - currently we support only expiry + filter
(non current days is not supported yet)
Implementation changes -

1. Changed config.NC_LIFECYCLE_GPFS_ILM_ENABLED to be true by default.
2. Added a directory for storing the ilm policies - const ILM_POLICIES_TMP_DIR = path.join(config.NC_LIFECYCLE_LOGS_DIR, 'lifecycle_ilm_policies') - notice it's inside the lifecycle logs dir which means it's not shared between nodes.
3. Added a directory for storing the candidates files - const ILM_CANDIDATES_TMP_DIR = path.join(config.NC_LIFECYCLE_LOGS_DIR, 'lifecycle_ilm_candidates'); - notice it's inside the lifecycle logs dir which means it's not shared between nodes.
4. Added the following new functions for GPFS ILM policies usage -

• create_gpfs_candidates_files() - creates a candidates file per mount point that is used by at least one bucket
  • 1. creates a map of mount point to buckets
  • 2. for each bucket -
  • 2.1. finds the mount point it belongs to
  • 2.2. convert the bucket's lifecycle policy to a GPFS ILM policy
  • 2.3. concat the bucket's GPFS ILM policy to the mount point policy file string
  • 3. for each mount point -
  • 3.1. writes the ILM policy to a tmp file
  • 3. creates the candidates file by applying the ILM policy
• get_candidates_by_expiration_rule_gpfs() - does the following -
  • 1. parses the candidates from the ILM candidates file
• convert_lifecycle_policy_to_gpfs_ilm_policy() - converts the base definition + path search and then calls the next 2 functions for converting the filter and expiry lifecycle configuration properties to ILM policy.
• convert_expiry_rule_to_gpfs_ilm_policy ()
• convert_filter_to_gpfs_ilm_policy()
• write_tmp_ilm_policy() - writes the generated ilm policy to a tmp file under ILM_POLICIES_TMP_DIR.
• get_candidates_by_gpfs_ilm_policy() - applies the policy by using mmapplypolicy - this command will generate the candidates file under ILM_CANDIDATES_TMP_DIR.
• parse_candidates_from_gpfs_ilm_policy() - reads 1000 lines of the ILM candidates file, and if the rule state has a candidates_file_offset and it's not finished it will restaet reading from it.

Issues: Fixed #xxx / Gap #xxx

1. Directory objects tests/nested objects - implemented and checked
2. Check --continue implementation after NC | lifecycle | continue last run #8925 will get merged - according to Nadav it should work
3. Gap - optimization on GPFS ILM policy - instead of mountpoint it's better to use fileset (the lowest common fileset of buckets on the same file system)
4. Gap - noncurrent_version_expiration
5. Gap - Content dir size 0 optimization will not be found by these code changes

Testing Instructions:

1. sudo jest --testRegex=jest_tests/test_nc_lifecycle_gpfs
   Manual tests -
2. Create a directory for storage -mkdir /mnt/gpfs0/romy/
3. Create an account - `noobaa-cli account add --name account1 --user root --new_buckets_path=/mnt/gpfs0/romy/
4. Create a directory for objects stored in the bucket bucket1 - mkdir /mnt/gpfs0/romy/bucket1_storage/
5. Create a bucket - noobaa-cli bucket add --name bucket1 --owner account1 --path /mnt/gpfs0/romy/bucket1_storage/
6. Install AWS CLI -

sudo dnf install unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

10. Create an alias to the S3 command - alias s3-nb='AWS_ACCESS_KEY_ID=<access_key> AWS_SECRET_ACCESS_KEY=<secret_key> aws --endpoint https://localhost:6443 --no-verify-ssl'
11. Create a lifecycle policy file -
    vi lifecycle.json -

{
    "Rules": [
        {
            "ID": "Expiration Rule",
            "Status": "Enabled",
            "Filter": {
                "Prefix": "a"
            },
            "Expiration": {
                "Date": "2022-07-12"
            }
        }
    ]
}

7. Put the policy on bucket1 - s3-nb s3api put-bucket-lifecycle-configuration --bucket bucket1 --lifecycle-configuration file://lifecycle.json
8. Check that the policy was configured - s3-nb s3api get-bucket-lifecycle-configuration --bucket bucket1
9. Create objects -

echo "mdalmdlamdlakd" > /mnt/gpfs0/romy/bucket1_storage/a.txt
echo "mdalmdlamdlakd" > /mnt/gpfs0/romy/bucket1_storage/a1.txt
echo "mdalmdlamdlakd" > /mnt/gpfs0/romy/bucket1_storage/a2.txt
echo "mdalmdlamdlakd" > /mnt/gpfs0/romy/bucket1_storage/b.txt

14. Run lifecycle worker manually - noobaa-cli lifecycle --disable_runtime_validation
15. Check that the ilm policy was created under /var/log/noobaa/lifecycle/lifecycle_ilm_policies/
16. Check that a candidates file was created under /var/log/noobaa/lifecycle/lifecycle_ilm_candidates/
17. Check that files starting with 'a' where deleted and the file starting with 'b' was not deleted.

• Doc added/updated
• Tests added

[nadavMiz]

what if we have both expiration and noncurrent_version_expiration?

[romayalon]

on this PR I don't plan to support noncurrent_version_expiration really, removing it

[nadavMiz]

note that for notifications you need more information like size and eTag

[romayalon]

are these mandatory? I don't have the ETag in the candidate line, I can head object if you think it's needed

[nadavMiz]

doesn't do anything

[romayalon]

what do you mean? it sets the rule state