record aborted testcases, part 1

Standards/scs-0100-w1-flavor-naming-implementation-testing.md

@@ -167,9 +167,12 @@ None so far.
 
 ### Implementation
 
-The script [`flavor-names-openstack.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming/flavor-names-openstack.py)
-talks to the OpenStack API of the cloud specified by the `OS_CLOUD` environment and queries properties and
-checks the names for standards compliance.
+We implemented two testcases, paralleling the two items in the "Errors" section above:
+
+- `scs-0100-syntax-check` ensures that any name starting with `SCS-` adheres to the standard;
+- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard.
+
+These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py).
 
 ## Manual tests
 

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -60,8 +60,16 @@ as ensured by the image metadata standard.
 
 ### Implementation
 
-The script [`entropy-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/entropy/entropy-check.py)
-connects to OpenStack and performs the checks described in this section.
+We implemented the following testcases that reflect the items in the above section
+on automated tests:
+
+- `scs-0101-image-property`,
+- `scs-0101-flavor-property`,
+- `scs-0101-entropy-avail`,
+- `scs-0101-rngd`,
+- `scs-0101-fips-test` (covers both the error and warning case).
+
+These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
 
 ## Manual tests
 

Standards/scs-0102-v1-image-metadata.md

@@ -239,9 +239,3 @@ A boolean property that is not present is considered to be `false`.
   contact for issues with this image. Note that this field must only be set if the
   service provider does provide support for this image included in the image/flavor
   pricing (but it might be provided by a contracted 3rd party, e.g. the OS vendor).
-
-### Conformance Tests
-
-The script `image-md-check.py` retrieves the
-image list from a configured cloud and checks each image for the
-completeness and consistency of mandatory properties.

Standards/scs-0102-w1-image-metadata-implementation-testing.md

@@ -16,16 +16,35 @@ for these images.
 
 ## Automated tests
 
-### Images sample
-
-Some checks need to be performed on a live instance. All publicly available images on this instance
-will be checked for either only the mandatory properties or possibly also the recommended ones.
-Additionally, a user can also decide to test their private images, although this isn't a necessity.
-
-### Implementation
-
-The script [`image-md-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/image-metadata/image-md-check.py)
-connects to OpenStack and performs the checks described in this section.
+We implemented a host of testcases to reflect the requirements and recommendations of the standard. The following
+testcases ensure that fields have proper values:
+
+- `scs-0102-prop-architecture`,
+- `scs-0102-prop-hash_algo`,
+- `scs-0102-prop-min_disk`,
+- `scs-0102-prop-min_ram`,
+- `scs-0102-prop-os_version`,
+- `scs-0102-prop-os_distro`,
+- `scs-0102-prop-hw_disk_bus`,
+- `scs-0102-prop-hypervisor_type`,
+- `scs-0102-prop-hw_rng_model`,
+- `scs-0102-prop-image_build_date`,
+- `scs-0102-prop-image_original_user`,
+- `scs-0102-prop-image_source`,
+- `scs-0102-prop-image_description`,
+- `scs-0102-prop-replace_frequency`,
+- `scs-0102-prop-provided_until`,
+- `scs-0102-prop-uuid_validity`,
+- `scs-0102-prop-hotfix_hours`.
+
+The property `patchlevel` is not tested because it is entirely optional.
+
+The following testcase ensures that each image is as recent as claimed by its `replace_frequency`:
+
+- `scs-0102-image-recency`
+
+The script [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+can be used to check these testcases.
 
 ## Manual tests
 

Standards/scs-0103-v1-standard-flavors.md

@@ -156,27 +156,6 @@ with `block_device_mapping_v2`, e.g.
 to create a bootable 12G cinder volume from image `IMGUUID` that gets tied to the VM
 instance life cycle.)
 
-## Conformance Tests
-
-The script [`flavors-openstack.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/standard-flavors/flavors-openstack.py)
-will read the lists of mandatory and recommended flavors
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the flavors are present and their extra_specs are correct.
-
-Missing flavors will be reported on various logging channels: error for mandatory, warning for
-recommended flavors. Incorrect extra_specs will be reported as error in any case.
-The return code will be non-zero if the test could not be performed or if any error was
-reported.
-
-The script does not check whether a name given via the extra_spec `scs:name-vN` is indeed valid according
-to any major version of the SCS standard on flavor naming.
-
-## Operational tooling
-
-The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
-create all standard, mandatory SCS flavors for you. It takes input that can be generated by
-`flavor-manager-input.py`.
-
 ## Previous standard versions
 
 The list of standard flavors used to be part of the flavor naming standard up until

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -0,0 +1,27 @@
+---
+title: "SCS Standard Flavors: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0103-v1-standard-flavors.md
+---
+
+## Operational tooling
+
+The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
+create all standard, mandatory SCS flavors for you. It takes input that can be generated by
+[`flavor-manager-input.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs_0100_flavor_naming/flavor-manager-input.py).
+
+## Automated tests
+
+We implemented a set of testcases corresponding 1:1 to the standard flavors:
+
+- `scs-0103-flavor-X` with varying `X`: ensures that flavor `SCS-X` is present and has the
+  required `extra_specs`.
+
+_NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard
+flavors are correct as well.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0104-v1-standard-images.md

@@ -128,18 +128,3 @@ next version. This standard makes no statement as to what is supposed to happen
 corresponding images in a live cloud environment. It is recommended to keep the
 once-mandatory images in the live environment. As for new environments, it is up to the
 operator whether to provide any or all of these images, as stated above.
-
-## Conformance Tests
-
-The script `images-openstack.py` will read the lists of mandatory and recommended images
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the images are present. Missing images will be reported on various
-logging channels: error for mandatory, info for recommended images. Additionally, images
-whose `image_source` does not conform with the specifications will be reported on the
-error channel. The return code will be non-zero if the test could not be performed or
-if any errors have been reported.
-
-## Operational tooling
-
-The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
-create all standard, mandatory SCS images for you given image definitions from a YAML file.

Standards/scs-0104-w1-standard-images-implementation.md

@@ -1,5 +1,5 @@
 ---
-title: "SCS Standard Images: Implementation Notes"
+title: "SCS Standard Images: Implementation and Testing Notes"
 type: Supplement
 track: IaaS
 status: Draft
@@ -59,3 +59,20 @@ Run the test script on your environment and check the error messages :)
 The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
 create all standard, mandatory SCS images for you given image definitions from a YAML file.
 Please see [its documentation](https://docs.scs.community/docs/iaas/components/image-manager/) for details.
+
+## Automated tests
+
+We implemented testcases reflecting the information given in the YAML file(s). Be advised that
+the approach with the YAML file has considerable drawbacks, and it will be abandoned in
+future versions of the standard in favor of a more classical approach. The testcases already
+anticipate this future development.
+
+There are two classes of testcases:
+
+- `scs-0104-source-X` with varying `X`:
+  these ensure that certain images have the correct `image_source`;
+- `scs-0104-image-X` with varying `X`:
+  these ensure that certain images can be found in the list of public images.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0114-v1-volume-type-standard.md

@@ -124,15 +124,3 @@ It should look like the following part:
 ## Related Documents
 
 - corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)
-
-## Conformance Tests
-
-The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests
-the following:
-
-- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list
-  (sorted, each entry at most once), and each entry is one of the possible features described here,
-- the recommended volume types are present (otherwise, a WARNING is produced).
-
-The return code is zero precisely when the test could be performed and the conditions are satisfied.
-Otherwise, detailed errors and warnings are output to stderr.

Standards/scs-0114-w1-volume-type-implementation.md

@@ -0,0 +1,20 @@
+---
+title: "SCS Volume Types: Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0114-v1-volume-type-standard.md
+---
+
+## Automated tests
+
+We implemented the following testcases:
+
+- `scs-0114-syntax-check` ensures that, for every volume type description,
+  the list of aspects, if present, is formatted according to the standard.
+- `scs-0114-encrypted-type` ensures that a volume type featuring encryption is present.
+- `scs-0114-replicated-type` ensures that a volume type featuring replication is present.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0115-v1-default-rules-for-security-groups.md

@@ -134,8 +134,3 @@ $ openstack default security group rule list
 The spec for introducing configurability for the default Security Groups Rules can be found [here](https://specs.openstack.org/openstack/neutron-specs/specs/2023.2/configurable-default-sg-rules.html).
 
 More about Security Groups as a resource in OpenStack can be found [here](https://docs.openstack.org/nova/latest/user/security-groups.html).
-
-## Conformance Tests
-
-The conformance tests should check for the absence of any ingress traffic rules except traffic from the same Security Group in the `openstack default security group rule list`.
-As having egress rules is allowed by this standard, but not forced and can be set in various ways, the tests should check for presence of any egress rules.

Standards/scs-0115-w1-security-groups-implementation.md

@@ -0,0 +1,22 @@
+---
+title: "Default Rules for Security Groups: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0115-v1-default-rules-for-security-groups.md
+---
+
+## Automated tests
+
+We implemented a single test case,
+
+- `scs-0115-default-rules`,
+
+which ensures
+
+1. the absence of any ingress traffic rules except traffic from the same Security Group in the `openstack default security group rule list`,
+2. the presence of any egress traffic rules.
+
+The testcase can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0116-w1-key-manager-implementation-testing.md

@@ -41,13 +41,13 @@ This can be done with a small change in the policy.yaml file. The `creator` has
 
 ## Automated Tests
 
-The check for the presence of a Key Manager is done with a test script, that checks the presence of a Key Manager service in the catalog endpoint of Openstack.
-This check can eventually be moved to the checks for the mandatory an supported service/API list, in case of a promotion of the Key Manager to the mandatory list.
+We implemented the following testcases, in accordance with the standard:
 
-### Implementation
+- `scs-0116-presence` ensures that a service of type "key-manager" occurs in the service catalog;
+- `scs-0116-permissions` ensures that a regular user has suitable access to the key-manager API.
 
-The script [`check-for-key-manager.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/key-manager/check-for-key-manager.py)
-connects to OpenStack and performs the checks described in this section.
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
 
 ## Manual Tests
 

Standards/scs-0117-v1-volume-backup-service.md

@@ -84,15 +84,3 @@ The volume backup target storage SHOULD be a separate storage system from the on
 
 - [OpenStack Block Storage v3 Backup API reference](https://docs.openstack.org/api-ref/block-storage/v3/index.html#backups-backups)
 - [OpenStack Volume Backup Drivers](https://docs.openstack.org/cinder/latest/configuration/block-storage/backup-drivers.html)
-
-## Conformance Tests
-
-Conformance tests include using the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it as a non-admin user and subsequently restore the backup on a new volume while verifying the success of each operation.
-These tests verify the mandatory part of the standard: providing the Volume Backup API.
-
-There is a test suite in [`volume-backup-tester.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/volume-backup-tester.py).
-The test suite connects to the OpenStack API and executes basic operations using the volume backup API to verify that the functionality requested by the standard is available.
-Please consult the associated [README.md](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/README.md) for detailed setup and testing instructions.
-
-Note that these tests don't verify the optional part of the standard: providing a separate storage backend for Cinder volume backups.
-This cannot be checked from outside of the infrastructure as it is an architectural property of the infrastructure itself and transparent to customers.

Standards/scs-0117-w1-volume-backup-service-implementation.md

@@ -0,0 +1,26 @@
+---
+title: "Volume Backup Functionality: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0117-v1-volume-backup-service.md
+---
+
+## Automated tests
+
+We implemented a single testcase,
+
+- `scs-0117-test-backup`,
+
+which verifies that a non-admin user can backup and restore volumes.
+
+To this end, the testcase uses the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it and subsequently restores the backup on a new volume while verifying the success of each operation.
+
+The testcase can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
+
+## Manual tests
+
+Note that the automated tests don't verify the optional part of the standard: providing a separate storage backend for Cinder volume backups.
+This cannot be checked from outside of the infrastructure as it is an architectural property of the infrastructure itself and transparent to customers.

Standards/scs-0123-w1-mandatory-and-supported-IaaS-services-implementation.md

@@ -0,0 +1,19 @@
+---
+title: "Mandatory and Supported IaaS Services: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0123-v1-mandatory-and-supported-IaaS-services.md
+---
+
+## Automated tests
+
+We implemented the following testcases in accordance with the standard:
+
+- `scs-0123-service-X` (with varying `X`) ensures that a service of type X can be found in the service catalog,
+- `scs-0123-storage-apis` ensures that a service of one of the following types can be found: "volume", "volumev3", "block-storage",
+- `scs-0123-swift-s3` ensures that S3 can be used to access object storage using EC2 credentials from the identity API.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).