Additional testing ideas

[dcermak]

Currently we perform only very basic tests and do not really check the image's functionality or additional properties.

This is a collection of the various images that we currently produce in the staging area and what their capabilities/properties are:

test-image-MicroOS
  builds a MicroOS image, this comes with a semi read-only
  rootfs which could be tested

test-image-azure
  build image suitable to run in Microsoft Azure,
  there is imho nothing you can test other than booting
  has vhd-fixed disk type in qemu

test-image-bundle-format
  build an image with a custom image name, the test
  can check if the image file name matches the given
  bundle format: https://osinside.github.io/kiwi/image_types_and_results.html?highlight=bundle_format#image-bundle-format

test-image-custom-partitions
  build an image including two custom scripts to change the
  partition table for including custom changes. The image exists
  to demo how extra scripts can be used to modify the disk layout.
  There is not much to test other than the boot. With the new
  feature of the <partitions> element users don't need to
  create custom scripts anymore, but the test is still there

test-image-disk
  build an image with the resize feature enabled, a test could
  check if the disk geometry was really resized

test-image-disk-legacy
  build an image using the legacy oemboot code. This means the
  image builds a custom initrd not based on dracut. There is not
  much to test other than the boot

test-image-disk-ramdisk
  build an image to be deployed onto a ramdisk. The image also builds
  an install.iso image which can be used to dump the system onto a
  ramdisk. A real test would call this .install.iso and see if the
  later system completely runs from a ramdisk

test-image-disk-simple
  build a simple disk image, no resize feature

test-image-docker
  build a docker image

test-image-docker-derived
  build a docker image derived from another image

test-image-ec2
  build an image for the Amazon EC2 cloud, there is nothing you
  can test outside of the cloud

test-image-gce
  build an image for the Google cloud, there is nothing you
  can test outside of the cloud

test-image-live
  build a live system. A test can boot and see if it's writable
  everywhere to check if the overlay technology works as expected

test-image-luks
  build a luks encrypted system. This also includes /boot to be
  encrypted. Boot test would be enough

test-image-lvm
  build an image with LVM volumes, a test can check if they exist
  as expected

test-image-orthos
  build an image for the SUSE orthos system. I think that one should
  be deleted

test-image-overlayroot
  build a disk image using kiwi's overlayroot technology. The root
  partition gets overlayed via overlayfs and all writes are targeted
  into a dedicated write partition on the same disk. A test can
  check the layout and the write capabilities. With overlayroot you
  can build relatively small systems because all of the root is turn
  into a squashfs. It'a also a real immutable system but no transactional
  updates

test-image-partitions-and-volumes
  testing the <partitions> feature, a test can check if the partitions
  matches the setup from the description

test-image-pxe
  build a pxe image for the old netboot concept. Testing is only
  possible in this PXE environment

test-image-qcow-openstack
  build a qcow2 image

test-image-raid
  build a raid image. All kiwi raid images produces raids in degraded
  mode. This is needed because you have no access to other raid disks
  at image build time. A test can check the state of the raid and could
  also attach a disk and see if the raid turns into a non degraded raid

test-image-suse-on-dnf
  build a suse image with dnf

test-image-tbz
  build a tarball from the root

test-image-vagrant
  build a vagrant box, testing requires to run vagrant

test-image-wsl
  build a WSL (Windows Subsystem for Linux) container image. Testing
  can only be done on Windows