Adding Test Job

Overview

This Wiki page aims to provide instructions to add test jobs for assets. To make sure the components are working properly, please make sure all assets you want to publish are covered by test jobs. Our test frameworks will run these test jobs during the different stages of publishing. All tests should be submitted and run successfully locally before uploading to the repository. It is your responsibility to find proper test jobs for your assets and adjust the jobs accordingly to integrate with our test framework.

How to add test jobs

There are two steps to adding test jobs: 1. upload test-jobs-related files, and 2. create or update tests.yml in your team's folder.

1. upload test-jobs-related files

Your team should have a folder to include all files you upload to the repository, e.g. training team has a training folder. Under this folder, you could have many different groups (also called test_group), e.g. training team has a general group and a vision group. Under a group folder, you should have a component folder (or other assets folder) and a job folder (or other folders you might have). Under the job folder, you should include all test-job-related files for this group. Since our .gitignore will ignore data folder in this repo, please rename your data folder to resources. Remember to modify your test jobs and pre/post scripts accordingly with this name change of data folder.

2. create or update tests.yml in your team's folder

tests.yml is an imperative concept in the test framework. It indicates the files/folders needed for test jobs and the process of running test jobs. The schema of the tests.yml is attached below. If you need to execute some scripts before submitting jobs like generating data, and some scripts after the jobs are completed, you can have pre or/and post jobs. They will be executed accordingly, and for now, we only support python scripts without any arguments. If you need workspace information in your pre/post scripts, you can retrieve it from env vars:

subscription_id = os.environ.get('subscription_id')
resource_group = os.environ.get('resource_group')
workspace = os.environ.get('workspace')

If you want to have pre/post scripts for the whole test group, you could also add them to the pre/post under test groups.

Important! You should put all paths of files/folders needed for your test jobs in includes, otherwise test jobs will not find those files to submit.

Test coverage check

When you submit a PR to add assets or test jobs, it will automatically run a test coverage check to make sure all assets you want to publish later should be covered by test jobs.

Open the details of this check, you can see a Check test coverage task, and inside you can see a warning telling you which assets are not covered by test jobs. You also need to, if it has not been done yet, add your team's folder to main-to-release workflow and assets-validation. Please refer to how training team did.

If your PR is including any test jobs, please always request an approval from Chuyuan

#schema 
 test_group_name:
   pre: path
   post: path
   includes: 
      - path 1
      - path 2
    ...
   jobs:
      job_name:
        job: path
        pre: path
        post: path
      job_name:
        job: path
        pre: path
        post: path
      ...
 test_group_name:
   pre: path
   post: path
   includes: 
      - path 1
      - path 2
    ...
   jobs:
      job_name:
        job: path
        pre: path
        post: path
      job_name:
        job: path
        pre: path
        post: path
      ...