1.0.5

• Fixed the newly introduced issue of needing more ebs size because of cwltool copying output instead of moving due to symlink
• the rerun and rerun_many commands now can take non-integer values (e.g. "3x") for --ebs-size
• non-admin user cannot run workflows on a tibanna step function from the group that they don't belong to (Previously they could run things, but the jobs would fail if the jobs require access to a bucket that this user didn't have access to)

1.0.4

• No change in behavior
• Minor change in tibanna.cw_utils API to allow a more flexible extension.
  • TibannaResources.create_html generates an html string that requires one additional element in the format tuple (report title).

1.0.2

no significant change in behavior, just using the latest goofys version for s3 mounting.

1.0.1

• Language wdl is recognized as wdl_v1 not wdl_draft2 (to be consistent with the documentation)
• cromwell 35 is used for wdl_draft2 instead of cromwell 31.

1.0.0

Release note for v1.0.0

Note : To use the functionality of version 1.0.0, the locally installed Tibanna Python package must also be upgraded and the Tibanna unicorns must be redeployed to AWS from the upgraded package.

Awsem

• Most of the stuff needed by an Awsem instance is now Dockerized, including the AWSF scripts, Cwltool, Cromwell and docker itself. The AWSF scripts are now in Python 3.8 and the base image is Ubuntu 20.04. The docker image is 4dndcic/tibanna-awsf:<latest> (e.g. 4dndcic/tibanna-awsf:1.0.0) and is available on Docker Hub. This image is not meant for any other use outside Tibanna Awsem instances which automatically pulls this image to to its job.
• New AMI based on Ubuntu 20.04 is used with fewer components than before, since most of them are mow moved to a docker image.
• The mounted EBS directory is now /mnt/data1, which is symlinked to /data1 (no real difference)
• The workflow / user Docker image will be now be pulled and run inside the Tibanna-AWSF Docker container and it will use the data EBS to store the docker container rather than the root EBS. The users don't have to increase root_ebs_size for a large Docker image any more. Instead, increasing ebs_size (data EBS size) may be necessary depending on the size of the user's Docker image.
• A Tibanna unicorn will partly depend on the Tibanna repo's awsf3 directory (instead of the awsf directory).

log

• The log file (<job_id>.log) format has changed (much better now!)
• The Top command output is no longer a part of the <job_id>.log but it in a separate <job_id>.top and <job_id>.top_latest files, which can be retrieves by the following commands:

tibanna log -j <job_id> -t  # all top command output at 1-minute interval

tibanna log -j <job_id> -T # the latest top command output

metrics plot

• The metrics plot now includes per-process CPU and memory barcharts based on the top command outputs. These plots are useful to figure out which process or step takes a lot of CPU or memory or not using enough CPU / memory, etc.

output specification

• The output target specification in the input json (job description json) can now be a dictionary.
• A zipped output can be directly unzipped into a directory in S3 by specifying 'unzip': True in this dictionary. See example below:

"output_target": {
    "my_zipfile": {
        "unzip": true,
        "object_prefix": "some_directory_for_zip_content/"
    }
}

Commands and options

• The stat functionality now has an option to specify job IDs.
• The new run_batch_workflows subcommand allows running many workflows in a batch.
• The new info command prints out the information about each job stored in dynamoDB (complementary to stat).

AMI

• Code that builds AMI from scratch is available now, though most users don't need it, since the Tibanna AMI is publicly available.

AWS regions

• Public Tibanna AMI for the following new regions are now available: me-south-1, af-south-1, ap-east-1 and eu-south-1. (Total 20 different AWS regions)

WDL and CWL versions

• CWL draft-3 support is now dropped.
• WDL v1.0 is now supported (through Cromwell 53.1). To continue using and old WDL draft-2 workflow, specify language as wdl_draft2 in the input job description json. Cromwell 53.1 supports both v1.0 and draft-2, but some old components are not supported. Try switching to wdl_draft2 in such a case. For backward compatibility, we use the same Cromwell 31 that we used in previous Tibanna versions, to run WDL draft-2.

0.18.3

plot_metrics now has an optional field --instance-id (-i) to provide instance ID manually in case Tibanna somehow fails to find it.

tibanna plot_metrics -j <job_id> -i <instance_id>

0.18.2

• fix for key error (key: commands) when postrun json is very large

0.18.1

• fix for EC2Idle error message
• allowing cleanup for group names starting with tibanna (but not with tibanna_)

0.18.0

• Support for docker images on Elastic Container Registry (ECR) (in the same region as the running Tibanna/EC2)

  • Requires redeployment of tibanna to enable this feature

  tibanna deploy_unicorn [-b buckets] [-g usergroup] [-s suffix]
  

• The problem of throwing a wrong error message when the run fails (AccessDenied to ListExecutions operation ...) is fixed.

0.17.3

The problem of resource metrics not collected for regions other than us-east-1 is now fixed.

Metrics can be collected retroactively (to recent jobs) using tibanna plot_metrics -j <job_id> without redeployment.

For automatic metric collection for future jobs, redeployment is required as below.

tibanna deploy_core -n check_task_awsem [-g group] [-s suffix]