Transfer Files from Basespace

Synopsis

# directory we are going to mount with BaseMount; create it and mount it!
basepace_dir="$HOME/BaseSpace"
mkdir -p "$basepace_dir"
basemount "${basepace_dir}/"

# ID of the project we are interested in
project_ID="NS17-03"

# run the transfer script
/ifs/data/molecpathlab/scripts/transfer_BaseSpace_to_quicksilver.sh "$project_ID" "$basepace_dir"

# unmount your BaseSpace directory when finished
basemount --unmount "${basepace_dir}/"

Overview

If the Next-Seq DNA sequencer is set to demultiplex its run data with the BaseSpace Cloud app, then the final demultiplexed fastq files need to be transferred back to the phoenix server for further analysis.

This workflow will show you how to mount a BaseSpace account on the HPC server so you can copy these files.

BaseMount Help Page:

https://help.basespace.illumina.com/articles/descriptive/introduction-to-basemount/

Before you start:

• in a web browser, log into the Basespace account holding the runs to be transferred
• if applicable, make sure the Basespace account has loaded the associated group that has your desired projects
  • e.g. if your project is under your Personal account, then switch to your Personal account in BaseSpace before you start

Loading BaseMount

BaseMount is already installed on the phoenix server. You just need to create and mount a directory to load the files into

# directory we are going to mount with BaseMount; create it and mount it!
basepace_dir="$HOME/BaseSpace"
mkdir -p "$basepace_dir"
basemount "${basepace_dir}/"

Now you will get a BaseMount message with a URL to open in your BaseSpace account; copy/paste the URL into the web browser

Should look like this:

# 
# ,-----.                        ,--.   ,--.                         ,--.
# |  |) /_  ,--,--. ,---.  ,---. |   `.'   | ,---. ,--.,--.,--,--, ,-'  '-.
# |  .-.  \' ,-.  |(  .-' | .-. :|  |'.'|  || .-. ||  ||  ||      \'-.  .-'
# |  '--' /\ '-'  |.-'  `)\   --.|  |   |  |' '-' ''  ''  '|  ||  |  |  |
# `------'  `--`--'`----'  `----'`--'   `--' `---'  `----' `--''--'  `--'
# Illumina BaseMount v0.15.15.1872 public  2016-12-16 10:47
# 
# Command called:
#     basemount /ifs/home/kellys04/projects/Clinical_580_gene_panel/BaseSpace/
# From:
#     /ifs/home/kellys04/projects/Clinical_580_gene_panel
# 
# Starting authentication.
# 
# You need to authenticate by opening this URL in a browser:
#   https://basespace.illumina.com/oauth/device?code=NwiY2
# ........
# It worked!
# Your identification has been saved.
# 
# Mounting BaseSpace account.
# To unmount, run: basemount --unmount /ifs/home/kellys04/projects/Clinical_580_gene_panel/BaseSpace

Copy the Files

Its strongly advised to run this from within screen on the server.

Use this scripted workflow. Run this in a screen.

# ID of the project we are interested in
project_ID="NS17-03"

/ifs/data/molecpathlab/scripts/transfer_BaseSpace_to_quicksilver.sh "$project_ID" "$basepace_dir"

# unmount your BaseSpace directory when finished
basemount --unmount "${basepace_dir}/"

An old way that you shouldn't use anymore:

Make a list of the fastq files to get for the Project you want, and copy them with rsync to the sequencer directory.

Since we want to preserve the directory structure from BaseSpace, using the Project ID as the parent directory, we will first assemble a list of the fastq files to be copied with relative paths. Then we will run rsync from within the mounted BaseSpace directory to copy the files.

sequencer_dir="/ifs/data/molecpathlab/quicksilver"

basepace_dir="/ifs/home/kellys04/projects/Clinical_580_gene_panel/BaseSpace"
basepace_projects="/ifs/home/kellys04/projects/Clinical_580_gene_panel/BaseSpace/Projects"
files_list_dir="/ifs/home/kellys04/projects/Clinical_580_gene_panel/BaseSpace_files_lists"; mkdir -p "$files_list_dir"

project_ID="NS17-02"
project_files_list="${files_list_dir}/${project_ID}.txt"


(
cd "${basepace_projects}"

# make list of fastq files for the run; preserve the 
find "${basepace_projects}/${project_ID}" -name "*.fastq.gz" | sed -e "s|${basepace_projects}/||g" > "$project_files_list"

printf "\nFiles to be copied:\n\n"
cat "$project_files_list"

# transfer the files
rsync --progress -vRt --files-from="$project_files_list" "$basepace_projects" "$sequencer_dir" # --dry-run
)

Notes

Authentication

The first time you run BaseMount, you will be directed to a web URL and asked to enter your BaseSpace Sequence Hub user credentials. BaseMount will use these credentials to authenticate your interactions with BaseSpace Sequence Hub. By default, the credentials are cached in your home directory and they can be password-encrypted for security, just like an ssh key.

• this means that if you need to reset your login credentials on the server with BaseMount, you can just nuke the ~/.basespace dir. It will get re-created when you start the procedure over again.