Cannot download log files from azure whilst simulation is running

[joehcollins]

Currently batch-download will only download logs from runs that have completed (including failure). We currently don't have a permanent and robust method in which to download unfinished logs as the simulation runs. This would be particularly useful for very long runs as we would be able to conduct initial preliminary analyses.

As of 04/07/2025 the below code snippet will however allow you to achieve this when ran in terminal. However this is not a permanent feature as it may not be compatible with future azure configurations or other changes. Please discuss with programming team before use and before there is an finalised approach to downloading in progress runs. (currently whilst the below snippet does run it will take a long time to download (150k pop, 280 runs, exceeding 2hrs)

az login

export AZURE_BATCH_ENDPOINT=https://tlob1ba.uksouth.batch.azure.com/ 
export AZURE_BATCH_ACCOUNT=tlob1ba
JOBID="service_integration_scenario-2025-07-01T144012Z"
DRAWS=28
RUNS=10
for DRAW in $(seq 0 $((DRAWS - 1)))
do
    for RUN in $(seq 0 $((RUNS - 1)))
    do
        TASKID="draw_${DRAW}-run_${RUN}"
        LOGFILE=$(az batch task file list --recursive --job-id $JOBID --task-id $TASKID --query "[?ends_with(name, '.log')].name | [0]" | tr -d '"')
        echo $DRAW, $RUN, $LOGFILE
        DIR=$(dirname "$LOGFILE")
        mkdir -p "$DIR"
        az batch task file download --file-path $LOGFILE --job-id $JOBID --task-id $TASKID --destination $LOGFILE
    done
done