From 74b7ca34395f48dd52cf25c5d82aaf2f2b81826e Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Mon, 3 Mar 2025 10:00:42 -0600
Subject: [PATCH 01/13] Created Condor_submit Quick Ref

---
 _news                                         |  2 +-
 .../condor-submit-quick-ref.md                | 92 +++++++++++++++++++
 staff-list                                    |  2 +-
 3 files changed, 94 insertions(+), 2 deletions(-)
 create mode 100644 _uw-research-computing/condor-submit-quick-ref.md

diff --git a/_news b/_news
index f3ed5d56..d2a7c240 160000
--- a/_news
+++ b/_news
@@ -1 +1 @@
-Subproject commit f3ed5d56a1cd7ce837b30dce1dd4d901b10eb36c
+Subproject commit d2a7c2404590499f111907be4880fdef274faf3c
diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
new file mode 100644
index 00000000..aa71ccd4
--- /dev/null
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -0,0 +1,92 @@
+---
+layout: guide
+title: "Quick reference: HTCondor Submission commands"
+alt_title: "Quick reference: HTCondor Submission commands"
+guide:
+    category: Get started
+    tag: htc
+---
+
+## Introduction
+
+This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
+
+{% capture content %}
+- [Introduction](#introduction)
+- [Submit jobs](#submit-jobs)
+- [Monitor jobs](#monitor-jobs)
+- [Machine information](#machine-information)
+- [Glossary](#glossary)
+- [Related Pages](#related-pages)
+{% endcapture %}
+{% include /components/directory.html title="Table of Contents" %}
+
+> **Please note the following!** 
+> 
+> * Bracketed items (`<>`) denote where to place your input. 
+>   Do not include the brackets in your command.
+> * All commands should be entered on a single line.
+
+## Submit jobs
+
+You can use these commands to submit, hold, or remove your jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `condor_submit` | submits job(s) as specified by `submit_file` | `condor_submit helloWorld.sub` | See [monitor your jobs](condor_q) |
+| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | `condor_submit -i helloWorld.sub`| 
+| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed.  
+{:.command-table}
+
+## Basic Submit File Options
+
+These commands display information about the execution points - machines that execute/run jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `arguments` | lists all execution point slots |
+| `environment` | lists information about the specified `execution_point` |
+| `log = <job.log>` | lists all attributes of `execution_point` |
+| `out = <job.out>` | lists all attributes of `execution_point` |
+| `error = <job.err>` | lists all attributes of `execution_point` |
+| `executable = <executable.sh>` | lists all attributes of `execution_point` |
+| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+{:.command-table}
+
+## Queue Statement Options
+
+These commands display information about the execution points - machines that execute/run jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `arguments` | lists all execution point slots |
+| `environment` | lists information about the specified `execution_point` |
+| `log = <job.log>` | lists all attributes of `execution_point` |
+| `out = <job.out>` | lists all attributes of `execution_point` |
+| `error = <job.err>` | lists all attributes of `execution_point` |
+| `executable = <executable.sh>` | lists all attributes of `execution_point` |
+| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+{:.command-table}
+
+## Glossary
+
+| Term | Meaning |
+| --- | --- |
+| access point | The machine which you log into to access CHTC's servers. This is the machine you use to prepare files for job submission and submit jobs. |
+| cluster ID | A unique number associated with a single job submission. |
+| error file / standard error | The file in which your job typically prints any errors. |
+| execution point | The machine that executes or runs your job. |
+| held/hold | A job state in which your job has stopped running due to an error. |
+| idle | A job state in which your job has not yet matched to an execution point and hasn't started running yet. |
+| job ID | The unique number associated with a job. This consists of a cluster ID, followed by a period and a process ID. For example, the job ID `12345.0` has a cluster ID of `12345` and a process ID of `0`. |
+| log file | The file where HTCondor prints messages about your job's execution and resource usage. |
+| output file / standard out | The file where your job typically prints output. Any messages printed to the "screen" in a job will be saved in this file. |
+| process ID | A unique number associated with each job within a job submission. See [submit multiple jobs](multiple-jobs). |
+| running | A job state in which your job has matched to the execution point and is currently executing/running. |
+| submit file | A text-based file that tells HTCondor details about your job, including the commands to run, what files need to be transferred, where to save the outputs, and more. See the [job submission](htcondor-job-submission). |
+
+## Related Pages
+
+* [Job submission](htcondor-job-submission)
+* [Monitor your jobs](condor_q)
+* [HTCondor manual](https://htcondor.readthedocs.io/en/latest/)
diff --git a/staff-list b/staff-list
index 7a79e9d7..b4c7b214 160000
--- a/staff-list
+++ b/staff-list
@@ -1 +1 @@
-Subproject commit 7a79e9d7a04d91835a59c92f917ed4e2ff77ba0f
+Subproject commit b4c7b21414983da79c75e7c5e54b13a4777b7bbb

From d527844d98a5d9f14ae477b5b6aeb5bd3268a4bb Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Mon, 3 Mar 2025 17:26:33 -0600
Subject: [PATCH 02/13] Updated condor_submit ref page

---
 .../condor-submit-quick-ref.md                | 51 +++++++++++++++++--
 1 file changed, 46 insertions(+), 5 deletions(-)

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index aa71ccd4..8371df35 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -44,8 +44,23 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `arguments` | lists all execution point slots |
-| `environment` | lists information about the specified `execution_point` |
+| `arguments = <args>` | lists all execution point slots |
+| `environment = <parameter_list>` | lists information about the specified `execution_point` |
+| `log = <job.log>` | lists all attributes of `execution_point` |
+| `output = <job.out>` | lists all attributes of `execution_point` |
+| `error = <job.err>` | lists all attributes of `execution_point` |
+| `executable = <executable.sh>` | lists all attributes of `execution_point` |
+| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+{:.command-table}
+
+## Managing File Transfers in HTCondor
+
+These commands display information about the execution points - machines that execute/run jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `transfer_input_files = ` | lists all execution point slots |
+| `transfer_input_files = ` | lists information about the specified `execution_point` |
 | `log = <job.log>` | lists all attributes of `execution_point` |
 | `out = <job.out>` | lists all attributes of `execution_point` |
 | `error = <job.err>` | lists all attributes of `execution_point` |
@@ -53,15 +68,41 @@ These commands display information about the execution points - machines that ex
 | `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
 {:.command-table}
 
+## Controlling Where Your Job Runs
+
+These commands display information about the execution points - machines that execute/run jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `request_cpus = <int>` | lists all execution point slots |
+| `request_disk = <quantity>` | lists information about the specified `execution_point` |
+| `request_memory = <quantity>` | lists all attributes of `execution_point` |
+| `request_gpus = <int>` | lists all attributes of `execution_point` |
+| `requirements = <ClassAd Boolean>` | lists all attributes of `execution_point` |
+| `gpus_minimum_capacity = <version>` | lists all attributes of `execution_point` |
+| `cuda_version = <version>` | lists all attributes of `execution_point` |
+{:.command-table}
+
+## Controlling How Your Job Runs
+
+These commands display information about the execution points - machines that execute/run jobs.
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `universe = <vanilla, scheduler, local, grid, java, vm, parallel, docker, container>` | lists all execution point slots |
+| `container_image = ` | lists information about the specified `execution_point` |
+| `container_target_dir = </path/to/dir >` | lists all attributes of `execution_point` |
+{:.command-table}
+
 ## Queue Statement Options
 
 These commands display information about the execution points - machines that execute/run jobs.
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `arguments` | lists all execution point slots |
-| `environment` | lists information about the specified `execution_point` |
-| `log = <job.log>` | lists all attributes of `execution_point` |
+| `queue` | lists all execution point slots |
+| `queue <int>` | lists information about the specified `execution_point` |
+| `queue ` | lists all attributes of `execution_point` |
 | `out = <job.out>` | lists all attributes of `execution_point` |
 | `error = <job.err>` | lists all attributes of `execution_point` |
 | `executable = <executable.sh>` | lists all attributes of `execution_point` |

From 7144d9f7ead3e16e5bca1ad8f1ef37c0720f75a3 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 5 Mar 2025 12:41:50 -0600
Subject: [PATCH 03/13] Updated condor_submit quickref - adding basic and
 submit info

---
 .../condor-submit-quick-ref.md                | 26 +++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index 8371df35..cd15d780 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -35,7 +35,7 @@ You can use these commands to submit, hold, or remove your jobs.
 | --- | --- | --- | --- |
 | `condor_submit` | submits job(s) as specified by `submit_file` | `condor_submit helloWorld.sub` | See [monitor your jobs](condor_q) |
 | `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | `condor_submit -i helloWorld.sub`| 
-| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed.  
+| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. |
 {:.command-table}
 
 ## Basic Submit File Options
@@ -44,12 +44,12 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `arguments = <args>` | lists all execution point slots |
-| `environment = <parameter_list>` | lists information about the specified `execution_point` |
-| `log = <job.log>` | lists all attributes of `execution_point` |
-| `output = <job.out>` | lists all attributes of `execution_point` |
-| `error = <job.err>` | lists all attributes of `execution_point` |
-| `executable = <executable.sh>` | lists all attributes of `execution_point` |
+| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` ("hello", "world" are passed as two separate arguments)| arguments are wrapped by quotes(") and space separated |
+| `environment = <parameter_list>` | lists environmental variables | `environment = "VARIABLE1=Value1 VAR2='hello world'"` | wrapped by quotes(") and space separated |
+| `log = <job.log>` | denotes the path to the log file | `log = ./log_files/job1.log`| if not provided, no log file is recorded |
+| `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
+| `error = <job.err>` | path to file capturing `stderr` screen output | `output = ./log_files/job1.err` | can be merged with `stdout` by denoting the same path in `output = <path>` |
+| `executable = <executable.sh>` | path to the executable script | `executable = helloWorld.py` | the executable script is automatically transferred to the Execution Point (EP) by HTCondor |
 | `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
 {:.command-table}
 
@@ -89,7 +89,7 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `universe = <vanilla, scheduler, local, grid, java, vm, parallel, docker, container>` | lists all execution point slots |
+| `universe = <vanilla, docker, or container>` | lists all execution point slots |
 | `container_image = ` | lists information about the specified `execution_point` |
 | `container_target_dir = </path/to/dir >` | lists all attributes of `execution_point` |
 {:.command-table}
@@ -102,11 +102,11 @@ These commands display information about the execution points - machines that ex
 | --- | --- | --- | --- |
 | `queue` | lists all execution point slots |
 | `queue <int>` | lists information about the specified `execution_point` |
-| `queue ` | lists all attributes of `execution_point` |
-| `out = <job.out>` | lists all attributes of `execution_point` |
-| `error = <job.err>` | lists all attributes of `execution_point` |
-| `executable = <executable.sh>` | lists all attributes of `execution_point` |
-| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+| `queue <var> from <list>` | lists all attributes of `execution_point` |
+| `queue <var1>, <var2> from <list>` | lists all attributes of `execution_point` |
+| `queue <int_expr>, <var2> from <list>` | lists all attributes of `execution_point` |
+| `queue <var1> <var> in [slice] <list>` | lists all attributes of `execution_point` |
+| `queue <var> matching <globbing_string>` | lists all attributes of `execution_point` |
 {:.command-table}
 
 ## Glossary

From bce898c647d1c5e11d2f56a39dba17473ded9114 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 5 Mar 2025 15:08:16 -0600
Subject: [PATCH 04/13] Updated condor_submit quickref

---
 .../condor-submit-quick-ref.md                | 40 +++++++++----------
 1 file changed, 18 insertions(+), 22 deletions(-)

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index cd15d780..cf2bcc36 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -50,7 +50,7 @@ These commands display information about the execution points - machines that ex
 | `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
 | `error = <job.err>` | path to file capturing `stderr` screen output | `output = ./log_files/job1.err` | can be merged with `stdout` by denoting the same path in `output = <path>` |
 | `executable = <executable.sh>` | path to the executable script | `executable = helloWorld.py` | the executable script is automatically transferred to the Execution Point (EP) by HTCondor |
-| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+| `notification = <Always, Complete, Error, or None>` | notifies by e-mail when certain events occur | | **use carefully:* it can create an large # of inbound emails for multi-job HTCondor clusters |
 {:.command-table}
 
 ## Managing File Transfers in HTCondor
@@ -59,13 +59,10 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `transfer_input_files = ` | lists all execution point slots |
-| `transfer_input_files = ` | lists information about the specified `execution_point` |
-| `log = <job.log>` | lists all attributes of `execution_point` |
-| `out = <job.out>` | lists all attributes of `execution_point` |
-| `error = <job.err>` | lists all attributes of `execution_point` |
-| `executable = <executable.sh>` | lists all attributes of `execution_point` |
-| `notification = <Always, Complete, Error, or None>` | lists all attributes of `execution_point` |
+| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | `transfer_input_files = /staging/<user>/<dir>` \ `transfer_input_files = <protocol>:///chtc/staging/<user>/<dir>` | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`|
+| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) from the Execute Point (EP) | `transfer_output_files = ./output_files/results.txt` | |
+| `transfer_output_remaps = "<path_on_EP> = <new_path_on_AP>` | remaps (redirects or renames) output files explicitly listed to the Access Point (AP) on job completion | `transfer_output_remaps = "./results.txt = /staging/<user>/job1_results.txt` (saves results.txt as `/staging/<user>/job1_results.txt`) | |
+| `when_to_transfer_output = <ON_EXIT, ON_EXIT_OR_EVICT, or ON_SUCCESS>` | causes HTCondor to transfer job outputs based on the job's exit code | |
 {:.command-table}
 
 ## Controlling Where Your Job Runs
@@ -74,13 +71,14 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `request_cpus = <int>` | lists all execution point slots |
-| `request_disk = <quantity>` | lists information about the specified `execution_point` |
-| `request_memory = <quantity>` | lists all attributes of `execution_point` |
-| `request_gpus = <int>` | lists all attributes of `execution_point` |
-| `requirements = <ClassAd Boolean>` | lists all attributes of `execution_point` |
-| `gpus_minimum_capacity = <version>` | lists all attributes of `execution_point` |
-| `cuda_version = <version>` | lists all attributes of `execution_point` |
+| `request_cpus = <int>` | requests number of CPUs (cores)|
+| `request_disk = <quantity>` | requests amount of disk space (Default in KB) | `request_disk = 40GB` | can denote kilobytes using `K` or `KB`, megabytes using `M` or `MB`, gigabytes using `G` or `GB`, or terrabytes using `T` or `TB` |
+| `request_memory = <quantity>` | requests amount of memory for job (Default in KB) | `request_memory = 250GB` | |
+| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested |
+| `requirements = <ClassAd Boolean>` | appends additional requirements to the job submission file | `Requirements = (OpSysAndVer == "RedHat9")`| See PAGEHERE for all requirement options
+| `gpus_minimum_capacity = <version>` | minimum GPU Capability value | `gpus_minimum_capability = 8.5` | 
+| `gpus_minimum_runtime = <version>` | minimum GPU runtime (usually CUDA) version | `gpus_minimum_runtime = 9.1` | 
+| `cuda_version = <version>` | specifies version of CUDA to be used | | superceeded by `gpus_minimum_runtime` |
 {:.command-table}
 
 ## Controlling How Your Job Runs
@@ -89,9 +87,8 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `universe = <vanilla, docker, or container>` | lists all execution point slots |
-| `container_image = ` | lists information about the specified `execution_point` |
-| `container_target_dir = </path/to/dir >` | lists all attributes of `execution_point` |
+| `universe = <vanilla or container>` | specifies which HTCondor universe environment to use when running this job | | `universe = vanilla` is the default HTCondor environment \ `universe = container` specifies an environment built for running container images (Docker, Apptainer, and Singularity)|
+| `container_image = <image_file>` | specifies the path to a container image to be used | `container_image = docker://pytorch/pytorch:latest` | can pull image from DockerHub or load `.sif` files directly |
 {:.command-table}
 
 ## Queue Statement Options
@@ -100,10 +97,9 @@ These commands display information about the execution points - machines that ex
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `queue` | lists all execution point slots |
-| `queue <int>` | lists information about the specified `execution_point` |
-| `queue <var> from <list>` | lists all attributes of `execution_point` |
-| `queue <var1>, <var2> from <list>` | lists all attributes of `execution_point` |
+| `queue` | command line queuing the job to start | | if no other options specified, a single job is queued |
+| `queue <int>` | places zero or more copies of the job into the HTCondor queue | `queue 10` | |
+| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./list.txt` |
 | `queue <int_expr>, <var2> from <list>` | lists all attributes of `execution_point` |
 | `queue <var1> <var> in [slice] <list>` | lists all attributes of `execution_point` |
 | `queue <var> matching <globbing_string>` | lists all attributes of `execution_point` |

From f59d68a7d8a6f5c600151f20c5d8ee5d8ee8b705 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 5 Mar 2025 16:11:49 -0600
Subject: [PATCH 05/13] Updated quick-ref condor_submit

---
 _news                                         |  2 +-
 .../condor-submit-quick-ref.md                | 39 ++++++++++---------
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/_news b/_news
index d2a7c240..fd915d2e 160000
--- a/_news
+++ b/_news
@@ -1 +1 @@
-Subproject commit d2a7c2404590499f111907be4880fdef274faf3c
+Subproject commit fd915d2e2bb37b9e6bb439224bc4bb5b21c8bf4e
diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index cf2bcc36..e6fc8e72 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -14,8 +14,11 @@ This page lists common HTCondor commands and options for jobs. Users familiar wi
 {% capture content %}
 - [Introduction](#introduction)
 - [Submit jobs](#submit-jobs)
-- [Monitor jobs](#monitor-jobs)
-- [Machine information](#machine-information)
+- [Basic Submit File Options](#basic-submit-file-options)
+- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
+- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
+- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
+- [Queue Statement Options](#queue-statement-options)
 - [Glossary](#glossary)
 - [Related Pages](#related-pages)
 {% endcapture %}
@@ -29,22 +32,22 @@ This page lists common HTCondor commands and options for jobs. Users familiar wi
 
 ## Submit jobs
 
-You can use these commands to submit, hold, or remove your jobs.
+You can use these commands to submit jobs to HTCondor. 
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `condor_submit` | submits job(s) as specified by `submit_file` | `condor_submit helloWorld.sub` | See [monitor your jobs](condor_q) |
-| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | `condor_submit -i helloWorld.sub`| 
-| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. |
+| `condor_submit` | submits job(s) as specified by `submit_file` | `condor_submit helloWorld.sub` | See [monitor your jobs](condor_q) 
+| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | `condor_submit -i helloWorld.sub`
+| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. 
 {:.command-table}
 
 ## Basic Submit File Options
 
-These commands display information about the execution points - machines that execute/run jobs.
+These commands specify the basic submit file options needed to start a job on HTCondor 
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` ("hello", "world" are passed as two separate arguments)| arguments are wrapped by quotes(") and space separated |
+| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` | arguments are wrapped by quotes(") and space separated |
 | `environment = <parameter_list>` | lists environmental variables | `environment = "VARIABLE1=Value1 VAR2='hello world'"` | wrapped by quotes(") and space separated |
 | `log = <job.log>` | denotes the path to the log file | `log = ./log_files/job1.log`| if not provided, no log file is recorded |
 | `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
@@ -55,11 +58,11 @@ These commands display information about the execution points - machines that ex
 
 ## Managing File Transfers in HTCondor
 
-These commands display information about the execution points - machines that execute/run jobs.
+These commands control how and when files are transferred during submission and following job ending 
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | `transfer_input_files = /staging/<user>/<dir>` \ `transfer_input_files = <protocol>:///chtc/staging/<user>/<dir>` | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`|
+| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | `transfer_input_files = /staging/<user>/<dir>` <br><br> `transfer_input_files = <protocol>:///chtc/<path>` | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`|
 | `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) from the Execute Point (EP) | `transfer_output_files = ./output_files/results.txt` | |
 | `transfer_output_remaps = "<path_on_EP> = <new_path_on_AP>` | remaps (redirects or renames) output files explicitly listed to the Access Point (AP) on job completion | `transfer_output_remaps = "./results.txt = /staging/<user>/job1_results.txt` (saves results.txt as `/staging/<user>/job1_results.txt`) | |
 | `when_to_transfer_output = <ON_EXIT, ON_EXIT_OR_EVICT, or ON_SUCCESS>` | causes HTCondor to transfer job outputs based on the job's exit code | |
@@ -67,7 +70,7 @@ These commands display information about the execution points - machines that ex
 
 ## Controlling Where Your Job Runs
 
-These commands display information about the execution points - machines that execute/run jobs.
+These commands control where you job, runs on which Execution Points, and what your job requires to match/run
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
@@ -83,26 +86,25 @@ These commands display information about the execution points - machines that ex
 
 ## Controlling How Your Job Runs
 
-These commands display information about the execution points - machines that execute/run jobs.
+These commands control how your job environment behaves on the Execution Point (EP)
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
-| `universe = <vanilla or container>` | specifies which HTCondor universe environment to use when running this job | | `universe = vanilla` is the default HTCondor environment \ `universe = container` specifies an environment built for running container images (Docker, Apptainer, and Singularity)|
+| `universe = <vanilla or container>` | specifies which HTCondor universe environment to use when running this job | | `universe = vanilla` is the default HTCondor environment <br> `universe = container` specifies an environment built for running container images (Docker, Apptainer, and Singularity)|
 | `container_image = <image_file>` | specifies the path to a container image to be used | `container_image = docker://pytorch/pytorch:latest` | can pull image from DockerHub or load `.sif` files directly |
 {:.command-table}
 
 ## Queue Statement Options
 
-These commands display information about the execution points - machines that execute/run jobs.
+These commands control how many jobs (and what jobs) are queued by HTCondor for execution
 
 | Command | Use | Example | Notes |
 | --- | --- | --- | --- |
 | `queue` | command line queuing the job to start | | if no other options specified, a single job is queued |
 | `queue <int>` | places zero or more copies of the job into the HTCondor queue | `queue 10` | |
-| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./list.txt` |
-| `queue <int_expr>, <var2> from <list>` | lists all attributes of `execution_point` |
-| `queue <var1> <var> in [slice] <list>` | lists all attributes of `execution_point` |
-| `queue <var> matching <globbing_string>` | lists all attributes of `execution_point` |
+| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./listOfEmployeeNames.txt` | the `<var>` value(s) can be passed as an environment or argument variable |
+| `queue <var> in [slice] <list>` | places jobs in the queue using a python style slice selecting only some of the items in the list of items | `queue name in [5:18] ./listOfEmployeeNames.txt` | 
+| `queue <var> matching <globbing_string>` | places jobs in the queue based on globbing matches of files/directories in a path | `queue sampleID matching ./samples/sampleID_*` | |
 {:.command-table}
 
 ## Glossary
@@ -110,7 +112,6 @@ These commands display information about the execution points - machines that ex
 | Term | Meaning |
 | --- | --- |
 | access point | The machine which you log into to access CHTC's servers. This is the machine you use to prepare files for job submission and submit jobs. |
-| cluster ID | A unique number associated with a single job submission. |
 | error file / standard error | The file in which your job typically prints any errors. |
 | execution point | The machine that executes or runs your job. |
 | held/hold | A job state in which your job has stopped running due to an error. |

From 9c56b6897103557490d94168f32fee11943e069c Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Fri, 7 Mar 2025 09:10:34 -0600
Subject: [PATCH 06/13] Created layout2 for submit quick ref

---
 .../condor-submit-quick-ref_layout2.md        | 133 ++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100644 _uw-research-computing/condor-submit-quick-ref_layout2.md

diff --git a/_uw-research-computing/condor-submit-quick-ref_layout2.md b/_uw-research-computing/condor-submit-quick-ref_layout2.md
new file mode 100644
index 00000000..79ad09f8
--- /dev/null
+++ b/_uw-research-computing/condor-submit-quick-ref_layout2.md
@@ -0,0 +1,133 @@
+---
+layout: guide
+title: "Quick reference: HTCondor Submission commands"
+alt_title: "Quick reference: HTCondor Submission commands"
+guide:
+    category: Get started
+    tag: htc
+---
+
+## Introduction
+
+This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
+
+{% capture content %}
+- [Introduction](#introduction)
+- [Submit jobs](#submit-jobs)
+- [Basic Submit File Options](#basic-submit-file-options)
+- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
+- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
+- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
+- [Queue Statement Options](#queue-statement-options)
+- [Glossary](#glossary)
+- [Related Pages](#related-pages)
+{% endcapture %}
+{% include /components/directory.html title="Table of Contents" %}
+
+> **Please note the following!** 
+> 
+> * Bracketed items (`<>`) denote where to place your input. 
+>   Do not include the brackets in your command.
+> * All commands should be entered on a single line.
+
+## Submit jobs
+
+You can use these commands to submit jobs to HTCondor. 
+
+| Command | Use | Notes |
+| --- | --- | --- |
+| `condor_submit` <br><br> `condor_submit helloWorld.sub` | submits job(s) as specified by `submit_file` | See [monitor your jobs](condor_q) 
+| `condor_submit -i <submit_file>`| submits an interactive job as specified by `submit_file` |
+| `condor_submit <submit_file> -append <attribute>`| appends Condor commands to `submit_file` during submission| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. 
+{:.command-table}
+
+## Basic Submit File Options
+
+These commands specify the basic submit file options needed to start a job on HTCondor 
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `executable = <executable.sh>` | path to the executable script | `executable = helloWorld.py` | the executable script is automatically transferred to the Execution Point (EP) by HTCondor |
+| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` | arguments are wrapped by quotes(") and space separated |
+| `log = <job.log>` | denotes the path to the log file | `log = ./log_files/job1.log`| if not provided, no log file is recorded |
+| `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
+| `error = <job.err>` | path to file capturing `stderr` screen output | `output = ./log_files/job1.err` | can be merged with `stdout` by denoting the same path in `output = <path>` |
+{:.command-table}
+
+## Managing File Transfers in HTCondor
+
+These commands control how and when files are transferred during submission and following job ending 
+
+| Command | Use | Notes |
+| --- | --- | --- |
+| `transfer_input_files = [protocol]/<path> ` | lists all the input files to be transferred to the Execute Point (EP) | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///` <br><br>`transfer_input_files = <protocol>:///chtc/<path>` |
+| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) from the Execute Point (EP) | |
+| `transfer_output_remaps =`<br>`"<path> = <path>"` | remaps (redirects or renames) output files explicitly listed to the Access Point (AP) on job completion | |
+| `when_to_transfer_output = <event>` | causes HTCondor to transfer job outputs based on the job's exit code | select from `ON_EXIT, ON_EXIT_OR_EVICT, or ON_SUCCESS`|
+{:.command-table}
+
+## Requesting Resources for Your Job 
+
+These commands control where you job, runs on which Execution Points, and what your job requires to match/run
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `request_cpus = <int>` | requests number of CPUs (cores)|
+| `request_disk = <quantity>` | requests amount of disk space (Default in KB) | `request_disk = 40GB` | can denote kilobytes using `K` or `KB`, megabytes using `M` or `MB`, gigabytes using `G` or `GB`, or terrabytes using `T` or `TB` |
+| `request_memory = <quantity>` | requests amount of memory for job (Default in KB) | `request_memory = 250GB` | |
+| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested <br> See [CHTC GPU Jobs](gpu-jobs) |
+| `want_campus_pools = true` | also sends jobs to HTCondor Pools on UW Madison campus | | |
+| `want_ospool = true` | also sends jobs to Open Science Pool | | |
+| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested <br> See [CHTC GPU Jobs](gpu-jobs) |
+| `+WantGPULab = true` | also sends jobs to shared CHTC GPUs | | See [CHTC GPU Jobs](gpu-jobs) |
+| `gpus_minimum_capability = <version>` | minimum GPU Capability value | `gpus_minimum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
+| `gpus_maximmum_capability = <version>` | maximum GPU Capability value | `gpus_maximum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
+| `gpus_minimum_memory = <quantity in MB>` | minimum GPU memory | `gpus_minimum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
+| `gpus_minimum_runtime = <version>` | minimum GPU runtime (usually CUDA) version | `gpus_minimum_runtime = 9.1` | See [CHTC GPU Jobs](gpu-jobs) |
+| `requirements = <ClassAd Boolean>` | appends additional requirements to the job submission file | `Requirements = (OpSysAndVer == "RedHat9")`| See PAGEHERE for all requirement options
+{:.command-table2}
+
+## Controlling Your Job's Environment
+
+These commands control how your job environment behaves on the Execution Point (EP)
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `container_image = <image_file>` | specifies the path to a container image to be used | `container_image = docker://pytorch/pytorch:latest` | can pull image from DockerHub or load `.sif` files directly |
+| `environment = <parameter_list>` | lists environmental variables | `environment = "VARIABLE1=Value1 VAR2='hello world'"` | wrapped by quotes(") and space separated |
+{:.command-table}
+
+## Queue Statement Options
+
+These commands control how many jobs (and what jobs) are queued by HTCondor for execution
+
+| Command | Use | Example | Notes |
+| --- | --- | --- | --- |
+| `queue` | command line queuing the job to start | | if no other options specified, a single job is queued |
+| `queue <int>` | places zero or more copies of the job into the HTCondor queue | `queue 10` | |
+| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./listOfEmployeeNames.txt` | the `<var>` value(s) can be passed as an environment or argument variable |
+| `queue <var> in [slice] <list>` | places jobs in the queue using a python style slice selecting only some of the items in the list of items | `queue name in [5:18] ./listOfEmployeeNames.txt` | 
+| `queue <var> matching <globbing_string>` | places jobs in the queue based on globbing matches of files/directories in a path | `queue sampleID matching ./samples/sampleID_*` | |
+{:.command-table}
+
+## Glossary
+
+| Term | Meaning |
+| --- | --- |
+| access point | The machine which you log into to access CHTC's servers. This is the machine you use to prepare files for job submission and submit jobs. |
+| error file / standard error | The file in which your job typically prints any errors. |
+| execution point | The machine that executes or runs your job. |
+| held/hold | A job state in which your job has stopped running due to an error. |
+| idle | A job state in which your job has not yet matched to an execution point and hasn't started running yet. |
+| job ID | The unique number associated with a job. This consists of a cluster ID, followed by a period and a process ID. For example, the job ID `12345.0` has a cluster ID of `12345` and a process ID of `0`. |
+| log file | The file where HTCondor prints messages about your job's execution and resource usage. |
+| output file / standard out | The file where your job typically prints output. Any messages printed to the "screen" in a job will be saved in this file. |
+| process ID | A unique number associated with each job within a job submission. See [submit multiple jobs](multiple-jobs). |
+| running | A job state in which your job has matched to the execution point and is currently executing/running. |
+| submit file | A text-based file that tells HTCondor details about your job, including the commands to run, what files need to be transferred, where to save the outputs, and more. See the [job submission](htcondor-job-submission). |
+
+## Related Pages
+
+* [Job submission](htcondor-job-submission)
+* [Monitor your jobs](condor_q)
+* [HTCondor manual](https://htcondor.readthedocs.io/en/latest/)

From 031bdf943fb7b97a72a9501832bacd3435c469da Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 16 Jul 2025 18:14:53 -0500
Subject: [PATCH 07/13] Updated condor-submit-quick-ref.md

---
 .../condor-submit-quick-ref.md                | 120 ++++++++----------
 1 file changed, 56 insertions(+), 64 deletions(-)

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index e6fc8e72..bce48dca 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -32,96 +32,88 @@ This page lists common HTCondor commands and options for jobs. Users familiar wi
 
 ## Submit jobs
 
-You can use these commands to submit jobs to HTCondor. 
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `condor_submit` | submits job(s) as specified by `submit_file` | `condor_submit helloWorld.sub` | See [monitor your jobs](condor_q) 
-| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | `condor_submit -i helloWorld.sub`
-| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | `condor_submit helloWorld.sub -append "request_disk = 1GB"`| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. 
+| Command | Use | Notes and Examples                                                                                                                                                                                                                        |
+| --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `condor_submit` | submits job(s) as specified by `submit_file` | See [monitor your jobs](condor_q) <br><br> **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                                                |
+| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | **Example:** <br>`condor_submit -i helloWorld.sub`                                                                                                                                                                                        |
+| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | Commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. <br><br> **Example:** <br>`condor_submit helloWorld.sub -append "request_disk = 1GB"` |
 {:.command-table}
 
 ## Basic Submit File Options
 
-These commands specify the basic submit file options needed to start a job on HTCondor 
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` | arguments are wrapped by quotes(") and space separated |
-| `environment = <parameter_list>` | lists environmental variables | `environment = "VARIABLE1=Value1 VAR2='hello world'"` | wrapped by quotes(") and space separated |
-| `log = <job.log>` | denotes the path to the log file | `log = ./log_files/job1.log`| if not provided, no log file is recorded |
-| `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
-| `error = <job.err>` | path to file capturing `stderr` screen output | `output = ./log_files/job1.err` | can be merged with `stdout` by denoting the same path in `output = <path>` |
-| `executable = <executable.sh>` | path to the executable script | `executable = helloWorld.py` | the executable script is automatically transferred to the Execution Point (EP) by HTCondor |
-| `notification = <Always, Complete, Error, or None>` | notifies by e-mail when certain events occur | | **use carefully:* it can create an large # of inbound emails for multi-job HTCondor clusters |
+| Command | Use | Notes and Examples                                                                                                                                 |
+| --- | --- |----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `executable = <executable.sh>` | path to the executable script | The executable script is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
+| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes(") and space separated. <br><br> **Example:** <br>`arguments = "hello world"`                                      |
+| `environment = <parameter_list>` | lists environmental variables | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
+| `log = <job.log>` | denotes the path to the log file | If not provided, no log file is recorded. <br><br> **Example:** <br>`log = ./log_files/job1.log`                                                   |
+| `output = <job.out>` | path to file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = ./log_files/job1.out`               |
+| `error = <job.err>` | path to file capturing `stderr` screen output | Can be merged with `stdout` by denoting the same path in `output = <path>`. <br><br> **Example:** <br>`error = ./log_files/job1.err`               |
 {:.command-table}
 
 ## Managing File Transfers in HTCondor
 
-These commands control how and when files are transferred during submission and following job ending 
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | `transfer_input_files = /staging/<user>/<dir>` <br><br> `transfer_input_files = <protocol>:///chtc/<path>` | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`|
-| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) from the Execute Point (EP) | `transfer_output_files = ./output_files/results.txt` | |
-| `transfer_output_remaps = "<path_on_EP> = <new_path_on_AP>` | remaps (redirects or renames) output files explicitly listed to the Access Point (AP) on job completion | `transfer_output_remaps = "./results.txt = /staging/<user>/job1_results.txt` (saves results.txt as `/staging/<user>/job1_results.txt`) | |
-| `when_to_transfer_output = <ON_EXIT, ON_EXIT_OR_EVICT, or ON_SUCCESS>` | causes HTCondor to transfer job outputs based on the job's exit code | |
+| Command | Use | Notes and Examples                                                                                                                                                                                                                                                             |
+| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
+| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) | **Example:** <br><br>`transfer_output_files = ./results.txt`                                                                                                                                                                                                                   |
+| `transfer_output_remaps = "<path_on_EP>=<new_path_on_AP>"` | remaps output files to a new location on the AP upon job completion | File paths on the left side (e.g., ./results.txt) refer to locations on the Execution Point (EP), while remapped paths on the right side refer to the Access Point (AP). <br><br> **Example:** <br>`transfer_output_remaps = "./results.txt=/staging/<user>/job1_results.txt"` |
 {:.command-table}
 
 ## Controlling Where Your Job Runs
 
-These commands control where you job, runs on which Execution Points, and what your job requires to match/run
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `request_cpus = <int>` | requests number of CPUs (cores)|
-| `request_disk = <quantity>` | requests amount of disk space (Default in KB) | `request_disk = 40GB` | can denote kilobytes using `K` or `KB`, megabytes using `M` or `MB`, gigabytes using `G` or `GB`, or terrabytes using `T` or `TB` |
-| `request_memory = <quantity>` | requests amount of memory for job (Default in KB) | `request_memory = 250GB` | |
-| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested |
-| `requirements = <ClassAd Boolean>` | appends additional requirements to the job submission file | `Requirements = (OpSysAndVer == "RedHat9")`| See PAGEHERE for all requirement options
-| `gpus_minimum_capacity = <version>` | minimum GPU Capability value | `gpus_minimum_capability = 8.5` | 
-| `gpus_minimum_runtime = <version>` | minimum GPU runtime (usually CUDA) version | `gpus_minimum_runtime = 9.1` | 
-| `cuda_version = <version>` | specifies version of CUDA to be used | | superceeded by `gpus_minimum_runtime` |
+| Command                               | Use                                                 | Notes and Examples                                                                                                                                                                       |
+|---------------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `request_cpus = <int>`                | requests number of CPUs (cores)                     | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
+| `request_disk = <quantity>`           | requests disk space (Default in KB)                 | Can use units like `K`, `M`, `G`, or `T`. <br><br> **Example:** <br>`request_disk = 40GB`                                                                                                |
+| `request_memory = <quantity>`         | requests memory for job (Default in KB)             | **Example:** <br>`request_memory = 250GB`                                                                                                                                                |
+| `request_gpus = <int>`                | requests number of GPUs                             | If not specified, no GPUs requested.                                                                                                                                                     |
+| `requirements = <ClassAd Boolean>`    | sets job execution constraints                      | See [ClassAd reference](https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html). <br><br> **Example:** <br>`requirements = (OpSysAndVer == "RedHat9")` |
+| `gpus_minimum_capability = <version>` | sets minimum GPU capability to specify the GPU type | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
+| `gpus_maximum_capability = <version>` | sets maximum GPU capability to specify the GPU type | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
+| `gpus_minimum_memory = <quantity in MB>` | specifies the minimum GPU VRAM memory to request | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
+| `want_campus_pools = True`            | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) | Best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. <br><br> **May not include `HasCHTCStaging` in the requirements**                            |
+| `want_ospool = True`                  | Allows jobs to match to the national Open Science Pool (OSPool) | Best for short (<8hr) or checkpointable jobs. Significantly increases available compute slots. <br><br> **May not include `HasCHTCStaging` in the requirements**                         |
 {:.command-table}
 
-## Controlling How Your Job Runs
+_Note: For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
 
-These commands control how your job environment behaves on the Execution Point (EP)
+## Controlling How Your Job Runs
 
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `universe = <vanilla or container>` | specifies which HTCondor universe environment to use when running this job | | `universe = vanilla` is the default HTCondor environment <br> `universe = container` specifies an environment built for running container images (Docker, Apptainer, and Singularity)|
-| `container_image = <image_file>` | specifies the path to a container image to be used | `container_image = docker://pytorch/pytorch:latest` | can pull image from DockerHub or load `.sif` files directly |
+| Command | Use | Notes and Examples                                                                                                                |
+| --- | --- |-----------------------------------------------------------------------------------------------------------------------------------|
+| `container_image = <image_file>` | defines container image path | Can pull from DockerHub or use a local `.sif` file. <br><br> **Example:** <br>`container_image = docker://pytorch/pytorch:latest` |
 {:.command-table}
 
-## Queue Statement Options
+_Note: For more information on using containers in your jobs, please visit our [Using Apptainer Containers](apptainer-htc.md) guide._
 
-These commands control how many jobs (and what jobs) are queued by HTCondor for execution
+## Queue Statement Options
 
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `queue` | command line queuing the job to start | | if no other options specified, a single job is queued |
-| `queue <int>` | places zero or more copies of the job into the HTCondor queue | `queue 10` | |
-| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./listOfEmployeeNames.txt` | the `<var>` value(s) can be passed as an environment or argument variable |
-| `queue <var> in [slice] <list>` | places jobs in the queue using a python style slice selecting only some of the items in the list of items | `queue name in [5:18] ./listOfEmployeeNames.txt` | 
-| `queue <var> matching <globbing_string>` | places jobs in the queue based on globbing matches of files/directories in a path | `queue sampleID matching ./samples/sampleID_*` | |
+| Command | Use | Notes and Examples                                                                                                                         |
+| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------|
+| `queue` | queues a single job | If no other options specified, queues one job.                                                                                             |
+| `queue <int>` | queues multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
+| `queue <var> from <file>` | queues jobs using values from file | The `<var>` value(s) can be used as arguments/environment variables. <br><br> **Example:** <br>`queue name from ./listOfEmployeeNames.txt` |
+| `queue <var1>,<var2> from <file>` | queues jobs using multiple vars from file | **Example:** <br>`queue first, last from ./listOfEmployeeNames.txt`                                                                        |
+| `queue <var> in [slice] <list>` | queues jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] ./listOfEmployeeNames.txt`                                                                          |
+| `queue <var> matching <globbing_string>` | queues jobs from file pattern matches | **Example:** <br>`queue sampleID matching ./samples/sampleID_*`                                                                            |
 {:.command-table}
 
 ## Glossary
 
 | Term | Meaning |
 | --- | --- |
-| access point | The machine which you log into to access CHTC's servers. This is the machine you use to prepare files for job submission and submit jobs. |
-| error file / standard error | The file in which your job typically prints any errors. |
-| execution point | The machine that executes or runs your job. |
-| held/hold | A job state in which your job has stopped running due to an error. |
-| idle | A job state in which your job has not yet matched to an execution point and hasn't started running yet. |
-| job ID | The unique number associated with a job. This consists of a cluster ID, followed by a period and a process ID. For example, the job ID `12345.0` has a cluster ID of `12345` and a process ID of `0`. |
-| log file | The file where HTCondor prints messages about your job's execution and resource usage. |
-| output file / standard out | The file where your job typically prints output. Any messages printed to the "screen" in a job will be saved in this file. |
-| process ID | A unique number associated with each job within a job submission. See [submit multiple jobs](multiple-jobs). |
-| running | A job state in which your job has matched to the execution point and is currently executing/running. |
-| submit file | A text-based file that tells HTCondor details about your job, including the commands to run, what files need to be transferred, where to save the outputs, and more. See the [job submission](htcondor-job-submission). |
+| access point | The machine you log into for submitting jobs (e.g., CHTC login node). |
+| error file / standard error | The file where your job writes error messages. |
+| execution point | The machine where your job actually runs. |
+| held/hold | Job has encountered an issue and paused. |
+| idle | Job hasn't matched to an execution point yet. |
+| job ID | Unique ID made of `ClusterID.ProcID` like `12345.0`. |
+| log file | Tracks job events and resource usage. |
+| output file / standard out | File where job writes standard output (e.g., print statements). |
+| process ID | ID of an individual job in a job cluster. |
+| running | Job has matched and is currently running. |
+| submit file | File specifying everything needed to run your job. |
 
 ## Related Pages
 

From d5b5568b1ac25267b88abf6880d91829ccb08d85 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 16 Jul 2025 18:15:39 -0500
Subject: [PATCH 08/13] Deleted layout2

---
 .../condor-submit-quick-ref_layout2.md        | 133 ------------------
 1 file changed, 133 deletions(-)
 delete mode 100644 _uw-research-computing/condor-submit-quick-ref_layout2.md

diff --git a/_uw-research-computing/condor-submit-quick-ref_layout2.md b/_uw-research-computing/condor-submit-quick-ref_layout2.md
deleted file mode 100644
index 79ad09f8..00000000
--- a/_uw-research-computing/condor-submit-quick-ref_layout2.md
+++ /dev/null
@@ -1,133 +0,0 @@
----
-layout: guide
-title: "Quick reference: HTCondor Submission commands"
-alt_title: "Quick reference: HTCondor Submission commands"
-guide:
-    category: Get started
-    tag: htc
----
-
-## Introduction
-
-This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
-
-{% capture content %}
-- [Introduction](#introduction)
-- [Submit jobs](#submit-jobs)
-- [Basic Submit File Options](#basic-submit-file-options)
-- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
-- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
-- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
-- [Queue Statement Options](#queue-statement-options)
-- [Glossary](#glossary)
-- [Related Pages](#related-pages)
-{% endcapture %}
-{% include /components/directory.html title="Table of Contents" %}
-
-> **Please note the following!** 
-> 
-> * Bracketed items (`<>`) denote where to place your input. 
->   Do not include the brackets in your command.
-> * All commands should be entered on a single line.
-
-## Submit jobs
-
-You can use these commands to submit jobs to HTCondor. 
-
-| Command | Use | Notes |
-| --- | --- | --- |
-| `condor_submit` <br><br> `condor_submit helloWorld.sub` | submits job(s) as specified by `submit_file` | See [monitor your jobs](condor_q) 
-| `condor_submit -i <submit_file>`| submits an interactive job as specified by `submit_file` |
-| `condor_submit <submit_file> -append <attribute>`| appends Condor commands to `submit_file` during submission| commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. 
-{:.command-table}
-
-## Basic Submit File Options
-
-These commands specify the basic submit file options needed to start a job on HTCondor 
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `executable = <executable.sh>` | path to the executable script | `executable = helloWorld.py` | the executable script is automatically transferred to the Execution Point (EP) by HTCondor |
-| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | `arguments = "hello world` | arguments are wrapped by quotes(") and space separated |
-| `log = <job.log>` | denotes the path to the log file | `log = ./log_files/job1.log`| if not provided, no log file is recorded |
-| `output = <job.out>` | path to file capturing `stdout` screen output | `output = ./log_files/job1.out` | can be merged with `stderr` by denoting the same path in `error = <path>` |
-| `error = <job.err>` | path to file capturing `stderr` screen output | `output = ./log_files/job1.err` | can be merged with `stdout` by denoting the same path in `output = <path>` |
-{:.command-table}
-
-## Managing File Transfers in HTCondor
-
-These commands control how and when files are transferred during submission and following job ending 
-
-| Command | Use | Notes |
-| --- | --- | --- |
-| `transfer_input_files = [protocol]/<path> ` | lists all the input files to be transferred to the Execute Point (EP) | comma-separated list various file transfer protocols can be used in `transfer_input_files` including `file:///`, `osdf:///`, `pelican:///`, and `s3:///` <br><br>`transfer_input_files = <protocol>:///chtc/<path>` |
-| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) from the Execute Point (EP) | |
-| `transfer_output_remaps =`<br>`"<path> = <path>"` | remaps (redirects or renames) output files explicitly listed to the Access Point (AP) on job completion | |
-| `when_to_transfer_output = <event>` | causes HTCondor to transfer job outputs based on the job's exit code | select from `ON_EXIT, ON_EXIT_OR_EVICT, or ON_SUCCESS`|
-{:.command-table}
-
-## Requesting Resources for Your Job 
-
-These commands control where you job, runs on which Execution Points, and what your job requires to match/run
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `request_cpus = <int>` | requests number of CPUs (cores)|
-| `request_disk = <quantity>` | requests amount of disk space (Default in KB) | `request_disk = 40GB` | can denote kilobytes using `K` or `KB`, megabytes using `M` or `MB`, gigabytes using `G` or `GB`, or terrabytes using `T` or `TB` |
-| `request_memory = <quantity>` | requests amount of memory for job (Default in KB) | `request_memory = 250GB` | |
-| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested <br> See [CHTC GPU Jobs](gpu-jobs) |
-| `want_campus_pools = true` | also sends jobs to HTCondor Pools on UW Madison campus | | |
-| `want_ospool = true` | also sends jobs to Open Science Pool | | |
-| `request_gpus = <int>` | requests number of GPUs | | if not specified, no GPUs requested <br> See [CHTC GPU Jobs](gpu-jobs) |
-| `+WantGPULab = true` | also sends jobs to shared CHTC GPUs | | See [CHTC GPU Jobs](gpu-jobs) |
-| `gpus_minimum_capability = <version>` | minimum GPU Capability value | `gpus_minimum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
-| `gpus_maximmum_capability = <version>` | maximum GPU Capability value | `gpus_maximum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
-| `gpus_minimum_memory = <quantity in MB>` | minimum GPU memory | `gpus_minimum_capability = 8.5` | See [CHTC GPU Jobs](gpu-jobs) |
-| `gpus_minimum_runtime = <version>` | minimum GPU runtime (usually CUDA) version | `gpus_minimum_runtime = 9.1` | See [CHTC GPU Jobs](gpu-jobs) |
-| `requirements = <ClassAd Boolean>` | appends additional requirements to the job submission file | `Requirements = (OpSysAndVer == "RedHat9")`| See PAGEHERE for all requirement options
-{:.command-table2}
-
-## Controlling Your Job's Environment
-
-These commands control how your job environment behaves on the Execution Point (EP)
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `container_image = <image_file>` | specifies the path to a container image to be used | `container_image = docker://pytorch/pytorch:latest` | can pull image from DockerHub or load `.sif` files directly |
-| `environment = <parameter_list>` | lists environmental variables | `environment = "VARIABLE1=Value1 VAR2='hello world'"` | wrapped by quotes(") and space separated |
-{:.command-table}
-
-## Queue Statement Options
-
-These commands control how many jobs (and what jobs) are queued by HTCondor for execution
-
-| Command | Use | Example | Notes |
-| --- | --- | --- | --- |
-| `queue` | command line queuing the job to start | | if no other options specified, a single job is queued |
-| `queue <int>` | places zero or more copies of the job into the HTCondor queue | `queue 10` | |
-| `queue <var> from <list>` | places copies of the job in the queue based on the lines in a comma-separated list `<list>` or `<file>` | `queue name from ./listOfEmployeeNames.txt` | the `<var>` value(s) can be passed as an environment or argument variable |
-| `queue <var> in [slice] <list>` | places jobs in the queue using a python style slice selecting only some of the items in the list of items | `queue name in [5:18] ./listOfEmployeeNames.txt` | 
-| `queue <var> matching <globbing_string>` | places jobs in the queue based on globbing matches of files/directories in a path | `queue sampleID matching ./samples/sampleID_*` | |
-{:.command-table}
-
-## Glossary
-
-| Term | Meaning |
-| --- | --- |
-| access point | The machine which you log into to access CHTC's servers. This is the machine you use to prepare files for job submission and submit jobs. |
-| error file / standard error | The file in which your job typically prints any errors. |
-| execution point | The machine that executes or runs your job. |
-| held/hold | A job state in which your job has stopped running due to an error. |
-| idle | A job state in which your job has not yet matched to an execution point and hasn't started running yet. |
-| job ID | The unique number associated with a job. This consists of a cluster ID, followed by a period and a process ID. For example, the job ID `12345.0` has a cluster ID of `12345` and a process ID of `0`. |
-| log file | The file where HTCondor prints messages about your job's execution and resource usage. |
-| output file / standard out | The file where your job typically prints output. Any messages printed to the "screen" in a job will be saved in this file. |
-| process ID | A unique number associated with each job within a job submission. See [submit multiple jobs](multiple-jobs). |
-| running | A job state in which your job has matched to the execution point and is currently executing/running. |
-| submit file | A text-based file that tells HTCondor details about your job, including the commands to run, what files need to be transferred, where to save the outputs, and more. See the [job submission](htcondor-job-submission). |
-
-## Related Pages
-
-* [Job submission](htcondor-job-submission)
-* [Monitor your jobs](condor_q)
-* [HTCondor manual](https://htcondor.readthedocs.io/en/latest/)

From 1e283f5c2c7ab37a173fa5aed64d20075e5a5cba Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 16 Jul 2025 18:19:43 -0500
Subject: [PATCH 09/13] Created condor-submit-quick-ref.md

squashed old branch and restarted due to conflicts.
---
 .../condor-submit-quick-ref (2).md            | 122 ++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100644 _uw-research-computing/condor-submit-quick-ref (2).md

diff --git a/_uw-research-computing/condor-submit-quick-ref (2).md b/_uw-research-computing/condor-submit-quick-ref (2).md
new file mode 100644
index 00000000..bce48dca
--- /dev/null
+++ b/_uw-research-computing/condor-submit-quick-ref (2).md	
@@ -0,0 +1,122 @@
+---
+layout: guide
+title: "Quick reference: HTCondor Submission commands"
+alt_title: "Quick reference: HTCondor Submission commands"
+guide:
+    category: Get started
+    tag: htc
+---
+
+## Introduction
+
+This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
+
+{% capture content %}
+- [Introduction](#introduction)
+- [Submit jobs](#submit-jobs)
+- [Basic Submit File Options](#basic-submit-file-options)
+- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
+- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
+- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
+- [Queue Statement Options](#queue-statement-options)
+- [Glossary](#glossary)
+- [Related Pages](#related-pages)
+{% endcapture %}
+{% include /components/directory.html title="Table of Contents" %}
+
+> **Please note the following!** 
+> 
+> * Bracketed items (`<>`) denote where to place your input. 
+>   Do not include the brackets in your command.
+> * All commands should be entered on a single line.
+
+## Submit jobs
+
+| Command | Use | Notes and Examples                                                                                                                                                                                                                        |
+| --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `condor_submit` | submits job(s) as specified by `submit_file` | See [monitor your jobs](condor_q) <br><br> **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                                                |
+| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | **Example:** <br>`condor_submit -i helloWorld.sub`                                                                                                                                                                                        |
+| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | Commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. <br><br> **Example:** <br>`condor_submit helloWorld.sub -append "request_disk = 1GB"` |
+{:.command-table}
+
+## Basic Submit File Options
+
+| Command | Use | Notes and Examples                                                                                                                                 |
+| --- | --- |----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `executable = <executable.sh>` | path to the executable script | The executable script is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
+| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes(") and space separated. <br><br> **Example:** <br>`arguments = "hello world"`                                      |
+| `environment = <parameter_list>` | lists environmental variables | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
+| `log = <job.log>` | denotes the path to the log file | If not provided, no log file is recorded. <br><br> **Example:** <br>`log = ./log_files/job1.log`                                                   |
+| `output = <job.out>` | path to file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = ./log_files/job1.out`               |
+| `error = <job.err>` | path to file capturing `stderr` screen output | Can be merged with `stdout` by denoting the same path in `output = <path>`. <br><br> **Example:** <br>`error = ./log_files/job1.err`               |
+{:.command-table}
+
+## Managing File Transfers in HTCondor
+
+| Command | Use | Notes and Examples                                                                                                                                                                                                                                                             |
+| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
+| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) | **Example:** <br><br>`transfer_output_files = ./results.txt`                                                                                                                                                                                                                   |
+| `transfer_output_remaps = "<path_on_EP>=<new_path_on_AP>"` | remaps output files to a new location on the AP upon job completion | File paths on the left side (e.g., ./results.txt) refer to locations on the Execution Point (EP), while remapped paths on the right side refer to the Access Point (AP). <br><br> **Example:** <br>`transfer_output_remaps = "./results.txt=/staging/<user>/job1_results.txt"` |
+{:.command-table}
+
+## Controlling Where Your Job Runs
+
+| Command                               | Use                                                 | Notes and Examples                                                                                                                                                                       |
+|---------------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `request_cpus = <int>`                | requests number of CPUs (cores)                     | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
+| `request_disk = <quantity>`           | requests disk space (Default in KB)                 | Can use units like `K`, `M`, `G`, or `T`. <br><br> **Example:** <br>`request_disk = 40GB`                                                                                                |
+| `request_memory = <quantity>`         | requests memory for job (Default in KB)             | **Example:** <br>`request_memory = 250GB`                                                                                                                                                |
+| `request_gpus = <int>`                | requests number of GPUs                             | If not specified, no GPUs requested.                                                                                                                                                     |
+| `requirements = <ClassAd Boolean>`    | sets job execution constraints                      | See [ClassAd reference](https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html). <br><br> **Example:** <br>`requirements = (OpSysAndVer == "RedHat9")` |
+| `gpus_minimum_capability = <version>` | sets minimum GPU capability to specify the GPU type | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
+| `gpus_maximum_capability = <version>` | sets maximum GPU capability to specify the GPU type | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
+| `gpus_minimum_memory = <quantity in MB>` | specifies the minimum GPU VRAM memory to request | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
+| `want_campus_pools = True`            | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) | Best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. <br><br> **May not include `HasCHTCStaging` in the requirements**                            |
+| `want_ospool = True`                  | Allows jobs to match to the national Open Science Pool (OSPool) | Best for short (<8hr) or checkpointable jobs. Significantly increases available compute slots. <br><br> **May not include `HasCHTCStaging` in the requirements**                         |
+{:.command-table}
+
+_Note: For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
+
+## Controlling How Your Job Runs
+
+| Command | Use | Notes and Examples                                                                                                                |
+| --- | --- |-----------------------------------------------------------------------------------------------------------------------------------|
+| `container_image = <image_file>` | defines container image path | Can pull from DockerHub or use a local `.sif` file. <br><br> **Example:** <br>`container_image = docker://pytorch/pytorch:latest` |
+{:.command-table}
+
+_Note: For more information on using containers in your jobs, please visit our [Using Apptainer Containers](apptainer-htc.md) guide._
+
+## Queue Statement Options
+
+| Command | Use | Notes and Examples                                                                                                                         |
+| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------|
+| `queue` | queues a single job | If no other options specified, queues one job.                                                                                             |
+| `queue <int>` | queues multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
+| `queue <var> from <file>` | queues jobs using values from file | The `<var>` value(s) can be used as arguments/environment variables. <br><br> **Example:** <br>`queue name from ./listOfEmployeeNames.txt` |
+| `queue <var1>,<var2> from <file>` | queues jobs using multiple vars from file | **Example:** <br>`queue first, last from ./listOfEmployeeNames.txt`                                                                        |
+| `queue <var> in [slice] <list>` | queues jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] ./listOfEmployeeNames.txt`                                                                          |
+| `queue <var> matching <globbing_string>` | queues jobs from file pattern matches | **Example:** <br>`queue sampleID matching ./samples/sampleID_*`                                                                            |
+{:.command-table}
+
+## Glossary
+
+| Term | Meaning |
+| --- | --- |
+| access point | The machine you log into for submitting jobs (e.g., CHTC login node). |
+| error file / standard error | The file where your job writes error messages. |
+| execution point | The machine where your job actually runs. |
+| held/hold | Job has encountered an issue and paused. |
+| idle | Job hasn't matched to an execution point yet. |
+| job ID | Unique ID made of `ClusterID.ProcID` like `12345.0`. |
+| log file | Tracks job events and resource usage. |
+| output file / standard out | File where job writes standard output (e.g., print statements). |
+| process ID | ID of an individual job in a job cluster. |
+| running | Job has matched and is currently running. |
+| submit file | File specifying everything needed to run your job. |
+
+## Related Pages
+
+* [Job submission](htcondor-job-submission)
+* [Monitor your jobs](condor_q)
+* [HTCondor manual](https://htcondor.readthedocs.io/en/latest/)

From 3cec5c8df677ebd68c0a004aedd6ddcc668f4ca6 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Wed, 16 Jul 2025 18:20:31 -0500
Subject: [PATCH 10/13] Rename condor-submit-quick-ref (2).md to
 condor-submit-quick-ref.md

---
 ...{condor-submit-quick-ref (2).md => condor-submit-quick-ref.md} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename _uw-research-computing/{condor-submit-quick-ref (2).md => condor-submit-quick-ref.md} (100%)

diff --git a/_uw-research-computing/condor-submit-quick-ref (2).md b/_uw-research-computing/condor-submit-quick-ref.md
similarity index 100%
rename from _uw-research-computing/condor-submit-quick-ref (2).md
rename to _uw-research-computing/condor-submit-quick-ref.md

From f4540c4091c48fc52a477508d0a591cfdaba02f9 Mon Sep 17 00:00:00 2001
From: Daniel Morales <79028039+dmora127@users.noreply.github.com>
Date: Fri, 22 Aug 2025 11:58:50 -0500
Subject: [PATCH 11/13] Updated condor-submit-quick-ref.md

---
 .../condor-submit-quick-ref.md                | 122 ------------------
 1 file changed, 122 deletions(-)
 delete mode 100644 _uw-research-computing/condor-submit-quick-ref.md

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
deleted file mode 100644
index bce48dca..00000000
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ /dev/null
@@ -1,122 +0,0 @@
----
-layout: guide
-title: "Quick reference: HTCondor Submission commands"
-alt_title: "Quick reference: HTCondor Submission commands"
-guide:
-    category: Get started
-    tag: htc
----
-
-## Introduction
-
-This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
-
-{% capture content %}
-- [Introduction](#introduction)
-- [Submit jobs](#submit-jobs)
-- [Basic Submit File Options](#basic-submit-file-options)
-- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
-- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
-- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
-- [Queue Statement Options](#queue-statement-options)
-- [Glossary](#glossary)
-- [Related Pages](#related-pages)
-{% endcapture %}
-{% include /components/directory.html title="Table of Contents" %}
-
-> **Please note the following!** 
-> 
-> * Bracketed items (`<>`) denote where to place your input. 
->   Do not include the brackets in your command.
-> * All commands should be entered on a single line.
-
-## Submit jobs
-
-| Command | Use | Notes and Examples                                                                                                                                                                                                                        |
-| --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `condor_submit` | submits job(s) as specified by `submit_file` | See [monitor your jobs](condor_q) <br><br> **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                                                |
-| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | **Example:** <br>`condor_submit -i helloWorld.sub`                                                                                                                                                                                        |
-| `condor_submit <submit_file> -append <attribute>` | appends Condor commands to `submit_file` during submission | Commands are added inplace prior to the `queue` command while leaving the `submit_file` unedited. Can pass `queue` commands as appended, if needed. <br><br> **Example:** <br>`condor_submit helloWorld.sub -append "request_disk = 1GB"` |
-{:.command-table}
-
-## Basic Submit File Options
-
-| Command | Use | Notes and Examples                                                                                                                                 |
-| --- | --- |----------------------------------------------------------------------------------------------------------------------------------------------------|
-| `executable = <executable.sh>` | path to the executable script | The executable script is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
-| `arguments = "<args>"` | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes(") and space separated. <br><br> **Example:** <br>`arguments = "hello world"`                                      |
-| `environment = <parameter_list>` | lists environmental variables | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
-| `log = <job.log>` | denotes the path to the log file | If not provided, no log file is recorded. <br><br> **Example:** <br>`log = ./log_files/job1.log`                                                   |
-| `output = <job.out>` | path to file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = ./log_files/job1.out`               |
-| `error = <job.err>` | path to file capturing `stderr` screen output | Can be merged with `stdout` by denoting the same path in `output = <path>`. <br><br> **Example:** <br>`error = ./log_files/job1.err`               |
-{:.command-table}
-
-## Managing File Transfers in HTCondor
-
-| Command | Use | Notes and Examples                                                                                                                                                                                                                                                             |
-| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `transfer_input_files = ` | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
-| `transfer_output_files = ` | explicitly lists the path to files to be returned to the Access Point (AP) | **Example:** <br><br>`transfer_output_files = ./results.txt`                                                                                                                                                                                                                   |
-| `transfer_output_remaps = "<path_on_EP>=<new_path_on_AP>"` | remaps output files to a new location on the AP upon job completion | File paths on the left side (e.g., ./results.txt) refer to locations on the Execution Point (EP), while remapped paths on the right side refer to the Access Point (AP). <br><br> **Example:** <br>`transfer_output_remaps = "./results.txt=/staging/<user>/job1_results.txt"` |
-{:.command-table}
-
-## Controlling Where Your Job Runs
-
-| Command                               | Use                                                 | Notes and Examples                                                                                                                                                                       |
-|---------------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `request_cpus = <int>`                | requests number of CPUs (cores)                     | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
-| `request_disk = <quantity>`           | requests disk space (Default in KB)                 | Can use units like `K`, `M`, `G`, or `T`. <br><br> **Example:** <br>`request_disk = 40GB`                                                                                                |
-| `request_memory = <quantity>`         | requests memory for job (Default in KB)             | **Example:** <br>`request_memory = 250GB`                                                                                                                                                |
-| `request_gpus = <int>`                | requests number of GPUs                             | If not specified, no GPUs requested.                                                                                                                                                     |
-| `requirements = <ClassAd Boolean>`    | sets job execution constraints                      | See [ClassAd reference](https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html). <br><br> **Example:** <br>`requirements = (OpSysAndVer == "RedHat9")` |
-| `gpus_minimum_capability = <version>` | sets minimum GPU capability to specify the GPU type | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
-| `gpus_maximum_capability = <version>` | sets maximum GPU capability to specify the GPU type | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
-| `gpus_minimum_memory = <quantity in MB>` | specifies the minimum GPU VRAM memory to request | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
-| `want_campus_pools = True`            | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) | Best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. <br><br> **May not include `HasCHTCStaging` in the requirements**                            |
-| `want_ospool = True`                  | Allows jobs to match to the national Open Science Pool (OSPool) | Best for short (<8hr) or checkpointable jobs. Significantly increases available compute slots. <br><br> **May not include `HasCHTCStaging` in the requirements**                         |
-{:.command-table}
-
-_Note: For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
-
-## Controlling How Your Job Runs
-
-| Command | Use | Notes and Examples                                                                                                                |
-| --- | --- |-----------------------------------------------------------------------------------------------------------------------------------|
-| `container_image = <image_file>` | defines container image path | Can pull from DockerHub or use a local `.sif` file. <br><br> **Example:** <br>`container_image = docker://pytorch/pytorch:latest` |
-{:.command-table}
-
-_Note: For more information on using containers in your jobs, please visit our [Using Apptainer Containers](apptainer-htc.md) guide._
-
-## Queue Statement Options
-
-| Command | Use | Notes and Examples                                                                                                                         |
-| --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------|
-| `queue` | queues a single job | If no other options specified, queues one job.                                                                                             |
-| `queue <int>` | queues multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
-| `queue <var> from <file>` | queues jobs using values from file | The `<var>` value(s) can be used as arguments/environment variables. <br><br> **Example:** <br>`queue name from ./listOfEmployeeNames.txt` |
-| `queue <var1>,<var2> from <file>` | queues jobs using multiple vars from file | **Example:** <br>`queue first, last from ./listOfEmployeeNames.txt`                                                                        |
-| `queue <var> in [slice] <list>` | queues jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] ./listOfEmployeeNames.txt`                                                                          |
-| `queue <var> matching <globbing_string>` | queues jobs from file pattern matches | **Example:** <br>`queue sampleID matching ./samples/sampleID_*`                                                                            |
-{:.command-table}
-
-## Glossary
-
-| Term | Meaning |
-| --- | --- |
-| access point | The machine you log into for submitting jobs (e.g., CHTC login node). |
-| error file / standard error | The file where your job writes error messages. |
-| execution point | The machine where your job actually runs. |
-| held/hold | Job has encountered an issue and paused. |
-| idle | Job hasn't matched to an execution point yet. |
-| job ID | Unique ID made of `ClusterID.ProcID` like `12345.0`. |
-| log file | Tracks job events and resource usage. |
-| output file / standard out | File where job writes standard output (e.g., print statements). |
-| process ID | ID of an individual job in a job cluster. |
-| running | Job has matched and is currently running. |
-| submit file | File specifying everything needed to run your job. |
-
-## Related Pages
-
-* [Job submission](htcondor-job-submission)
-* [Monitor your jobs](condor_q)
-* [HTCondor manual](https://htcondor.readthedocs.io/en/latest/)

From 5d8812670489311207b0a7a11a55c23e80580b80 Mon Sep 17 00:00:00 2001
From: Amber Lim <xalim@wisc.edu>
Date: Fri, 22 Aug 2025 16:50:42 -0500
Subject: [PATCH 12/13] Restore revised condor_submit quick ref guide

---
 .../condor-submit-quick-ref.md                | 130 ++++++++++++++++++
 1 file changed, 130 insertions(+)
 create mode 100644 _uw-research-computing/condor-submit-quick-ref.md

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
new file mode 100644
index 00000000..1415e854
--- /dev/null
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -0,0 +1,130 @@
+---
+layout: guide
+title: "Quick reference: HTCondor Submission commands"
+alt_title: "Quick reference: HTCondor Submission commands"
+guide:
+    category: Get started
+    tag: htc
+---
+
+## Introduction
+
+This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
+
+{% capture content %}
+- [Introduction](#introduction)
+- [Submit jobs](#submit-jobs)
+- [Basic Submit File Options](#basic-submit-file-options)
+- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
+- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
+- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
+- [Queue Statement Options](#queue-statement-options)
+- [Glossary](#glossary)
+- [Related Pages](#related-pages)
+{% endcapture %}
+{% include /components/directory.html title="Table of Contents" %}
+
+> **Please note the following!** 
+> 
+> * Bracketed items (`<>`) denote where to place your input. 
+>   Do not include the brackets in your command.
+> * All commands should be entered on a single line.
+
+## Commands to submit jobs
+
+| Command | Use | Notes and Examples                                                                                                                                                                                                                        |
+| --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `condor_submit` | submits job(s) as specified by `submit_file` | See [monitor your jobs](htcondor-job-submission.md) <br><br> **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                              |
+| `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | **Example:** <br>`condor_submit -i helloWorld.sub`                                                                                                                                                                                        |
+{:.command-table}
+
+## Basic Submit File Options
+
+| Option                           | Use | Notes and Examples                                                                                                                                 |
+|----------------------------------| --- |----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `executable = <executable.sh>`   | path to the executable script | The executable script is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
+| `arguments = "<args>"`           | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes(") and space separated. Embed spaces using single quotes.<br><br> **Example:** <br>`arguments = "'hello world'"`   |
+| `environment = <parameter_list>` | lists environmental variables | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
+| `log = <job.log>`                | denotes the path to the log file | We always recommend specifying `log`. If `log` is not provided, no log file is written. <br><br> **Example:** <br>`log = ./log_files/job1.log` |
+| `output = <job.out>`             | path to file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = ./log_files/job1.out` |
+| `error = <job.err>`              | path to file capturing `stderr` screen output | Can be merged with `stdout` by denoting the same path in `output = <path>`. <br><br> **Example:** <br>`error = ./log_files/job1.err`  |
+{:.command-table}
+
+## File transfers options
+
+Visit our [file transfers guide](htc-file-transfer) for more details.
+
+| Option                                                     | Use | Notes and Examples                                                                                                                                                                                                                                                             |
+|------------------------------------------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `transfer_input_files = `                                  | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
+| `transfer_output_files = `                                 | explicitly lists the path to files to be returned to the Access Point (AP) | If this is not specified, HTCondor will only transfer new and changed **files** in the top-level directory of the Execution Point. <br><br> **Example:** <br><br>`transfer_output_files = results.txt`|
+| `transfer_output_remaps = "<path_on_EP>=<new_path_on_AP>"` | remaps output files to a new location on the AP upon job completion | File paths on the left side (e.g., ./results.txt) refer to locations on the Execution Point (EP), while remapped paths on the right side refer to the Access Point (AP). <br><br> **Example:** <br>`transfer_output_remaps = "results.txt=/staging/<user>/job1_results.txt"` |
+{:.command-table}
+
+## Request resources
+
+| Option                                   | Use                                                                  | Notes and Examples                                                                                                                                                                       |
+|------------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `request_cpus = <int>`                   | requests number of CPUs (cores)                                      | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
+| `request_disk = <quantity>`              | requests disk space (Default in KiB)                                 | Can use units like `K`, `M`, `G`, or `T`. <br><br> **Example:** <br>`request_disk = 40GB`                                                                                                |
+| `request_memory = <quantity>`            | requests memory for job (Default in MB)                              | **Example:** <br>`request_memory = 250GB`                                                                                                                                                |
+| `request_gpus = <int>`                   | requests number of GPUs                                              | If not specified, no GPUs requested.                                                                                                                                                     |
+| `requirements = <ClassAd Boolean>`       | sets job execution constraints                                       | See [ClassAd reference](https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html). <br><br> **Example:** <br>`requirements = (OpSysAndVer == "RedHat9")` |
+| `gpus_minimum_capability = <version>`    | sets minimum GPU capability to specify the GPU type                  | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
+| `gpus_maximum_capability = <version>`    | sets maximum GPU capability to specify the GPU type                  | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
+| `gpus_minimum_memory = <quantity in MB>` | specifies the minimum GPU VRAM memory to request                     | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
+
+{:.command-table}
+
+_Note: For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
+
+## Specifying the software environment
+
+| Option                           | Use | Notes and Examples                                                                                                                |
+|----------------------------------| --- |-----------------------------------------------------------------------------------------------------------------------------------|
+| `container_image = <image_file>` | defines container image path | Can pull from DockerHub or use a local `.sif` file. <br><br> **Example:** <br>`container_image = docker://pytorch/pytorch:latest` |
+{:.command-table}
+
+_Note: For more information on using containers in your jobs, please visit our [Using Apptainer Containers](apptainer-htc) or [Running HTC Jobs Using Docker Containers](docker-jobs) guide._
+
+## Options for submitting a list of jobs
+
+| Option                                   | Use | Notes and Examples                                                                                                                         |
+|------------------------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------|
+| `queue`                                  | queues a single job | If no other options specified, queues one job.                                                                                             |
+| `queue <int>`                            | queues multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
+| `queue <var> from <file>`                | queues jobs using values from file | The `<var>` value(s) can be used as arguments/environment variables. <br><br> **Example:** <br>`queue name from ./listOfEmployeeNames.txt` |
+| `queue <var1>,<var2> from <file>`        | queues jobs using multiple vars from file | **Example:** <br>`queue first, last from ./listOfEmployeeNames.txt`                                                                        |
+| `queue <var> in [slice] <list>`          | queues jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] ./listOfEmployeeNames.txt`                                                                          |
+| `queue <var> matching <globbing_string>` | queues jobs from file pattern matches | **Example:** <br>`queue sampleID matching ./samples/sampleID_*`                                                                            |
+{:.command-table}
+
+## Scale beyond local capacity
+| Option                     | Use | Notes and Examples                                                                                                                         |
+|----------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------|
+| `want_campus_pools = True` | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) | Best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. <br><br> **May not include `HasCHTCStaging` in the requirements**                            |
+| `want_ospool = True`       | Allows jobs to match to the national Open Science Pool (OSPool)      | Best for short (<8hr) or checkpointable jobs. Significantly increases available compute slots. <br><br> **May not include `HasCHTCStaging` in the requirements**                         |
+{:.command-table}
+
+## Glossary
+
+| Term | Meaning |
+| --- | --- |
+| access point | The machine you log into for submitting jobs (e.g., CHTC login node). |
+| error file / standard error | The file where your job writes error messages. |
+| execution point | The machine where your job actually runs. |
+| held/hold | Job has encountered an issue and paused. |
+| idle | Job hasn't matched to an execution point yet. |
+| job ID | Unique ID made of `ClusterID.ProcID` like `12345.0`. |
+| log file | Tracks job events and resource usage. |
+| output file / standard out | File where job writes standard output (e.g., print statements). |
+| process ID | ID of an individual job in a job cluster. |
+| running | Job has matched and is currently running. |
+| submit file | File specifying everything needed to run your job. |
+
+## Related Pages
+
+* [Practice: Submit HTC Jobs using HTCondor](htcondor-job-submission)
+* [Monitor your jobs](condor_q)
+* [HTCondor manual - condor_submit](https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html)
+

From ca1a4b04ea855660649a66d5a2dd7eec47428438 Mon Sep 17 00:00:00 2001
From: Amber Lim <xalim@wisc.edu>
Date: Mon, 25 Aug 2025 10:55:19 -0500
Subject: [PATCH 13/13] Revisions to condor_submit quick ref

---
 .../condor-submit-quick-ref.md                | 92 +++++++++++--------
 1 file changed, 54 insertions(+), 38 deletions(-)

diff --git a/_uw-research-computing/condor-submit-quick-ref.md b/_uw-research-computing/condor-submit-quick-ref.md
index 1415e854..16a5b29d 100644
--- a/_uw-research-computing/condor-submit-quick-ref.md
+++ b/_uw-research-computing/condor-submit-quick-ref.md
@@ -9,16 +9,18 @@ guide:
 
 ## Introduction
 
-This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading the full guides (linked) to understand the full context of each command or option.
+This page lists common HTCondor commands and options for jobs. Users familiar with HTCondor and job submission on CHTC's High Throughput Computing (HTC) system can use this page as a quick reference. For users who are just starting out, we suggest reading our linked guides to understand the full context of each command or option.
 
 {% capture content %}
 - [Introduction](#introduction)
-- [Submit jobs](#submit-jobs)
-- [Basic Submit File Options](#basic-submit-file-options)
-- [Managing File Transfers in HTCondor](#managing-file-transfers-in-htcondor)
-- [Controlling Where Your Job Runs](#controlling-where-your-job-runs)
-- [Controlling How Your Job Runs](#controlling-how-your-job-runs)
-- [Queue Statement Options](#queue-statement-options)
+- [Commands to submit jobs](#commands-to-submit-jobs)
+- [Basic submit file options](#basic-submit-file-options)
+- [Transfer files](#transfer-files)
+- [Request resources](#request-resources)
+- [Software environment](#software-environment)
+- [Submit multiple jobs](#submit-multiple-jobs)
+- [HTCondor default variables](#htcondor-default-variables)
+- [Scale beyond local capacity](#scale-beyond-local-capacity)
 - [Glossary](#glossary)
 - [Related Pages](#related-pages)
 {% endcapture %}
@@ -32,78 +34,92 @@ This page lists common HTCondor commands and options for jobs. Users familiar wi
 
 ## Commands to submit jobs
 
+See [job submission basics](htcondor-job-submission)
+
 | Command | Use | Notes and Examples                                                                                                                                                                                                                        |
 | --- | --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `condor_submit` | submits job(s) as specified by `submit_file` | See [monitor your jobs](htcondor-job-submission.md) <br><br> **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                              |
+| `condor_submit <submit_file>` | submits job(s) as specified by `submit_file` | **Example:** <br>`condor_submit helloWorld.sub`                                                                                                                              |
 | `condor_submit -i <submit_file>` | submits an interactive job as specified by `submit_file` | **Example:** <br>`condor_submit -i helloWorld.sub`                                                                                                                                                                                        |
 {:.command-table}
 
-## Basic Submit File Options
+## Basic submit file options
 
 | Option                           | Use | Notes and Examples                                                                                                                                 |
 |----------------------------------| --- |----------------------------------------------------------------------------------------------------------------------------------------------------|
-| `executable = <executable.sh>`   | path to the executable script | The executable script is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
-| `arguments = "<args>"`           | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes(") and space separated. Embed spaces using single quotes.<br><br> **Example:** <br>`arguments = "'hello world'"`   |
-| `environment = <parameter_list>` | lists environmental variables | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
-| `log = <job.log>`                | denotes the path to the log file | We always recommend specifying `log`. If `log` is not provided, no log file is written. <br><br> **Example:** <br>`log = ./log_files/job1.log` |
-| `output = <job.out>`             | path to file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = ./log_files/job1.out` |
-| `error = <job.err>`              | path to file capturing `stderr` screen output | Can be merged with `stdout` by denoting the same path in `output = <path>`. <br><br> **Example:** <br>`error = ./log_files/job1.err`  |
+| `executable = <executable.sh>`   | path to the executable script or binary | The executable is automatically transferred to the Execution Point (EP) by HTCondor. <br><br> **Example:** <br>`executable = helloWorld.py` |
+| `arguments = "<args>"`           | lists arguments to be passed to the executable as part of the command line | Arguments are wrapped by quotes (") and space separated. Embed spaces using single quotes.<br><br> **Example:** <br>`arguments = "--print 'hello world'"`   |
+| `log = <job.log>`                | denotes the path to the log file | We recommend always specifying `log` to help with troubleshooting. If `log` is not provided, no log file is written. <br><br> **Example:** <br>`log = log_files/job1.log` |
+| `output = <job.out>`             | denotes the path to the file capturing `stdout` screen output | Can be merged with `stderr` by denoting the same path in `error = <path>`. <br><br> **Example:** <br>`output = log_files/job1.out` |
+| `error = <job.err>`              | denotes the path to file capturing `stderr` screen output | **Example:** <br>`error = log_files/job1.err`  |
 {:.command-table}
 
-## File transfers options
+## Transfer files
 
-Visit our [file transfers guide](htc-file-transfer) for more details.
+Visit our [file transfer guide](htc-job-file-transfer) for more details.
 
 | Option                                                     | Use | Notes and Examples                                                                                                                                                                                                                                                             |
 |------------------------------------------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `transfer_input_files = `                                  | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
-| `transfer_output_files = `                                 | explicitly lists the path to files to be returned to the Access Point (AP) | If this is not specified, HTCondor will only transfer new and changed **files** in the top-level directory of the Execution Point. <br><br> **Example:** <br><br>`transfer_output_files = results.txt`|
-| `transfer_output_remaps = "<path_on_EP>=<new_path_on_AP>"` | remaps output files to a new location on the AP upon job completion | File paths on the left side (e.g., ./results.txt) refer to locations on the Execution Point (EP), while remapped paths on the right side refer to the Access Point (AP). <br><br> **Example:** <br>`transfer_output_remaps = "results.txt=/staging/<user>/job1_results.txt"` |
+| `transfer_input_files = <file1>, <file2>`                                  | lists all the input files to be transferred to the Execute Point (EP) | Comma-separated list. Various file transfer protocols can be used including `file:///`, `osdf:///`, `pelican:///`, and `s3:///`. <br><br> **Examples:** <br> `transfer_input_files = osdf:///chtc/staging/...`                                                                 |
+| `transfer_output_files = <file1>, <file2>`                                 | explicitly lists the path to files on the EP to be returned to the working directory on the AP. | If this is not specified, HTCondor will only transfer new and changed **files** in the top-level directory of the Execution Point. Subdirectories are not transferred. <br><br> **Example:**<br>`transfer_output_files = results.txt`|
+| `transfer_output_remaps = "<file>=<new_path>; <file2>=<new_path>"` | remaps output files to a specified path. Can be used for renaming files. | Delimited by semicolons. Can be used in conjunction with various file transfer protocols. <br><br> **Example:** <br>`transfer_output_remaps = "results.txt=osdf:///chtc/staging/<user>/job1_results.txt"` |
 {:.command-table}
 
 ## Request resources
 
 | Option                                   | Use                                                                  | Notes and Examples                                                                                                                                                                       |
 |------------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `request_cpus = <int>`                   | requests number of CPUs (cores)                                      | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
+| `request_cpus = <integer>`                   | requests number of CPUs (cores)                                      | **Example:** <br>`request_cpus = 4`                                                                                                                                                      |
 | `request_disk = <quantity>`              | requests disk space (Default in KiB)                                 | Can use units like `K`, `M`, `G`, or `T`. <br><br> **Example:** <br>`request_disk = 40GB`                                                                                                |
 | `request_memory = <quantity>`            | requests memory for job (Default in MB)                              | **Example:** <br>`request_memory = 250GB`                                                                                                                                                |
-| `request_gpus = <int>`                   | requests number of GPUs                                              | If not specified, no GPUs requested.                                                                                                                                                     |
+| `request_gpus = <integer>`                   | requests number of GPUs                                              | See our [GPU jobs guide](gpu-jobs). |
+| `gpus_minimum_capability = <version>`    | sets minimum GPU capability               | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
+| `gpus_maximum_capability = <version>`    | sets maximum GPU capability                | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
+| `gpus_minimum_memory = <quantity>` | requests minimum GPU VRAM memory (Default in MB)                 | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
 | `requirements = <ClassAd Boolean>`       | sets job execution constraints                                       | See [ClassAd reference](https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html). <br><br> **Example:** <br>`requirements = (OpSysAndVer == "RedHat9")` |
-| `gpus_minimum_capability = <version>`    | sets minimum GPU capability to specify the GPU type                  | **Example:** <br>`gpus_minimum_capability = 8.5`                                                                                                                                         |
-| `gpus_maximum_capability = <version>`    | sets maximum GPU capability to specify the GPU type                  | **Example:** <br>`gpus_maximum_capability = 9.0`                                                                                                                                         |
-| `gpus_minimum_memory = <quantity in MB>` | specifies the minimum GPU VRAM memory to request                     | **Example:** <br>`gpus_minimum_memory = 3200`                                                                                                                                            |
 
 {:.command-table}
 
-_Note: For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
+_For more information on submitting GPU jobs, please visit our [Using GPUs](gpu-jobs.md) guide._
 
-## Specifying the software environment
+## Software environment
 
 | Option                           | Use | Notes and Examples                                                                                                                |
 |----------------------------------| --- |-----------------------------------------------------------------------------------------------------------------------------------|
 | `container_image = <image_file>` | defines container image path | Can pull from DockerHub or use a local `.sif` file. <br><br> **Example:** <br>`container_image = docker://pytorch/pytorch:latest` |
+| `environment = <parameter_list>` | lists environmental variables for use in your jobs | Wrapped by quotes(") and space separated. <br><br> **Example:** <br>`environment = "VARIABLE1=Value1 VAR2='hello world'"`                          |
 {:.command-table}
 
 _Note: For more information on using containers in your jobs, please visit our [Using Apptainer Containers](apptainer-htc) or [Running HTC Jobs Using Docker Containers](docker-jobs) guide._
 
-## Options for submitting a list of jobs
+## Submit multiple jobs
+
+See our [multiple jobs guide](multiple-jobs#variables).
 
 | Option                                   | Use | Notes and Examples                                                                                                                         |
 |------------------------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------|
-| `queue`                                  | queues a single job | If no other options specified, queues one job.                                                                                             |
-| `queue <int>`                            | queues multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
-| `queue <var> from <file>`                | queues jobs using values from file | The `<var>` value(s) can be used as arguments/environment variables. <br><br> **Example:** <br>`queue name from ./listOfEmployeeNames.txt` |
-| `queue <var1>,<var2> from <file>`        | queues jobs using multiple vars from file | **Example:** <br>`queue first, last from ./listOfEmployeeNames.txt`                                                                        |
-| `queue <var> in [slice] <list>`          | queues jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] ./listOfEmployeeNames.txt`                                                                          |
-| `queue <var> matching <globbing_string>` | queues jobs from file pattern matches | **Example:** <br>`queue sampleID matching ./samples/sampleID_*`                                                                            |
+| `queue`                                  | submits a single job | If no other options specified, submits one job.                                                                                             |
+| `queue <int>`                            | submits multiple copies of the job | **Example:** <br>`queue 10`                                                                                                                |
+| `queue <var> from <file>`                | submits jobs using values from a file | The `<var>` value(s) can be used elsewhere in the submit file.<br><br> **Example:** <br>`queue name from listOfEmployeeNames.txt` |
+| `queue <var1>,<var2> from <file>`        | submits jobs using multiple vars from file | **Example:** <br>`queue first, last from listOfEmployeeNames.txt`                                                                        |
+| `queue <var> in [slice] <list>`          | submits jobs using Python-style slicing | **Example:** <br>`queue name in [5:18] listOfEmployeeNames.txt`                                                                          |
+| `queue <var> matching <globbing_string>` | submits jobs from file pattern matches | **Example:** <br>`queue sampleID matching samples/sampleID_*`                                                                            |
 {:.command-table}
 
+## HTCondor default variables
+
+| Option | Use | Notes and Examples | 
+| --- | --- | --- |
+| `$(Cluster)` | The unique job ID generated for each job submission. | **Example:** <br>`log = job_$(Cluster).log` generates a unique log file, i.e., `job_12345.log` |
+| `$(Process)` | The unique process ID generated for each job in a cluster of submissions. Starts at `0`. | **Example:** <br>`out = $(Cluster)_$(Process).out` generates a unique stdout file for each job, i.e., `12345_0.out`, `12345_1.out`, `12345_2.out` |
+
 ## Scale beyond local capacity
-| Option                     | Use | Notes and Examples                                                                                                                         |
-|----------------------------| --- |--------------------------------------------------------------------------------------------------------------------------------------------|
-| `want_campus_pools = True` | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) | Best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. <br><br> **May not include `HasCHTCStaging` in the requirements**                            |
-| `want_ospool = True`       | Allows jobs to match to the national Open Science Pool (OSPool)      | Best for short (<8hr) or checkpointable jobs. Significantly increases available compute slots. <br><br> **May not include `HasCHTCStaging` in the requirements**                         |
+
+These options are best for short (<8hr) or checkpointable jobs. Expands matching opportunities across campus. Read more about [scaling beyond local capacity](scaling-htc). **Do not include `HasCHTCStaging` in the requirements**.
+
+| Option | Use |
+| --- | --- | --- |
+| `want_campus_pools = true` | Allows jobs to run on other HTCondor pools on campus (e.g., UW Grid) |
+| `want_ospool = true`       | Allows jobs to match to the national Open Science Pool (OSPool)      |
 {:.command-table}
 
 ## Glossary