Skip to content

Commit 683d8e4

Browse files
kpedro88kpedro88
authored
Merge pull request #31 from kpedro88/pipe_condor
Usage of condor within containers
2 parents c3b113c + a4b648f commit 683d8e4

File tree

3 files changed

+239
-0
lines changed

3 files changed

+239
-0
lines changed

‎README.md

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,115 @@
11
# lpc-scripts
22
scripts of use on the cmslpc cluster
33

4+
## `pipe_condor.sh`
5+
6+
HTCondor commands are installed on cmslpc interactive nodes, but by default they are not accessible inside containers.
7+
8+
The script [pipe_condor.sh](./pipe_condor.sh) enables calling HTCondor commands *on the host node* from inside a container.
9+
10+
### Usage
11+
12+
In your `.bashrc`:
13+
```bash
14+
source pipe_condor.sh
15+
```
16+
17+
Whenever you edit your `.bashrc`, you should log out and log back in for the changes to take effect.
18+
19+
To check if this line in your `.bashrc` is being executed when you log in, make sure the following command shows some output:
20+
```bash
21+
echo $APPTAINERENV_APPTAINER_ORIG
22+
```
23+
24+
Starting a container (the arguments are necessary for your `.bashrc` to be loaded inside the container):
25+
```bash
26+
cmssw-el7 -- /bin/bash
27+
```
28+
29+
### Details
30+
31+
What happens:
32+
* The `apptainer` command is replaced with a function that will create a set of pipes on the host node before running `apptainer`.
33+
* Inside the container, all executables starting with `condor_` will automatically run on the host node.
34+
* To run other commands on the host node, use `call_host cmd`, where `cmd` is the command you want to run (with any arguments).
35+
* Nested containers are supported (the enable/disable status (see "Options" just below) is inherited from the top-level container and cannot be changed)
36+
37+
Options:
38+
* Before sourcing the script in your `.bashrc`, you can add this line to change the directory where the pipes will be created (the default is `~/nobackup/pipes`):
39+
```bash
40+
export PIPE_CONDOR_DIR=your_dir
41+
```
42+
* If you want to disable this by default and only enable it on the fly, put this line in your `.bashrc`:
43+
```bash
44+
export PIPE_CONDOR_STATUS=${PIPE_CONDOR_STATUS:=disable}
45+
```
46+
Then to enable it temporarily:
47+
```bash
48+
PIPE_CONDOR_STATUS=enable cmssw-el7 ...
49+
```
50+
* Instead, if you have this enabled by default and you want to temporarily disable this for a specific container invocation:
51+
```bash
52+
PIPE_CONDOR_STATUS=disable cmssw-el7 ...
53+
````
54+
55+
Caveats:
56+
* cmslpc autodetection of the correct operating system for jobs is currently based on the host OS. Therefore, if you are submitting jobs in a container with a different OS, you will have to manually specify in your JDL file (the `X` in `condor_submit X`):
57+
```
58+
+DesiredOS = SL7
59+
```
60+
(other possible values are EL8 or EL9)
61+
* if you are running in a non-RHEL container, then you should manually set a different line in your JDL file:
62+
```
63+
+ApptainerImage = "/path/to/your/container"
64+
```
65+
* CMS connect support is planned, but has not been tested yet.
66+
67+
## `bind_condor.sh`
68+
69+
It is also possible to use the HTCondor Python bindings inside a container.
70+
This requires correctly specifying the HTCondor configuration.
71+
A simple approach is provided in [bind_condor.sh](./bind_condor.sh).
72+
73+
### Usage
74+
75+
In your `.bashrc`:
76+
```bash
77+
source bind_condor.sh
78+
```
79+
That's it!
80+
81+
### Setting up bindings
82+
83+
You will also need to have the HTCondor Python bindings installed in your working environment.
84+
85+
For newer CMSSW versions, the installation procedure is simple:
86+
```bash
87+
cmsrel CMSSW_X_Y_Z
88+
cd CMSSW_X_Y_Z/src
89+
cmsenv
90+
scram-venv
91+
cmsenv
92+
pip3 install htcondor
93+
```
94+
(Click [here](http://cms-sw.github.io/venv.html) to learn more about `scram-venv`)
95+
96+
For `CMSSW_10_6_X`, the Run 2 ultra-legacy analysis release that is only available for EL7 operating systems, there are some extra steps:
97+
```bash
98+
cmsrel CMSSW_10_6_30
99+
cd CMSSW_10_6_30/src
100+
cmsenv
101+
scram-venv
102+
cmsenv
103+
pip3 install --upgrade pip
104+
cmsenv
105+
pip3 install --upgrade htcondor==10.3.0
106+
```
107+
In this particular case, it is necessary to upgrade `pip` and install a specific version of the bindings
108+
because the Python version in `CMSSW_10_6_X` is old (Python 3.6.4).
109+
110+
**NOTE**: These recipes only install the bindings for Python3. (Python2 was still the default in `CMSSW_10_6_X`.)
111+
You will need to make sure any scripts using the bindings are compatible with Python3.
112+
4113
## Unit and Integration testing
5114
6115
### Automated

‎bind_condor.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
3+
BIND_CONDOR_CONFIG=/etc/condor/config.d/01_cmslpc_interactive
4+
BIND_CONDOR_PY=/usr/local/bin/cmslpc-local-conf.py
5+
export APPTAINER_BIND=${APPTAINER_BIND}${APPTAINER_BIND:+,}${BIND_CONDOR_CONFIG},${BIND_CONDOR_PY}
6+
7+
export APPTAINERENV_CONDOR_CONFIG=/etc/condor/config.d/01_cmslpc_interactive

‎pipe_condor.sh

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/bin/bash
2+
# shellcheck disable=SC2155
3+
4+
# default values
5+
# shellcheck disable=SC2076
6+
if [ -z "$PIPE_CONDOR_STATUS" ]; then
7+
export PIPE_CONDOR_STATUS=enable
8+
elif [[ ! " enable disable " =~ " $PIPE_CONDOR_STATUS " ]]; then
9+
echo "Warning: unsupported value $PIPE_CONDOR_STATUS for PIPE_CONDOR_STATUS; disabling"
10+
export PIPE_CONDOR_STATUS=disable
11+
fi
12+
if [ -z "$PIPE_CONDOR_DIR" ]; then
13+
export PIPE_CONDOR_DIR=~/nobackup/pipes
14+
fi
15+
export PIPE_CONDOR_DIR=$(readlink -f "$PIPE_CONDOR_DIR")
16+
mkdir -p "$PIPE_CONDOR_DIR"
17+
# ensure the pipe dir is bound
18+
export APPTAINER_BIND=${APPTAINER_BIND}${APPTAINER_BIND:+,}${PIPE_CONDOR_DIR}
19+
20+
# concept based on https://stackoverflow.com/questions/32163955/how-to-run-shell-script-on-host-from-docker-container
21+
22+
# execute command sent to host pipe; send output to container pipe; store exit code
23+
listenhost(){
24+
# stop when host pipe is removed
25+
while [ -e "$1" ]; do
26+
# "|| true" is necessary to stop "Interrupted system call"
27+
# must be *inside* eval to ensure EOF once command finishes
28+
# now replaced with assignment of exit code to local variable (which also returns true)
29+
tmpexit=0
30+
eval "$(cat "$1") || tmpexit="'$?' >& "$2"
31+
echo "$tmpexit" > "$3"
32+
done
33+
}
34+
export -f listenhost
35+
36+
# creates randomly named pipe and prints the name
37+
makepipe(){
38+
PREFIX="$1"
39+
PIPETMP=${PIPE_CONDOR_DIR}/${PREFIX}_$(uuidgen)
40+
mkfifo "$PIPETMP"
41+
echo "$PIPETMP"
42+
}
43+
export -f makepipe
44+
45+
# to be run on host before launching each apptainer session
46+
startpipe(){
47+
HOSTPIPE=$(makepipe HOST)
48+
CONTPIPE=$(makepipe CONT)
49+
EXITPIPE=$(makepipe EXIT)
50+
# export pipes to apptainer
51+
echo "export APPTAINERENV_HOSTPIPE=$HOSTPIPE; export APPTAINERENV_CONTPIPE=$CONTPIPE; export APPTAINERENV_EXITPIPE=$EXITPIPE"
52+
}
53+
export -f startpipe
54+
55+
# sends function to host, then listens for output, and provides exit code from function
56+
call_host(){
57+
if [ "${FUNCNAME[0]}" = "call_host" ]; then
58+
FUNCTMP=
59+
else
60+
FUNCTMP="${FUNCNAME[0]}"
61+
fi
62+
echo "cd $PWD; $FUNCTMP $*" > "$HOSTPIPE"
63+
cat < "$CONTPIPE"
64+
return "$(cat < "$EXITPIPE")"
65+
}
66+
export -f call_host
67+
68+
# from https://stackoverflow.com/questions/1203583/how-do-i-rename-a-bash-function
69+
copy_function() {
70+
test -n "$(declare -f "$1")" || return
71+
eval "${_/$1/$2}"
72+
eval "export -f $2"
73+
}
74+
export -f copy_function
75+
76+
if [ -z "$APPTAINER_ORIG" ]; then
77+
export APPTAINER_ORIG=$(which apptainer)
78+
fi
79+
# always set this (in case of nested containers)
80+
export APPTAINERENV_APPTAINER_ORIG=$APPTAINER_ORIG
81+
82+
apptainer(){
83+
if [ "$PIPE_CONDOR_STATUS" = "disable" ]; then
84+
(
85+
# shellcheck disable=SC2030
86+
export APPTAINERENV_PIPE_CONDOR_STATUS=disable
87+
$APPTAINER_ORIG "$@"
88+
)
89+
else
90+
# in subshell to contain exports
91+
(
92+
# shellcheck disable=SC2031
93+
export APPTAINERENV_PIPE_CONDOR_STATUS=enable
94+
# only start pipes on host
95+
# i.e. don't create more pipes/listeners for nested containers
96+
if [ -z "$APPTAINER_CONTAINER" ]; then
97+
eval "$(startpipe)"
98+
listenhost "$APPTAINERENV_HOSTPIPE" "$APPTAINERENV_CONTPIPE" "$APPTAINERENV_EXITPIPE" &
99+
LISTENER=$!
100+
fi
101+
# actually run apptainer
102+
$APPTAINER_ORIG "$@"
103+
# avoid dangling cat process after exiting container
104+
# (again, only on host)
105+
if [ -z "$APPTAINER_CONTAINER" ]; then
106+
pkill -P "$LISTENER"
107+
rm -f "$APPTAINERENV_HOSTPIPE" "$APPTAINERENV_CONTPIPE" "$APPTAINERENV_EXITPIPE"
108+
fi
109+
)
110+
fi
111+
}
112+
export -f apptainer
113+
114+
# on host: get list of condor executables
115+
if [ -z "$APPTAINER_CONTAINER" ]; then
116+
export APPTAINERENV_HOSTFNS=$(compgen -c | grep ^condor_)
117+
# in container: replace with call_host versions
118+
elif [ "$PIPE_CONDOR_STATUS" = "enable" ]; then
119+
# shellcheck disable=SC2153
120+
for HOSTFN in $HOSTFNS; do
121+
copy_function call_host "$HOSTFN"
122+
done
123+
fi

0 commit comments

Comments
 (0)