Skip to content

Commit f36262f

Browse files
mishaschwartzmishaschwartz
committed
review comments
1 parent 279b841 commit f36262f

File tree

8 files changed

+70
-70
lines changed

8 files changed

+70
-70
lines changed

‎CHANGES.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
- Introduce a scheduler job to delete old files that may accumulate over time
1919

20-
Creates the `optional-component-clean_old_files` job that deletes old THREDDS log files and WPS output files.
20+
Creates the `optional-component/clean_old_files` job that deletes old THREDDS log files and WPS output files.
2121
Allows individual cleanup jobs to be enabled for each of `raven`, `finch`, `hummingbird`, and `thredds` components.
2222
Allows the user to configure how old a file must be before it is deleted (age in days) and how to calculate the age
2323
of the file (time since last modified, time since last accessed, time since created).
@@ -434,7 +434,7 @@
434434
create additional jobs by adding them as custom components instead.
435435

436436
What about... ?
437-
- just schedule these jobs for a non-existant day linvalid arg that triggers usage messageike February 31st?
437+
- just schedule these jobs for a non-existant day like February 31st?
438438
- Answer: This would technically work but is not obvious to the user. It is better to make this explicit.
439439
- just set the schedule to the `'#'` string?
440440
- Answer: This is a hack that would work based on the specific way that the docker-crontab image sets schedules.

‎birdhouse/env.local.example‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -218,28 +218,28 @@ export GEOSERVER_ADMIN_PASSWORD="${__DEFAULT__GEOSERVER_ADMIN_PASSWORD}"
218218
# value is a string that conforms to the cron schedule format.
219219
#
220220
# Delete old WPS output files generated by the finch WPS component
221-
#export FINCH_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
222-
#export FINCH_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
223-
#export FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="5 4 * * 0" # weekly on Sunday at 4:05
224-
#export FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
221+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
222+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_TIME_MODE=atime
223+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_FREQUENCY="5 4 * * 0" # weekly on Sunday at 4:05
224+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_ENABLED=false
225225
#
226226
# Delete old WPS output files generated by the hummingbird WPS component
227-
#export HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
228-
#export HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
229-
#export HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
230-
#export HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
227+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
228+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_TIME_MODE=atime
229+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
230+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_ENABLED=false
231231
#
232232
# Delete old WPS output files generated by the raven WPS component
233-
#export RAVEN_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
234-
#export RAVEN_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
235-
#export RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
236-
#export RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
233+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
234+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_TIME_MODE=atime
235+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
236+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_ENABLED=false
237237
#
238238
# Delete old log files generated by the thredds component
239-
#export THREDDS_LOGS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
240-
#export THREDDS_LOGS_DELETE_FILES_TIME_MODE=mtime
241-
#export THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="20 4 * * 0" # weekly on Sunday at 4:20
242-
#export THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
239+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
240+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_TIME_MODE=mtime
241+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_FREQUENCY="20 4 * * 0" # weekly on Sunday at 4:20
242+
#export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_ENABLED=false
243243

244244
# These variables configure the scheduler-job-backup component
245245

‎birdhouse/optional-components/scheduler-job-clean_old_files/config/finch/config.yml.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- name: clean_old_files_finch
22
comment: clean old WPS output files generated by Finch
3-
schedule: '${FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY}'
4-
command: 'sh /clean-old-files.sh "${FINCH_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS}" "${FINCH_WPS_OUTPUTS_DELETE_FILES_TIME_MODE}" /wps_outputs/finch'
3+
schedule: '${SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_FREQUENCY}'
4+
command: 'sh /clean-old-files.sh "${SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_OLDER_THAN_DAYS}" "${SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_TIME_MODE}" /wps_outputs/finch'
55
dockerargs: >-
66
--rm --name scheduler-job-clean_old_files_finch
77
--volume ${COMPOSE_DIR}/optional-components/scheduler-job-clean_old_files/clean-old-files.sh:/clean-old-files.sh:ro

‎birdhouse/optional-components/scheduler-job-clean_old_files/config/hummingbird/config.yml.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- name: clean_old_files_hummingbird
22
comment: clean old WPS output files generated by Hummingbird
3-
schedule: '${HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY}'
4-
command: 'sh /clean-old-files.sh "${HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS}" "${HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_TIME_MODE}" /wps_outputs/hummingbird'
3+
schedule: '${SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_FREQUENCY}'
4+
command: 'sh /clean-old-files.sh "${SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_OLDER_THAN_DAYS}" "${SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_TIME_MODE}" /wps_outputs/hummingbird'
55
dockerargs: >-
66
--rm --name scheduler-job-clean_old_files_hummingbird
77
--volume ${COMPOSE_DIR}/optional-components/scheduler-job-clean_old_files/clean-old-files.sh:/clean-old-files.sh:ro

‎birdhouse/optional-components/scheduler-job-clean_old_files/config/raven/config.yml.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- name: clean_old_files_raven
22
comment: clean old WPS output files generated by Raven
3-
schedule: '${RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY}'
4-
command: 'sh /clean-old-files.sh "${RAVEN_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS}" "${RAVEN_WPS_OUTPUTS_DELETE_FILES_TIME_MODE}" /wps_outputs/raven'
3+
schedule: '${SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_FREQUENCY}'
4+
command: 'sh /clean-old-files.sh "${SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_OLDER_THAN_DAYS}" "${SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_TIME_MODE}" /wps_outputs/raven'
55
dockerargs: >-
66
--rm --name scheduler-job-clean_old_files_raven
77
--volume ${COMPOSE_DIR}/optional-components/scheduler-job-clean_old_files/clean-old-files.sh:/clean-old-files.sh:ro

‎birdhouse/optional-components/scheduler-job-clean_old_files/config/thredds/config.yml.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- name: clean_old_files_thredds
22
comment: clean old log files generated by Thredds
3-
schedule: '${THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY}'
4-
command: 'sh /clean-old-files.sh "${THREDDS_LOGS_DELETE_FILES_OLDER_THAN_DAYS}" "${THREDDS_LOGS_DELETE_FILES_TIME_MODE}" /thredds'
3+
schedule: '${SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_FREQUENCY}'
4+
command: 'sh /clean-old-files.sh "${SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_OLDER_THAN_DAYS}" "${SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_TIME_MODE}" /thredds'
55
dockerargs: >-
66
--rm --name scheduler-job-clean_old_files_thredds
77
--volume ${COMPOSE_DIR}/optional-components/scheduler-job-clean_old_files/clean-old-files.sh:/clean-old-files.sh:ro

‎birdhouse/optional-components/scheduler-job-clean_old_files/default.env‎

Lines changed: 32 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,29 +2,29 @@ export SCHEDULER_JOB_CLEAN_OLD_FILES_DOCKER=alpine # alpine contains find with -
22
export SCHEDULER_JOB_CLEAN_OLD_FILES_VERSION=3.21
33
export SCHEDULER_JOB_CLEAN_OLD_FILES_IMAGE='${SCHEDULER_JOB_CLEAN_OLD_FILES_DOCKER}:${SCHEDULER_JOB_CLEAN_OLD_FILES_VERSION}'
44

5-
export FINCH_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
6-
export FINCH_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
7-
export FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="5 4 * * 0" # weekly on Sunday at 4:05
8-
export FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
9-
export __SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_CONFIG_LOC='$( [ "${FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED}" = "true" ] && echo "config/finch/config.yml" )'
5+
export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
6+
export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_TIME_MODE=atime
7+
export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_FREQUENCY="5 4 * * 0" # weekly on Sunday at 4:05
8+
export SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_ENABLED=false
9+
export __SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_CONFIG_LOC='$( [ "${SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_ENABLED}" = "true" ] && echo "config/finch/config.yml" )'
1010

11-
export HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
12-
export HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
13-
export HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
14-
export HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
15-
export __SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_CONFIG_LOC='$( [ "${HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED}" = "true" ] && echo "config/hummingbird/config.yml" )'
11+
export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
12+
export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_TIME_MODE=atime
13+
export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_FREQUENCY="10 4 * * 0" # weekly on Sunday at 4:10
14+
export SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_ENABLED=false
15+
export __SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_CONFIG_LOC='$( [ "${SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_ENABLED}" = "true" ] && echo "config/hummingbird/config.yml" )'
1616

17-
export RAVEN_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
18-
export RAVEN_WPS_OUTPUTS_DELETE_FILES_TIME_MODE=atime
19-
export RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="15 4 * * 0" # weekly on Sunday at 4:15
20-
export RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
21-
export __SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_CONFIG_LOC='$( [ "${RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED}" = "true" ] && echo "config/raven/config.yml" )'
17+
export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
18+
export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_TIME_MODE=atime
19+
export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_FREQUENCY="15 4 * * 0" # weekly on Sunday at 4:15
20+
export SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_ENABLED=false
21+
export __SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_CONFIG_LOC='$( [ "${SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_ENABLED}" = "true" ] && echo "config/raven/config.yml" )'
2222

23-
export THREDDS_LOGS_DELETE_FILES_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
24-
export THREDDS_LOGS_DELETE_FILES_TIME_MODE=mtime
25-
export THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY="20 4 * * 0" # weekly on Sunday at 4:20
26-
export THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED=false
27-
export __SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_CONFIG_LOC='$( [ "${THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_ENABLED}" = "true" ] && echo "config/thredds/config.yml" )'
23+
export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_OLDER_THAN_DAYS= # unset by default if this job is enabled this must be set to an integer
24+
export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_TIME_MODE=mtime
25+
export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_FREQUENCY="20 4 * * 0" # weekly on Sunday at 4:20
26+
export SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_ENABLED=false
27+
export __SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_CONFIG_LOC='$( [ "${SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_ENABLED}" = "true" ] && echo "config/thredds/config.yml" )'
2828

2929
export DELAYED_EVAL="
3030
$DELAYED_EVAL
@@ -37,18 +37,18 @@ export DELAYED_EVAL="
3737

3838
OPTIONAL_VARS="
3939
$OPTIONAL_VARS
40-
\$FINCH_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS
41-
\$FINCH_WPS_OUTPUTS_DELETE_FILES_TIME_MODE
42-
\$FINCH_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY
43-
\$HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS
44-
\$HUMMINGBIRD_WPS_OUTPUTS_DELETE_FILES_TIME_MODE
45-
\$HUMMINGBIRD_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY
46-
\$RAVEN_WPS_OUTPUTS_DELETE_FILES_OLDER_THAN_DAYS
47-
\$RAVEN_WPS_OUTPUTS_DELETE_FILES_TIME_MODE
48-
\$RAVEN_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY
49-
\$THREDDS_LOGS_DELETE_FILES_OLDER_THAN_DAYS
50-
\$THREDDS_LOGS_DELETE_FILES_TIME_MODE
51-
\$THREDDS_SCHEDULER_JOB_CLEAN_OLD_FILES_FREQUENCY
40+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_OLDER_THAN_DAYS
41+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_TIME_MODE
42+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_FINCH_FREQUENCY
43+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_OLDER_THAN_DAYS
44+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_TIME_MODE
45+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_HUMMINGBIRD_FREQUENCY
46+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_OLDER_THAN_DAYS
47+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_TIME_MODE
48+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_RAVEN_FREQUENCY
49+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_OLDER_THAN_DAYS
50+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_TIME_MODE
51+
\$SCHEDULER_JOB_CLEAN_OLD_FILES_THREDDS_FREQUENCY
5252
"
5353

5454
VARS="

0 commit comments

Comments
 (0)