update stac, weaver, magpie, twitcher, cowbird - security fixes for EOL Python and http-related libraires #622
Conversation
…OL Python and http-related libraires
github-actions
bot
added
documentation
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3908/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca
|
|
run tests |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3940/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
mishaschwartz
left a comment
mishaschwartz
left a comment
There was a problem hiding this comment.
There's a change in the way that magpie creates cookies now that means that weaver cannot connect to magpie in the weaver/post-docker-compose script:
On line 150 of that script you currently have:
cookie_jar=$(echo "${cookie_jar}" | grep -v '# ' | grep -v -e '^$' | grep -v '_\.')where that last grep filters out lines with _. in them. This was supposed to deal with the fact that magpie was supplying two identical cookies for the domains:
#HttpOnly_.${BIRDHOUSE_FQDN_PUBLIC}#HttpOnly_${BIRDHOUSE_FQDN_PUBLIC}
Now it seems that magpie is only sending a cookie for the domain with the . (#HttpOnly_.${BIRDHOUSE_FQDN_PUBLIC}) which means that line 150 of weaver/post-docker-compose filters out the only provided cookie and then reports that it cannot connect to Magpie.
I'm not sure what the best solution is but we should either update weaver/post-docker-compose or we should investigate why magpie cookies have changed (was that intentional?)
|
@mishaschwartz I think this is caused by internal library changes (pyramid / urllib3) that behave slightly differently, notably when I guess the fix would be to consider the opposite |
|
@mishaschwartz @tlvu |
That would work. If you want to make it a bit more future-proof though you could just include all cookies from the jar. You'd just have to delimit the cookies with a
All IPs it looks like, not just on a dev server. |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3943/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
github-actions
bot
added
the
ci/deployment
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3945/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
mishaschwartz
left a comment
mishaschwartz
left a comment
There was a problem hiding this comment.
All of the actual component updates are working well! Thanks
The only issues are with the log changes to the weaver post-compose-up script and logging scripts. None of these are deal-breakers but they should be fixed so we can ensure that the logs are written nicely for this script and that the logging documentation is maintained.
| if [ ${delta_time} -ge "${WEAVER_WPS_PROVIDERS_MAX_TIME}" ]; then | ||
| msg="Failed to register all providers specified in WEAVER_WPS_PROVIDERS: [${prov}] is not responding." | ||
| printf "\n%s\n" "${ERROR}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | ||
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
There was a problem hiding this comment.
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | |
| echo | |
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
See above
| if [ ${ret} -ne 0 ] || [ "${code}" -ne 201 ]; then | ||
| printf "\n%s\n" "${WARN}Failed registration of remote WPS provider [${prov}] on [${prov_url}]${retry_msg}." | ||
| printf "Error:\n%s\n" "${body}" | ||
| print "\n" |
There was a problem hiding this comment.
print is not a shell function. I'd recommend echo to be consistent with the rest of my suggestions in this review but printf "\n" would work too.
There was a problem hiding this comment.
Typo. Yes, it was meant to be printf "\n"
I will use the standalone printf "\n" where you commented elsewhere since it is hard to track where the prefix \n applies across the lines when embedded within the log message.
| if [ ${delta_time} -ge "${WEAVER_WPS_PROVIDERS_MAX_TIME}" ]; then | ||
| msg="Failed to register all providers specified in WEAVER_WPS_PROVIDERS. Magpie is not responding." | ||
| printf "\n%s\n" "${ERROR}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | ||
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
There was a problem hiding this comment.
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | |
| echo | |
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
| printf " %s\n" "OK!" | ||
| else | ||
| printf "\n%s\n" "${ERROR}Failed administrative validation of Magpie token for Weaver WPS providers registration." | ||
| log ERROR "${PREFIX}Failed administrative validation of Magpie token for Weaver WPS providers registration." |
There was a problem hiding this comment.
| log ERROR "${PREFIX}Failed administrative validation of Magpie token for Weaver WPS providers registration." | |
| echo | |
| log ERROR "${PREFIX}Failed administrative validation of Magpie token for Weaver WPS providers registration." |
| if [ ${delta_time} -ge "${WEAVER_WPS_PROVIDERS_MAX_TIME}" ]; then | ||
| msg="Failed to register all providers specified in WEAVER_WPS_PROVIDERS. Weaver is not responding." | ||
| printf "\n%s\n" "${ERROR}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | ||
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
There was a problem hiding this comment.
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" | |
| echo | |
| log ERROR "${PREFIX}Timeout (${WEAVER_WPS_PROVIDERS_MAX_TIME}s)! ${msg}" |
| # register the new provider and validate | ||
| printf "\n%s" "${PREFIX}Registering remote WPS provider [${prov}] on [${prov_url}]${retry_msg}... " | ||
| payload="{\"id\": \"${prov}\", \"url\": \"${prov_url}\"}" | ||
| log DEBUG "${PREFIX}Registration payload: ${payload}" |
There was a problem hiding this comment.
| log DEBUG "${PREFIX}Registration payload: ${payload}" | |
| echo | |
| log DEBUG "${PREFIX}Registration payload: ${payload}" |
birdhouse/scripts/logging.include.sh
Outdated
| exit 2 | ||
| fi | ||
|
|
||
| # Usage: log {LEVEL} "{message}" [...] |
There was a problem hiding this comment.
Add info about new log options in the usage string and a quick sentence about what it does in the comment below.
| log_dest() { | ||
| level=$1 | ||
| end_line="\n" | ||
| if [ "$2" = "-n" ]; then |
There was a problem hiding this comment.
Add a description of this option in the documentation above this. You can refer to the documentation and usage for the log function.
github-actions
bot
added
ci/operations
|
@mishaschwartz Following your review, I realized that using |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3953/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca
|
mishaschwartz
left a comment
mishaschwartz
left a comment
There was a problem hiding this comment.
I've added my suggested fixes for the logging in #634
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3969/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/607/NOTEBOOK TEST RESULTS |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3973/Result 🆘 ABORTEDBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-216.rdext.crim.ca
|
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/3974/Result ❌ FAILUREBIRDHOUSE_DEPLOY_BRANCH : security-updates DACCS_IAC_BRANCH : master DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-91.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/611/NOTEBOOK TEST RESULTS |
|
@tlvu @mishaschwartz |
Overview
Update multiple components with corresponding updates of
urllib,requests, etc.At the same time, bump to Python 3.13 versions as applicable.
Changes
Non-breaking changes
STAC API: Security update to version 2.2.0
Weaver: Security update to version 6.8.0
Cowbird: Security update to version 4.3.0
Magpie: Security update to version 4.3.0
Twitcher: Security update to version 0.11.0
Weaver: Update
post-docker-compose-upscript.This can happen depending on specific internal HTTP libraries versions of the services.
To retain backward/forward compatibility, all cookies returned from Magpie are chained in following
curlcommands.logutility to report operations produced by the script rather than custom "echo level".Birdhouse: Allow
log <LEVEL> -n ...andlog <LEVEL> -p ...to generate log outputs without newline/prefixes.These options allow writing multiple log entries onto the same line for correct visual rendering of distinct
logcalls separated to allow some intermediate logic. The
logfunction invocations with these options respect thelog levels in order to make the messages consistent with enabled redirections and verbosity.
Breaking changes
CI Operations
birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false