Retrieve the CR3 download secrets with 1PW #1262
Conversation
mddilley
requested review from
frankhereford,
patrickm02L,
chiaberry,
johnclary,
tillyw,
roseeichelmann and
RR216
| # ask user for a set of valid cookies for requests to the CRIS website | ||
| CRIS_BROWSER_COOKIES = input( | ||
| "Please login to CRIS and extract the contents of the Cookie: header and please paste it here:" | ||
| ) |
There was a problem hiding this comment.
I agree with you that it's a real bummer that this didn't work out on account of the cookie validation system they have in place. I really liked your work on bringing this into Airflow. 😢
But in much brighter news: I used this branch this morning to download the CR3s in production and it worked like a champ. I have a minor suggestion about the docker image, but already as this is, it's an easier daily task thanks to your docker compose stack usage. Big 👍 from me!
atd-etl/cr3_download/README.md
Outdated
| $ docker compose run -it cr3_download python cr3_download.py | ||
| $ docker compose run --build -it cr3_download python cr3_download.py |
There was a problem hiding this comment.
What would you think of adding CMD python /app/cr3_download.py as the last line of the Dockerfile so that the daily invocation becomes docker compose run?
There was a problem hiding this comment.
Nice! ✨ This is a great idea. I'll add this right now. 🙏
There was a problem hiding this comment.
I was able to build the docker image and paste the 🍪 in. Got a List of 0 crashes needing CR3 download: [] so I don't think I actually tested the 1pass integration because I didn't try to upload anything to S3 or use hasura?
It is in your checklist but I use these docs when I'm on cr-3 duty so those will need to be updated.
🚢 
|
@Charlie-Henry thanks for taking a look at this! Thanks for pointing to those docs too 🙏 And, true, testing without missing pdfs doesn't fully test this. I'm taking Frank's test this morning as confirmation that this will work going forward. ✅ |
|
Okie dokie - docs are updated and the rest of the checklist is done. Merging! |
Associated issues
cityofaustin/atd-data-tech#12849
Sorry again for all the noise on this one y'all - I think we learned a lot on the way, though. This removes the part of the GitHub Action workflow that pushed updated images of this code for Airflow to run. It also removes the
ATD_ETL_CONFIGandconfig.pyfile because it was easier to replace those references withos.getenvthan it was to resolve when env vars were set in the script and when they were retrieved in that module. The.envonly requires your 1PW connect server token, vault id, and 1PW connect server endpoint now.Testing
URL to test:
Steps to test:
cdto theatd-etl/cr3_downloadfolder in this branchShip list
CRIS CR3 Download Cookieentry from the API Accessible Secrets vault in 1PWatddocker/vz-cr3-downloadimage from DockerHub