Code

GH action to generate report#199

Code

Draft

asmacdo wants to merge 96 commits intomaindandi/dandi-hub:mainfrom cron-data-usage-reportdandi/dandi-hub:cron-data-usage-reportCopy head branch name to clipboard

+538-63

Commits

Commits on Jan 17, 2025

• Inital commit to add GH action to generate report

  asmacdo

  committed
  30e9cbf

  Copy full SHA for 30e9cbf

• Assume Jupyterhub Provisioning Role

  asmacdo

  committed
  3bcba91

  Copy full SHA for 3bcba91

• Fixup: indent

  asmacdo

  committed
  b5cdcf3

  Copy full SHA for b5cdcf3

• Rename job

  asmacdo

  committed
  6e118cc

  Copy full SHA for 6e118cc

• Add assumed role to update-kubeconfig

  asmacdo

  committed
  5062f08

  Copy full SHA for 5062f08

• No need to add ProvisioningRole to masters

  asmacdo

  committed
  d21a3a9

  Copy full SHA for d21a3a9

• Deploy a pod to the cluster, and schedule with Karpenter

  asmacdo

  committed
  403028f

  Copy full SHA for 403028f

• Fixup: correct path to pod manifest

  asmacdo

  committed
  92b9925

  Copy full SHA for 92b9925

• Fixup again ugh, rename file

  asmacdo

  committed
  478a31f

  Copy full SHA for 478a31f

• Delete Pod even if previous step times out

  Show description for 9db914e
  asmacdo

  committed
  9db914e

  Copy full SHA for 9db914e

• Hack out initial du

  asmacdo

  committed
  8458d01

  Copy full SHA for 8458d01

• tmp comment out job deployment, test dockerhub build

  asmacdo

  committed
  7999455

  Copy full SHA for 7999455

• Fixup hyphens for image name

  asmacdo

  committed
  d2e65de

  Copy full SHA for d2e65de

• Write file to output location

  asmacdo

  committed
  5e9e7df

  Copy full SHA for 5e9e7df

• use kubectl cp to retrieve report

  asmacdo

  committed
  d33973c

  Copy full SHA for d33973c

• Combine run blocks to use vars

  asmacdo

  committed
  98fecbc

  Copy full SHA for 98fecbc

• Mount efs and pass arg to du script

  asmacdo

  committed
  40ae0e8

  Copy full SHA for 40ae0e8

• Comment out repo pushing, lets see if the report runs

  asmacdo

  committed
  4c978f7

  Copy full SHA for 4c978f7

• Restrict job to asmacdo for testing

  asmacdo

  committed
  6bd7b82

  Copy full SHA for 6bd7b82

• Sanity check. Just list the directories

  asmacdo

  committed
  73c3e80

  Copy full SHA for 73c3e80

• Job was deployed, but never assigned to node, back to sanity check

  asmacdo

  committed
  685dfb1

  Copy full SHA for 685dfb1

• change from job to pod

  asmacdo

  committed
  f6afefc

  Copy full SHA for f6afefc

• deploy pod to same namespace as pvc

  asmacdo

  committed
  6dad759

  Copy full SHA for 6dad759

• Use ns in action

  asmacdo

  committed
  3a33937

  Copy full SHA for 3a33937

• increase timeout to 60s

  Show description for 1ffb1c9
  asmacdo

  committed
  1ffb1c9

  Copy full SHA for 1ffb1c9

• fixup: image name in manifest

  asmacdo

  committed
  58e0753

  Copy full SHA for 58e0753

• increase timeout to 150

  Show description for 6767755
  asmacdo

  committed
  6767755

  Copy full SHA for 6767755

• override entrypoint so i can debug with exec

  asmacdo

  committed
  cbf951e

  Copy full SHA for cbf951e

• bound /home actually meant path was /home/home/asmacdo

  asmacdo

  committed
  59eb045

  Copy full SHA for 59eb045

• Create output dir prior to writing report

  asmacdo

  committed
  db140d5

  Copy full SHA for db140d5

• pod back to job

  asmacdo

  committed
  f90176a

  Copy full SHA for f90176a

• Fixup use the correct job api

  asmacdo

  committed
  c31ccdd

  Copy full SHA for c31ccdd

• Add namespace to pod retrieval

  asmacdo

  committed
  3ee9d9f

  Copy full SHA for 3ee9d9f

• write directly to pv to test job

  asmacdo

  committed
  d7f81ba

  Copy full SHA for d7f81ba

• fixup script fstring

  asmacdo

  committed
  0856baa

  Copy full SHA for 0856baa

• no retry on failure, we were spinning up 5 pods, lets just fail 1 time

  asmacdo

  committed
  5301b1b

  Copy full SHA for 5301b1b

• Fixup backup limit job not template

  asmacdo

  committed
  7384274

  Copy full SHA for 7384274

• Initial report

  asmacdo

  committed
  8e81e38

  Copy full SHA for 8e81e38

• disable report

  Show description for cb5db49
  asmacdo

  committed
  cb5db49

  Copy full SHA for cb5db49

• deploy ec2 instance directly

  asmacdo

  committed
  5d188a7

  Copy full SHA for 5d188a7

• Update AMI image

  asmacdo

  committed
  2f39e9c

  Copy full SHA for 2f39e9c

• update sg and subnet

  asmacdo

  committed
  3a21106

  Copy full SHA for 3a21106

• terminate even if job fails

  asmacdo

  committed
  6a54da0

  Copy full SHA for 6a54da0

• debug: print public ip

  asmacdo

  committed
  87075fb

  Copy full SHA for 87075fb

• explicitly allocate public ip for ec2 instance

  asmacdo

  committed
  48c7f35

  Copy full SHA for 48c7f35

• Add WIP scripts

  asmacdo

  committed
  743359e

  Copy full SHA for 743359e

• rm old unused

  asmacdo

  committed
  0ba12f2

  Copy full SHA for 0ba12f2

• initial commit of scripts

  asmacdo

  committed
  2893ab2

  Copy full SHA for 2893ab2

• clean up launch script

  asmacdo

  committed
  5ef8f80

  Copy full SHA for 5ef8f80

• make scripe executable

  asmacdo

  committed
  b02720e

  Copy full SHA for b02720e

• fixup cleanup script

  asmacdo

  committed
  ae98909

  Copy full SHA for ae98909

• add a name to elastic ip (for easier manual cleanup)

  asmacdo

  committed
  7e80e4a

  Copy full SHA for 7e80e4a

• Exit on fail

  asmacdo

  committed
  f2a4116

  Copy full SHA for f2a4116

• Add permission for aws ec2 wait instance-status-ok

  asmacdo

  committed
  6ffef17

  Copy full SHA for 6ffef17

• Upload scripts to instance

  asmacdo

  committed
  20cc085

  Copy full SHA for 20cc085

• explicitly return

  asmacdo

  committed
  76477df

  Copy full SHA for 76477df

• output session variables to file

  Show description for b38ded1
  asmacdo

  committed
  b38ded1

  Copy full SHA for b38ded1

• modify cleanup script to retrieve instance from temporary file

  asmacdo

  committed
  f795570

  Copy full SHA for f795570

• All ec2 persmissions granted

  Show description for f8a92b2
  asmacdo

  committed
  f8a92b2

  Copy full SHA for f8a92b2

• Add EFS mount (hardcoded)

  asmacdo

  committed
  e9726df

  Copy full SHA for e9726df

• No pager for termination

  asmacdo

  committed
  c6e92f9

  Copy full SHA for c6e92f9

• force pseudo-terminal, otherwise hangs after yum install

  asmacdo

  committed
  17d77cd

  Copy full SHA for 17d77cd

• Add doublequotes to variable usage for proper expansion

  asmacdo

  committed
  2246af5

  Copy full SHA for 2246af5

• Fixup -t goes on ssh, not scp

  asmacdo

  committed
  b49b7b5

  Copy full SHA for b49b7b5

• Mount as a single command, since we dont have access to pty

  asmacdo

  committed
  584ac4d

  Copy full SHA for 584ac4d

• add todos for manual steps

  asmacdo

  committed
  4a700e5

  Copy full SHA for 4a700e5

• Disable job for now

  Show description for 6339924
  asmacdo

  committed
  6339924

  Copy full SHA for 6339924

• Update AMI to ubuntu

  Show description for 17130ef
  asmacdo

  committed
  17130ef

  Copy full SHA for 17130ef

• Roll back to AL 2023

  Show description for cc29df5
  asmacdo

  committed
  cc29df5

  Copy full SHA for cc29df5

• drop gzip, just write json

  Show description for 295361c
  asmacdo

  committed
  295361c

  Copy full SHA for 295361c

• include target dir in relative paths

  asmacdo

  committed
  a667c04

  Copy full SHA for a667c04

• Second script will not produce user report, but directory stats json

  asmacdo

  committed
  a91beb0

  Copy full SHA for a91beb0

• inital algorithm hackout

  asmacdo

  committed
  9371982

  Copy full SHA for 9371982

• Clean up and refactor for simplicity

  asmacdo

  committed
  8cead5a

  Copy full SHA for 8cead5a

• Add basic tests

  asmacdo

  committed
  86a7c72

  Copy full SHA for 86a7c72

• test multiple directories in root

  asmacdo

  committed
  fc1cab1

  Copy full SHA for fc1cab1

• comment about [:-1]

  asmacdo

  committed
  2308aed

  Copy full SHA for 2308aed

• support abspaths

  asmacdo

  committed
  84754fe

  Copy full SHA for 84754fe

• [DATALAD RUNCMD] blacken

  Show description for a1427ac
  asmacdo

  committed
  a1427ac

  Copy full SHA for a1427ac

• test propagation with files in all dirs

  asmacdo

  committed
  16e4890

  Copy full SHA for 16e4890

• Write files to disk as they are inspected

  Show description for 528833d
  asmacdo

  committed
  528833d

  Copy full SHA for 528833d

• Comment out column headers in output

  asmacdo

  committed
  3c0e7f7

  Copy full SHA for 3c0e7f7

• Write all fields for every file

  Show description for 260c69d
  asmacdo

  committed
  260c69d

  Copy full SHA for 260c69d

• Convert to reading tsv

  asmacdo

  committed
  87dd8ca

  Copy full SHA for 87dd8ca

• Fixup: update test to match tsv-read data

  asmacdo

  committed
  e0e0a32

  Copy full SHA for e0e0a32

• update for renamed script

  asmacdo

  committed
  41aaa2a

  Copy full SHA for 41aaa2a

• install pip

  asmacdo

  committed
  25e27eb

  Copy full SHA for 25e27eb

• install parallel

  asmacdo

  committed
  204b70e

  Copy full SHA for 204b70e

• install dependencies in launch script

  asmacdo

  committed
  64d653e

  Copy full SHA for 64d653e

• Output to tmp, accept only 1 arg, target dir

  asmacdo

  committed
  6475f11

  Copy full SHA for 6475f11

• add up sizes

  asmacdo

  committed
  b67c063

  Copy full SHA for b67c063

• print useful info as index is created

  asmacdo

  committed
  3241473

  Copy full SHA for 3241473

• dont fail if output dir exists

  asmacdo

  committed
  f4eb101

  Copy full SHA for f4eb101

• Create a report dict with only relevant stats

  asmacdo

  committed
  13e0e75

  Copy full SHA for 13e0e75

• output data reports

  asmacdo

  committed
  a7e6991

  Copy full SHA for a7e6991

• Remove unused

  asmacdo

  committed
  845df00

  Copy full SHA for 845df00