GDEX TDS CIRRUS Deployment

Deployment workflow

• Close the PR on merge to main branch in rds-tds/ would trigger the GitHub Action
• GitHub action trigger
  • GitHub Action docker image build and push to Docker registry (TDS, Log)
  • TDS, Log, TDM (pre-build) deployed through Helm Chart
  • Hosted on Kubernetes(k8s)-Cirrus
• Helm chart - rda-tds-helm will help the CIRRUS to get the container in the registry and deploy them

Github repository

Here's the repository tree for the NCAR/gdex-tds GitHub repository:

gdex-tds/
├── .github/
├── LICENSE
├── rda-tds-helm/ (CIRRUS deployment)
│       ├── Chart.yaml
│       ├── values.yaml
│       └── templates/
│           ├── deployment.yaml
│           ├── ....
├── src (useful python exec to create xml and ctl)
│            
└── rda-tds  (actual TDS code)
    ├── content
    ├── Dockerfiles
    └── ...

• rda-tds-helm : the k8s/CIRRUS deployment related (change in here will not trigger deployement)
  • Chart : Helm Chart setting (deployment chart viewable in ARGO CD)
  • values : define the variable values that is going to be used in the templates
  • templates : where the actual individual deployment setting in .yaml
    • deployment.yaml
      • contain all containers deploynment, persistent volume mount ...etc.
      • needed resources, ports, persistent volume so the container can access
      • containers include
        • TDS deploy include 3 mount (data, log, indexfile/xml)
        • Log deploy include 1 mount (log)
          • cat all log and can be viewed on Grafana
          • if not done one can only view if login to the exec pod
        • TDM (thredds data management) deploy 2 mount (data, indexfile/xml)
• rda-tds : where all the TDS detail is located including index files and xml
  • content : all xml
    • catalog.xml : all data catelog
    • catalog_d010079.xml : individual data catelog (can be generated by src/createXML.py)
    • threddsConfig.xml : detail configuration for the TDS appearance
    • template/tdsTemplateFragments.html : the html for the TDS appearance
• src : python scripts generating the control file for dsrqst or xml file for TDS
  • createCTL.py : create the rda control file to be added so the webpage can see the TDS column
    • dsrqst -sc -ds d010079 -if d010079.ctl -nc -md
  • createXML.py : create the xml catalog

K8s health check

• ARGO CD - check the chart, log, and pod run health
• Grafana - check the usage stats

Checking the container on K8s

• Install kubectl
• Install Azure kubelogin
• Ask for config file from the CIRRUS team and put under .kube/
• Check out container

  kubectl exec -it deploy/rda-tds -n rda -c rda-tds -- /bin/bash`
  

Useful info

• useful CIRRUS intro
  • https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/cirrus/guides/01-intro/

Basic steps for adding data

Adding data to the TDS

1. create dataset xml
   1. createXML.py dxxxxxx ./
2. add dataset xml file to the catalog.xml
3. GitHub action deploy the TDS container
4. This create the TDS thredds server with the new dataset included but cannot be seen in the GDEX data portal column view. Following steps add the associated TDS access to the GDEX data portal

Adding the GDEX TDS access

5. checking deployment on ARGO CD and Grafana
6. check log directly in the deployed container by using kubectl check
7. dsrqst add dataset to the database
   1. try if one can see the dataset control file through dsrqst

      • dsrqst -gc -ds d010077 if there is output CTL export from the database it should have some thing under the following line

        ControlIndex<:>Dataset<:>GroupIndex<:>RequestType<:>ControlMode<:>TarFlag<:>Specialist<:>ProcessCommand<:>EmptyOutput<:>URL<:>HostName<:>
        

   2. let the standard output to store in a temporary file to make sure one has a original copy

      dsrqst -gc -ds d010077 > d010077.orig.dsrqst`
      

   3. Create the new ctl file content using creataCTL.py

      creataCTL.py d010077
      

   4. add the standard output from above to d010077.orig.dsrqst
   5. update the dataset control information using
      • dsrqst -sc -ds d010077 -if d010077.orig.dsrqst -md (if original CTL item already exist)
      • dsrqst -sc -ds d010077 -if d010077.orig.dsrqst -md -nc (if original CTL item DOES NOT exist)