Deploy > Kubernetes > AdminUtils

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github Release Tag: release-4.6.0_RC4

@gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils@script to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils@script # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils@libs/deploy-conf # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: shounakacharya
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » Core » AdminUtils" build number 2
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=AdminUtils, absolute_job_path=ArtifactUpload/dev/Core/AdminUtils, agent=master, image_name=adminutil, image_tag=release-4.5.0_RC3_68b1eaa_6, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/kubernetes/helm_charts/core/adminutils release_name=adminutils image_tag=release-4.5.0_RC3_68b1eaa_6 image_name=adminutil role_name=helm-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/shounakacharya/ansible.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/private # timeout=10
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision fade1f07e0274988289795132183d8fc0a15c19b (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f fade1f07e0274988289795132183d8fc0a15c19b # timeout=10
Commit message: "update after Kong API token"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/AdminUtils/kubernetes/helm_charts/core/adminutils release_name=adminutils image_tag=release-4.5.0_RC3_68b1eaa_6 image_name=adminutil role_name=helm-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
AdminUtils/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/AdminUtils/ansible/inventory/env/hosts':
line 1, column 5, but may be elsewhere in the file depending on the exact
syntax problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
AdminUtils/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/worksp
ace/Deploy/dev/Kubernetes/AdminUtils/ansible/inventory/env/hosts:143: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Adm
inUtils/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [local] *******************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/AdminUtils/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (app_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/AdminUtils/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (sunbird_es_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/AdminUtils/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (log_es_snapshot_host). Using last defined
value only.

TASK [check for kong db migration] *********************************************
skipping: [localhost]

TASK [helm-deploy : getting kube dns ip] ***************************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [helm-deploy : setting variable] ******************************************
ok: [localhost]

TASK [helm-deploy : rename template] *******************************************
changed: [localhost]

TASK [Load role to decrypt private keys, copy to helm chart and encrypt private keys] ***

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
changed: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
ok: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
ok: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
ok: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0)

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
ok: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0)

TASK [helm-deploy : Create the token pubic key file] ***************************
fatal: [localhost]: FAILED! => {
"changed": false
}

MSG:

src (or content) is required

PLAY RECAP *********************************************************************
localhost : ok=8 changed=3 unreachable=0 failed=1 skipped=11 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• cut+ -d / -f1
  rev
• rev
• git name-rev --name-only HEAD
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/AdminUtils
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Before you proceed further, I want you to go back and check if you have updated all mandatory variables in your private repo. If not, please update them. Else you will start to get many such issues.

For this issue, please check if you have defined the below variable in your private repo?
https://github.com/project-sunbird/sunbird-devops/blob/master/private_repo/ansible/inventory/dev/Core/secrets.yml#L52

[shounakacharya]

@keshavprasadms : Got it. I have already raised the installation failure happening in this thread #77 for Keycloak Realm. I went ahead so that I could complete the other steps. If you could reply on that thread so that I can go ahead with the Keycloak Realm installation, then I will proceed further here.

[beepdot]

Ok, I will check and provide a solution on that thread shortly.

[shounakacharya]

@keshavprasadms: Solving KeycloakRealm in https://github.com/orgs/Sunbird-Ed/discussions/77, also solved this. Marking this as answered. Thanks for your help