Deploy>Kubernetes>Content

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github Tag: release-4.6.0_RC4

@gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content@script to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content@script # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content@libs/deploy-conf # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Using shallow clone with depth 1
Cloning repository https://github.com/project-sunbird/sunbird-devops.git
git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content # timeout=10
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/Content
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Content
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Content
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: shounakacharya
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » Core » Content" build number 2
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/dev/Kubernetes/Content
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=Content, absolute_job_path=ArtifactUpload/dev/Core/Content, agent=master, image_name=content-service, image_tag=release-4.6.0_RC2_1cf2c852_2, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/kubernetes/helm_charts/core/content release_name=content image_tag=release-4.6.0_RC2_1cf2c852_2 image_name=content-service role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/shounakacharya/ansible.git

git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/private # timeout=10
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 886d95ed8589f02fd96e748a15981e818e22f841 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 886d95ed8589f02fd96e748a15981e818e22f841 # timeout=10
Commit message: "after keycloak installation and tokens update"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Content/kubernetes/helm_charts/core/content release_name=content image_tag=release-4.6.0_RC2_1cf2c852_2 image_name=content-service role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Content/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/Content/ansible/inventory/env/hosts': line
1, column 5, but may be elsewhere in the file depending on the exact syntax
problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Content/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/workspace
/Deploy/dev/Kubernetes/Content/ansible/inventory/env/hosts:143: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Con
tent/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_time_zone). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(nginx_per_ip_connection_limit). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_portal_player_cdn_enabled). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(dial_liveness_readiness). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 928, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 938, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 951, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 961, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

PLAY [local] *******************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (app_es_snapshot_host). Using last defined value
only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (sunbird_es_host). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Content/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (log_es_snapshot_host). Using last defined value
only.

TASK [check for kong db migration] *********************************************
skipping: [localhost]

TASK [sunbird-deploy : template values.yaml file] ******************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [sunbird-deploy : copy and template env file from swarm role] *************
failed: [localhost] (item=../../../../ansible/roles/stack-sunbird/templates/content-service_application.conf) => {
"ansible_loop_var": "item",
"changed": false,
"item": "../../../../ansible/roles/stack-sunbird/templates/content-service_application.conf"
}

MSG:

AnsibleUndefinedVariable: 'learning_content_drive_apiKey' is undefined

changed: [localhost] => (item=../../../../ansible/roles/stack-sunbird/templates/content-service_logback.xml)

PLAY RECAP *********************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• cut -d / -f1
• rev
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Content
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/Content
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Please create a variable in your Core/secrets.yml for this. Its missed in the repo.

learning_content_drive_apiKey: GOOGLE_DRIVE_API_KEY

You need to create a google drive API key and use that in the above variable - https://developers.google.com/drive/api/guides/enable-drive-api

[shounakacharya]

@keshavprasadms: Thanks so much! This solved the problem! Marking as answered