Deploy>Kubernetes>Learner deploy fail /bin/sh: 1: opa: not found

[daveanvit]

Jenkins: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
Github Release Tag: release-4.6.0_RC4
Jenkins Logs:

Started by user [admin](http://34.138.204.52:8080/user/admin)
Rebuilds build #[1](http://34.138.204.52:8080/job/Deploy/job/dev/job/Kubernetes/job/Learner/1)
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner@script/7ef7a8c56617eaf0ab553d210db263f0667890b3b60cc5d0a492293898e109c7 to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner@script/7ef7a8c56617eaf0ab553d210db263f0667890b3b60cc5d0a492293898e109c7/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
 > git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --no-tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/*:refs/remotes/origin/* refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
 > git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on [Jenkins](http://34.138.204.52:8080/computer/(built-in)/) in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh
+ git clean -fxd
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
 > git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] stage
[Pipeline] { (parameter checks)
[Pipeline] echo
Found environment variable named hub_org with value as: sidsearce
[Pipeline] copyArtifacts
Copied 1 artifact from "[ArtifactUpload » dev » Core » Learner](http://34.138.204.52:8080/job/ArtifactUpload/job/dev/job/Core/job/Learner/)" build number [1](http://34.138.204.52:8080/job/ArtifactUpload/job/dev/job/Core/job/Learner/1/)
[Pipeline] sh
+ jq -r .image_name metadata.json
[Pipeline] sh
+ jq -r .node_name metadata.json
[Pipeline] echo
image_tag not specified, using the image_tag specified in metadata.json. 
[Pipeline] sh
+ jq -r .image_tag metadata.json
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] sh
+ pwd
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=Learner, absolute_job_path=ArtifactUpload/dev/Core/Learner, agent=built-in, image_name=learner_service, image_tag=release-4.6.0_RC6_bd4ca63b_3, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/kubernetes/helm_charts/core/learner release_name=learner image_tag=release-4.6.0_RC6_bd4ca63b_3 image_name=learner_service role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
Cloning the remote Git repository
Using shallow clone with depth 1
Avoid fetching tags
Cloning repository https://github.com/daveanvit/searce-diksha-sunbird.git
 > git init /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/private # timeout=10
Fetching upstream changes from https://github.com/daveanvit/searce-diksha-sunbird.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress --depth=1 -- https://github.com/daveanvit/searce-diksha-sunbird.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/daveanvit/searce-diksha-sunbird.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
 > git rev-parse origin/main^{commit} # timeout=10
Checking out Revision d3866aa62da9b6d757885e293293295526a8d1d3 (origin/main)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d3866aa62da9b6d757885e293293295526a8d1d3 # timeout=10
Commit message: "Update common.yml"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh
+ rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/ansible/inventory/env/
+ [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/ansible/inventory/env/kubernetes.yaml ]
+ ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/kubernetes/helm_charts/core/learner release_name=learner image_tag=release-4.6.0_RC6_bd4ca63b_3 image_name=learner_service role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

[WARNING]:  * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Learner/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML.   did not find expected <document start>  The error appears to be
in '/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Learner/ansible/inventory/
env/hosts': line 3, column 1, but may be elsewhere in the file depending on the
exact syntax problem.  The offending line appears to be:  # Replace the value
deployer to your server login user name, for example ubuntu
ansible_ssh_user=deployer ^ here

[WARNING]:  * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Learner/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/workspace
/Deploy/dev/Kubernetes/Learner/ansible/inventory/env/hosts:10: Section
[keycloak:children] includes undefined group: keycloak-1

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Lea
rner/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_time_zone). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(nginx_per_ip_connection_limit). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_portal_player_cdn_enabled). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(dial_liveness_readiness). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 928, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 938, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 951, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 961, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.


PLAY [local] *******************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (app_es_snapshot_host). Using last defined value
only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (sunbird_es_host). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Learner/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (log_es_snapshot_host). Using last defined value
only.


TASK [check for kong db migration] *********************************************
skipping: [localhost]

TASK [sunbird-deploy : template values.yaml file] ******************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use 
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with 
prior Ansible releases. A future Ansible release will default to using the 
discovered platform python for this host. See [https://docs.ansible.com/ansible/](https://docs.ansible.com/ansible/%1B[0m)
2.8/reference_appendices/interpreter_discovery.html for more information. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [sunbird-deploy : copy and template env file from swarm role] *************
changed: [localhost] => (item=../../../../ansible/roles/stack-sunbird/templates/sunbird_learner-service.env)
changed: [localhost] => (item=../../../../ansible/roles/stack-sunbird/templates/learner-service_logback.xml)

TASK [sunbird-deploy : create env configmap] ***********************************
changed: [localhost]

TASK [sunbird-deploy : create aplication configmap] ****************************
changed: [localhost]

TASK [sunbird-deploy : create xml configmap] ***********************************
changed: [localhost]

TASK [Load role to decrypt private keys, copy to helm chart and encrypt private keys] ***

TASK [mount-keys : Decrypt the data first] *************************************
skipping: [localhost] => (item=0) 

TASK [mount-keys : Generate Public Keys for private keys] **********************
skipping: [localhost] => (item=0) 

TASK [mount-keys : Create the keys directory in case user is overriding the private_key_path] ***
changed: [localhost]

TASK [mount-keys : Copy keys directory to chart path] **************************
skipping: [localhost] => (item=0) 

TASK [sunbird-deploy : Create the token pubic key file] ************************
changed: [localhost]

TASK [sunbird-deploy : Replace ansible variables with values] ******************
changed: [localhost] => (item=main.rego)
changed: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Copy the common opa policies to release folder] *********
changed: [localhost] => (item=main.rego)
changed: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Create OPA bundle] **************************************
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "opa build --bundle . --optimize 0 --entrypoint \"main/allow\"",
    "delta": "0:00:00.004318",
    "end": "2022-09-06 10:35:17.759002",
    "rc": 127,
    "start": "2022-09-06 10:35:17.754684"
}

STDERR:

/bin/sh: 1: opa: not found


MSG:

non-zero return code


PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=9    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0   

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh
+ git rev-parse --short HEAD
[Pipeline] sh
+ git name-rev --name-only HEAD
+ rev
+ cut -d / -f1
+ rev
[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] echo
Could not find env specific Slack channel. Check for global slack channel..
[Pipeline] echo
Could not find slack environment variable. Skipping slack notification..
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh
+ echo Deploy/dev/Kubernetes/Learner
[Pipeline] echo
Could not find env specific email group. Check for global email group..
[Pipeline] echo
Could not find global email group variable. Skipping email notification..
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

Facing same issue with Analytics, CertRegistry and Lms

[beepdot]

Please install opa on jenkins vm.
https://github.com/project-sunbird/sunbird-devops/blob/master/deploy/jenkins/jenkins-server-setup.sh#L164-L167

Please also disable opa module for now as it will give some trouble during post install.. Please see this post https://github.com/orgs/Sunbird-Ed/discussions/112#discussioncomment-3509631

[daveanvit]

Worked thanks