Keycloak Realm Installation Failing

[shounakacharya]

Hi Team,

The Keycloak Realm deployment step Deploy/Kubernetes/KeycloakRealm is failing. The previous step, Deploy/Kubernetes/Keycloak was successful.

System Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP

@gandham-santhosh and @keshavprasadms

The complete Jenkins Log is as under:

Started by user Admin Deployer
Rebuilds build #17
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm@script to read pipelines/deploy/keycloak-realm/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm@script/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/secrets.yml
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] sh

• pwd
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/KeycloakRealm
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/KeycloakRealm
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/KeycloakRealm
  [Pipeline] echo
  {currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm, env=dev, module=Kubernetes, jobName=KeycloakRealm, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/keycloak.yml, ansibleExtraArgs=--extra-vars "host=keycloak" --tags bootstrap --vault-password-file /var/lib/jenkins/secrets/vault-pass}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm, env=dev, module=Kubernetes, jobName=KeycloakRealm, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/keycloak.yml, ansibleExtraArgs=--extra-vars "host=keycloak" --tags bootstrap --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision fade1f07e0274988289795132183d8fc0a15c19b (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f fade1f07e0274988289795132183d8fc0a15c19b # timeout=10
Commit message: "update after Kong API token"
git rev-list --no-walk fade1f07e0274988289795132183d8fc0a15c19b # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/keycloak.yml --extra-vars host=keycloak --tags bootstrap --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
KeycloakRealm/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/inventory/env/hosts':
line 1, column 5, but may be elsewhere in the file depending on the exact
syntax problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
KeycloakRealm/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/wor
kspace/Deploy/dev/Kubernetes/KeycloakRealm/ansible/inventory/env/hosts:143:
Section [core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Key
cloakRealm/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [keycloak] ****************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/KeycloakRealm/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (app_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/KeycloakRealm/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (sunbird_es_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/KeycloakRealm/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (log_es_snapshot_host). Using last defined
value only.

TASK [Gathering Facts] *********************************************************
ok: [10.128.0.51]

TASK [keycloak-deploy : start keycloak service] ********************************
changed: [10.128.0.51] => {}

MSG:

starting keycloak

RUNNING HANDLER [keycloak-deploy : start service] ******************************
fatal: [10.128.0.51]: FAILED! => {
"changed": true,
"cmd": [
"/etc/init.d/keycloak",
"start"
],
"delta": "0:00:00.004799",
"end": "2022-08-22 15:41:41.285000",
"rc": 1,
"start": "2022-08-22 15:41:41.280201"
}

STDOUT:

/etc/init.d/keycloak already running.
[FAILED]

MSG:

non-zero return code

...ignoring

RUNNING HANDLER [keycloak-deploy : wait for keycloak to start] *****************
fatal: [10.128.0.51]: FAILED! => {}

MSG:

The conditional check '_result.status == 200' failed. The error was: error while evaluating conditional (_result.status == 200): 'dict object' has no attribute 'status'

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
10.128.0.51 : ok=3 changed=2 unreachable=0 failed=1 skipped=0 rescued=0 ignored=1

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• cut -d / -f1
• git name-rev --name-only HEAD
• rev
• rev
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/KeycloakRealm
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/KeycloakRealm
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Can you please post the output of below two commands from your jenkins VM

ansible --version

python --version

[shounakacharya]

@keshavprasadms . PFA the output below

jenkins@jenkins:$ ansible --version
ansible 2.8.19
config file = None
configured module search path = ['/var/lib/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Jun 29 2022, 11:45:57) [GCC 8.4.0]
jenkins@jenkins:$ python --version
Python 2.7.17

[beepdot]

Maybe the the ansible uri module has changed some values. We are checking for _result.status - https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/handlers/main.yml#L27

But I guess we might need to use status_code - https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri_module.html#parameter-status_code

Please modify the role and use _result.status_code and test if it works. Steps to do this are same - Either fork and change in your fork or modify role in jenkins workspace and run the ansible-playbook command (see jenkins job console log, ensure you put all arguments in --extra-vars in double quotes in this way --extra-vars "host=keycloak" --tags bootstrap --vault-password-file /var/lib/jenkins/secrets/vault-pass -v).

[beepdot]

Try with this branch - https://github.com/project-sunbird/sunbird-devops/tree/release-4.10.0
If it still doesn't work, I am guessing some issue in your python libs. I am posting few info from our server where this role is working well.

python --version
Python 2.7.18

python3 --version
Python 3.8.10

ansible --version
ansible 2.8.19
  config file = None
  configured module search path = ['/var/lib/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]

pip --version
pip 20.3.4 from /var/lib/jenkins/.local/lib/python2.7/site-packages/pip (python 2.7)

pip2 --version
pip 20.3.4 from /var/lib/jenkins/.local/lib/python2.7/site-packages/pip (python 2.7)

pip3 --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Package                      Version
---------------------------- --------------------
backports.ssl-match-hostname 3.7.0.1
certifi                      2021.10.8
cffi                         1.15.0
chardet                      4.0.0
decorator                    4.4.2
docker-py                    1.10.6
docker-pycreds               0.4.0
enum34                       1.1.10
idna                         2.10
ipaddress                    1.0.23
pip                          20.3.4
py                           1.11.0
pyasn1                       0.4.8
pycparser                    2.21
PyJWT                        1.7.1
pyOpenSSL                    21.0.0
python-apt                   2.0.0+ubuntu0.20.4.7
requests                     2.27.1
retry                        0.9.2
setuptools                   44.1.1
simplejson                   3.16.0
six                          1.16.0
urllib3                      1.26.9
websocket-client             0.59.0
wheel                        0.37.1

pip3 list
Package                Version             
---------------------- --------------------
ansible                2.8.19              
attrs                  19.3.0              
Automat                0.8.0               
blinker                1.4                 
certifi                2019.11.28          
cffi                   1.15.0              
chardet                3.0.4               
Click                  7.0                 
cloud-init             22.2                
colorama               0.4.3               
command-not-found      0.3                 
configobj              5.0.6               
constantly             15.1.0              
cryptography           2.8                 
dbus-python            1.2.16              
decorator              5.1.1               
distro                 1.4.0               
distro-info            0.23ubuntu1         
docker                 5.0.3               
docker-pycreds         0.4.0               
entrypoints            0.3                 
httplib2               0.14.0              
hyperlink              19.0.0              
idna                   2.8                 
importlib-metadata     1.5.0               
incremental            16.10.1             
j2cli                  0.3.10              
Jinja2                 2.10.1              
jmespath               0.9.4               
jsonpatch              1.22                
jsonpointer            2.0                 
jsonschema             3.2.0               
jwt                    1.3.1               
keyring                18.0.1              
language-selector      0.1                 
launchpadlib           1.10.13             
lazr.restfulclient     0.14.2              
lazr.uri               1.0.3               
MarkupSafe             1.1.0               
more-itertools         4.2.0               
ndg-httpsclient        0.5.1               
netifaces              0.10.4              
oauthlib               3.1.0               
pexpect                4.6.0               
pip                    20.0.2              
psycopg2               2.8.6               
py                     1.11.0              
pyasn1                 0.4.2               
pyasn1-modules         0.2.1               
pycparser              2.21                
PyGObject              3.36.0              
PyHamcrest             1.9.0               
PyJWT                  1.7.1               
pymacaroons            0.13.0              
PyNaCl                 1.3.0               
pyOpenSSL              19.0.0              
pyparted               3.11.2              
pyrsistent             0.15.5              
pyserial               3.4                 
python-apt             2.0.0+ubuntu0.20.4.7
python-debian          0.1.36ubuntu1       
PyYAML                 5.3.1               
requests               2.22.0              
requests-unixsocket    0.2.0               
retry                  0.9.2               
SecretStorage          2.3.1               
service-identity       18.1.0              
setuptools             45.2.0              
simplejson             3.16.0              
six                    1.14.0              
sos                    4.3                 
ssh-import-id          5.10                
systemd-python         234                 
Twisted                18.9.0              
ubuntu-advantage-tools 27.7                
ufw                    0.36                
unattended-upgrades    0.1                 
urllib3                1.25.8              
wadllib                1.3.3               
WALinuxAgent           2.2.46              
websocket-client       1.3.2               
wheel                  0.34.2              
zipp                   1.0.0               
zope.interface         4.7.1               

[shounakacharya]

@keshavprasadms: Is this on the keycloak server or the Jenkins Server? I am guessing it should be on the jenkins server. Also, do you have steps to manually configure the Keycloak Realm as I can access the keycloak ui in the master realm, if the automation script does not work. As I see, the only difference is I have python 2.7.17, that's it. Also since I have followed the steps of the installation only on the Jenkins from the beginning, not sure how python libraries might have gotten effected.

[beepdot]

This is on the Jenkins vm. Doing it manually is a tedious process. But if you want to do it from command line, you will need to create an admin user, start keycloak in upload feature enabled and then import the realm and clients. Please look at the role - https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/tasks/bootstrap.yml

https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/tasks/bootstrap.yml#L11-L16
https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/tasks/bootstrap.yml#L27-L32
https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/tasks/bootstrap.yml#L91-L97

The realm and clients are ansible template, so you need to replace then ansible variables before importing it
https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/templates/keycloak-realm.j2
https://github.com/project-sunbird/sunbird-devops/blob/master/ansible/roles/keycloak-deploy/templates/clients.j2

I would suggest you to do it via ansible. You can also try other approaches like running a ubuntu 16 / 20 docker container, install ansible and then run the playbook from the container etc.

[shounakacharya]

@keshavprasadms: I figured out the issue was on the keycloak vm. Whenever the ansible script runs on the jenkins, it installs a certifi version through easy_install on the keycloak vm which makes the AnsiballZ python scripts fail on the keycloak vm. I manually deleted the certifi, requests and urllib3 egg package directories from the /usr/local/lib/python2.7/dist-packages and updating the easy_install.pth so that the ansible could run on the keycloak vm. then I kept on manually commenting the bootstrap.yaml and kept moving ahead and finally was able to deploy keycloakrealm successfully. You might want to have a look at your scripts, especially when it updates the easy_install.pth file.

[beepdot]

Great. Thanks for the info. We will take a look. Can you please provide your Keycloak VM service info like OS version, python versions etc so we can reproduce the same?

[shounakacharya]

@keshavprasadms: Sure

OS - Ubuntu 18.04 LTS
python version 2.7.17
No ansible is installed here

As I mentioned earlier, it was because of easy_install which was installing certifi-2022.6.15-py2.7.egg due to which this failing. However this may be needed for the keycloak python bootstrap scripts to work. So I alternatively deleted, commented and ran the ansible playbook to finally install the keycloak realm.