Deploy -> Kubernetes -> Cassandra - Migration of keyspace sunbird to version 1.86 failed

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github release Tag: release-4.6.0_RC4

@gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Rebuilds build #1
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@script to read pipelines/deploy/cassandra-deploy/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@script/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/secrets.yml
  Removing cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar
  Removing cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_1
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » Core » Cassandra" build number 2
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_1
  Archive: cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_1
  inflating: cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar
  [Pipeline] echo
  {env=dev, module=Kubernetes, jobName=Cassandra, absolute_job_path=ArtifactUpload/dev/Core/Cassandra, agent=master, artifact_name=cassandra_artifacts.zip, artifact_version=release-4.6.0_RC1_23630dee_1, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra, artifact=cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_1, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=Cassandra, absolute_job_path=ArtifactUpload/dev/Core/Cassandra, agent=master, artifact_name=cassandra_artifacts.zip, artifact_version=release-4.6.0_RC1_23630dee_1, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra, artifact=cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_1, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 9b404cbd9d273d08452839b72b2a56580f5f0b23 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 9b404cbd9d273d08452839b72b2a56580f5f0b23 # timeout=10
Commit message: "update load balancer IP for private k8s ingress"
git rev-list --no-walk 9b404cbd9d273d08452839b72b2a56580f5f0b23 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml --vault-password-file /var/lib/jenkins/secrets/vault-pass -v
  No config file found; using defaults
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Cassandra/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts':
line 1, column 5, but may be elsewhere in the file depending on the exact
syntax problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Cassandra/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/workspa
ce/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts:143: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cas
sandra/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [cassandra] ***************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (app_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (sunbird_es_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (log_es_snapshot_host). Using last defined
value only.

TASK [Gathering Facts] *********************************************************
ok: [10.128.0.52]

TASK [cassandra-deploy : checking for last failed migration] *******************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "cqlsh -e 'SELECT version from sunbird.cassandra_migration_version where success=false ALLOW FILTERING ;' | tail -n +4 | head -n -2 | sort | tail -n1 | xargs",
"delta": "0:00:00.482394",
"end": "2022-08-18 08:24:32.166744",
"rc": 0,
"start": "2022-08-18 08:24:31.684350"
}

STDOUT:

1.86

TASK [cassandra-deploy : Dropping failed migration 1.86] ***********************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "cqlsh -e "delete from sunbird.cassandra_migration_version where version='1.86';"",
"delta": "0:00:00.430935",
"end": "2022-08-18 08:24:32.816705",
"rc": 0,
"start": "2022-08-18 08:24:32.385770"
}

TASK [cassandra-deploy : Copy the migration jar] *******************************
ok: [10.128.0.52] => {
"changed": false,
"checksum": "a0e2c7f3d07be5c88dd73f99cfc1178e87cc3348",
"dest": "/home/deployer/cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar",
"gid": 0,
"group": "root",
"mode": "0644",
"owner": "root",
"path": "/home/deployer/cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar",
"size": 5005707,
"state": "file",
"uid": 0
}

TASK [cassandra-deploy : Run the application] **********************************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "java -cp "cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar" com.contrastsecurity.cassandra.migration.utils.MigrationScriptEntryPoint",
"delta": "0:00:03.282521",
"end": "2022-08-18 08:24:37.073696",
"rc": 0,
"start": "2022-08-18 08:24:33.791175"
}

STDOUT:

Migration started at ==1660811073856
Cassandra Migration 0.0.1-SNAPSHOT by Contrast Security
Connected to cluster: Test Cluster
Data center: datacenter1; Host: /10.128.0.52
Current version of keyspace sunbird: 1.85
Migrating keyspace sunbird to version 1.86 - cassandra
ERROR: Migration of keyspace sunbird to version 1.86 failed! Please restore backups and roll back database and code!
Migration Completed at ==1660811077059

STDERR:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
com.contrastsecurity.cassandra.migration.CassandraMigrationException: Unable to apply migration
at com.contrastsecurity.cassandra.migration.action.Migrate.applyMigration(Migrate.java:147)
at com.contrastsecurity.cassandra.migration.action.Migrate.run(Migrate.java:112)
at com.contrastsecurity.cassandra.migration.CassandraMigration$1.execute(CassandraMigration.java:88)
at com.contrastsecurity.cassandra.migration.CassandraMigration$1.execute(CassandraMigration.java:72)
at com.contrastsecurity.cassandra.migration.CassandraMigration.execute(CassandraMigration.java:201)
at com.contrastsecurity.cassandra.migration.CassandraMigration.migrate(CassandraMigration.java:71)
at com.contrastsecurity.cassandra.migration.utils.MigrationScriptEntryPoint.main(MigrationScriptEntryPoint.java:40)
Caused by: com.datastax.driver.core.exceptions.InvalidConfigurationInQueryException: Trigger class 'org.sunbird.cassandra.triggers.TransactionEventTrigger' doesn't exist
at com.datastax.driver.core.exceptions.InvalidConfigurationInQueryException.copy(InvalidConfigurationInQueryException.java:37)
at com.datastax.driver.core.exceptions.InvalidConfigurationInQueryException.copy(InvalidConfigurationInQueryException.java:27)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at com.contrastsecurity.cassandra.migration.script.CqlScript.execute(CqlScript.java:87)
at com.contrastsecurity.cassandra.migration.resolver.cql.CqlMigrationExecutor.execute(CqlMigrationExecutor.java:35)
at com.contrastsecurity.cassandra.migration.action.Migrate.applyMigration(Migrate.java:145)
... 6 more

TASK [cassandra-deploy : Check the migraion status] ****************************
fatal: [10.128.0.52]: FAILED! => {
"changed": false
}

MSG:

Failed migration: ['Migration started at ==1660811073856', 'Cassandra Migration 0.0.1-SNAPSHOT by Contrast Security', 'Connected to cluster: Test Cluster', 'Data center: datacenter1; Host: /10.128.0.52', 'Current version of keyspace sunbird: 1.85', 'Migrating keyspace sunbird to version 1.86 - cassandra', 'ERROR: Migration of keyspace sunbird to version 1.86 failed! Please restore backups and roll back database and code!', 'Migration Completed at ==1660811077059']

PLAY RECAP *********************************************************************
10.128.0.52 : ok=5 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• • rev
    git name-rev --name-only HEAD
• rev
• cut -d / -f1
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

@santhosh-tg Can you please suggest a solution for this?

[shounakacharya]

@santhosh-tg : Where should I remove these files. Please note that I do not have a fork of the repo. Kindly suggest in terms of Jenkins Workspace directories.

[santhosh-tg]

@shounakacharya To clarify - I was referring to the job Build/Core/Cassandra and repo https://github.com/Sunbird-Lern/sunbird-utils

The workspace path should be similar to the one i shared

sunbird-utils/sunbird-cassandra-migration/cassandra-migration/src/main/resources/db/migration/cassandra

[shounakacharya]

@santhosh-tg : Now I am getting error at the version 1.90 migration step. Please find the logs below. Please note that I did not remove the V1.86_cassandra.cql but commented the CQL lines as follows:

/*CREATE TRIGGER IF NOT EXISTS course_batch_trigger ON sunbird_courses.course_batch USING 'org.sunbird.cassandra.triggers.TransactionEventTrigger';
CREATE TRIGGER IF NOT EXISTS batch_enrollment_trigger ON sunbird_courses.user_courses USING 'org.sunbird.cassandra.triggers.TransactionEventTrigger';
*/

Please find the logs below:

Started by user Admin Deployer
Rebuilds build #3
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@script to read pipelines/deploy/cassandra-deploy/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@script/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Running in Durability level: MAX_SURVIVABILITY
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/secrets.yml
  Removing cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar
  Removing cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_4
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (get artifact)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] copyArtifacts
  Copied 2 artifacts from "ArtifactUpload » dev » Core » Cassandra" build number 4
  [Pipeline] sh
• jq -r .artifact_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  artifact_version not specified, using the artifact_version specified in metadata.json.
  [Pipeline] sh
• jq -r .artifact_version metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] echo
Option chosen is JenkinsJob, using the artifacts copied
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy artifact)
[Pipeline] sh

• unzip cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_5
  Archive: cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_5
  inflating: cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar
  [Pipeline] echo
  {env=dev, module=Kubernetes, jobName=Cassandra, absolute_job_path=ArtifactUpload/dev/Core/Cassandra, agent=master, artifact_name=cassandra_artifacts.zip, artifact_version=release-4.6.0_RC1_23630dee_5, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra, artifact=cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_5, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=Cassandra, absolute_job_path=ArtifactUpload/dev/Core/Cassandra, agent=master, artifact_name=cassandra_artifacts.zip, artifact_version=release-4.6.0_RC1_23630dee_5, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra, artifact=cassandra_artifacts.zip:release-4.6.0_RC1_23630dee_5, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/shounakacharya/ansible.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/shounakacharya/ansible.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/shounakacharya/ansible.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/main^{commit} # timeout=10
Checking out Revision 886d95ed8589f02fd96e748a15981e818e22f841 (origin/main)
git config core.sparsecheckout # timeout=10
git checkout -f 886d95ed8589f02fd96e748a15981e818e22f841 # timeout=10
Commit message: "after keycloak installation and tokens update"
git rev-list --no-walk 886d95ed8589f02fd96e748a15981e818e22f841 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/cassandra-deploy.yml --vault-password-file /var/lib/jenkins/secrets/vault-pass -v
  No config file found; using defaults
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Cassandra/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts':
line 1, column 5, but may be elsewhere in the file depending on the exact
syntax problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Cassandra/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/workspa
ce/Deploy/dev/Kubernetes/Cassandra/ansible/inventory/env/hosts:143: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Cas
sandra/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [cassandra] ***************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (app_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (sunbird_es_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Cassandra/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (log_es_snapshot_host). Using last defined
value only.

TASK [Gathering Facts] *********************************************************
ok: [10.128.0.52]

TASK [cassandra-deploy : checking for last failed migration] *******************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "cqlsh -e 'SELECT version from sunbird.cassandra_migration_version where success=false ALLOW FILTERING ;' | tail -n +4 | head -n -2 | sort | tail -n1 | xargs",
"delta": "0:00:00.500328",
"end": "2022-08-24 09:55:22.651738",
"rc": 0,
"start": "2022-08-24 09:55:22.151410"
}

STDOUT:

1.86

TASK [cassandra-deploy : Dropping failed migration 1.86] ***********************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "cqlsh -e "delete from sunbird.cassandra_migration_version where version='1.86';"",
"delta": "0:00:00.476052",
"end": "2022-08-24 09:55:23.335608",
"rc": 0,
"start": "2022-08-24 09:55:22.859556"
}

TASK [cassandra-deploy : Copy the migration jar] *******************************
changed: [10.128.0.52] => {
"changed": true,
"checksum": "db30af30e28786626a952f0dcc514f864bf372e5",
"dest": "/home/deployer/cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar",
"gid": 0,
"group": "root",
"md5sum": "fba3b7314eef4971197cc2417d3363f8",
"mode": "0644",
"owner": "root",
"size": 5005713,
"src": "/home/deployer/.ansible/tmp/ansible-tmp-1661334923.4172962-13422-180250832808385/source",
"state": "file",
"uid": 0
}

TASK [cassandra-deploy : Run the application] **********************************
changed: [10.128.0.52] => {
"changed": true,
"cmd": "java -cp "cassandra-migration-0.0.1-SNAPSHOT-jar-with-dependencies.jar" com.contrastsecurity.cassandra.migration.utils.MigrationScriptEntryPoint",
"delta": "0:00:07.781100",
"end": "2022-08-24 09:55:32.203044",
"rc": 0,
"start": "2022-08-24 09:55:24.421944"
}

STDOUT:

Migration started at ==1661334924501
Cassandra Migration 0.0.1-SNAPSHOT by Contrast Security
Connected to cluster: Test Cluster
Data center: datacenter1; Host: /10.128.0.52
Current version of keyspace sunbird: 1.85
Migrating keyspace sunbird to version 1.86 - cassandra
Migrating keyspace sunbird to version 1.87 - cassandra
Migrating keyspace sunbird to version 1.88 - cassandra
Migrating keyspace sunbird to version 1.89 - cassandra
Migrating keyspace sunbird to version 1.90 - cassandra
ERROR: Migration of keyspace sunbird to version 1.90 failed! Please restore backups and roll back database and code!
Migration Completed at ==1661334931215

STDERR:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
com.contrastsecurity.cassandra.migration.CassandraMigrationException: Unable to apply migration
at com.contrastsecurity.cassandra.migration.action.Migrate.applyMigration(Migrate.java:147)
at com.contrastsecurity.cassandra.migration.action.Migrate.run(Migrate.java:112)
at com.contrastsecurity.cassandra.migration.CassandraMigration$1.execute(CassandraMigration.java:88)
at com.contrastsecurity.cassandra.migration.CassandraMigration$1.execute(CassandraMigration.java:72)
at com.contrastsecurity.cassandra.migration.CassandraMigration.execute(CassandraMigration.java:201)
at com.contrastsecurity.cassandra.migration.CassandraMigration.migrate(CassandraMigration.java:71)
at com.contrastsecurity.cassandra.migration.utils.MigrationScriptEntryPoint.main(MigrationScriptEntryPoint.java:40)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot create secondary index on partition key column userid
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
at com.contrastsecurity.cassandra.migration.script.CqlScript.execute(CqlScript.java:87)
at com.contrastsecurity.cassandra.migration.resolver.cql.CqlMigrationExecutor.execute(CqlMigrationExecutor.java:35)
at com.contrastsecurity.cassandra.migration.action.Migrate.applyMigration(Migrate.java:145)
... 6 more
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot create secondary index on partition key column userid
at com.datastax.driver.core.Responses$Error.asException(Responses.java:136)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:184)
at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:798)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:617)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)

TASK [cassandra-deploy : Check the migraion status] ****************************
fatal: [10.128.0.52]: FAILED! => {
"changed": false
}

MSG:

Failed migration: ['Migration started at ==1661334924501', 'Cassandra Migration 0.0.1-SNAPSHOT by Contrast Security', 'Connected to cluster: Test Cluster', 'Data center: datacenter1; Host: /10.128.0.52', 'Current version of keyspace sunbird: 1.85', 'Migrating keyspace sunbird to version 1.86 - cassandra', 'Migrating keyspace sunbird to version 1.87 - cassandra', 'Migrating keyspace sunbird to version 1.88 - cassandra', 'Migrating keyspace sunbird to version 1.89 - cassandra', 'Migrating keyspace sunbird to version 1.90 - cassandra', 'ERROR: Migration of keyspace sunbird to version 1.90 failed! Please restore backups and roll back database and code!', 'Migration Completed at ==1661334931215']

PLAY RECAP *********************************************************************
10.128.0.52 : ok=5 changed=4 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• git name-rev --name-only HEAD
• rev
• rev
• cut -d / -f1
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/Cassandra
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[santhosh-tg]

Hi @amit-tarento appreciate your Inputs here.

[amitec006]

@shounakacharya Please run this below query on cassandra
update cassandra_migration_version set success= true where version = '1.90';
and then re-run the migration job. This will resolve the issue.
CC @santhosh-tg

[shounakacharya]

@amit-tarento : Could you please tell me in which keyspace should I run this query?

Thanks
Shounak

[amitec006]

its sunbird keyspace.
update sunbird.cassandra_migration_version set success= true where version = '1.90';

[shounakacharya]

@amit-tarento : This worked. Thanks for the help. Marking this as answered!