Build -> DataPipeline->EdDataProducts (Build Failed)

[shounakacharya]

System Configuration Details:

1. KP, DP, DB, Yarn, Jenkins, Other-Services: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
2. Google Kubernetes engine: 1.21.12-gke.2200 with 4 worker nodes. Note: Currently Sunbird is not compatible with K8s APIs >= 1.22 and we have checked this
3. All systems are running on GCP
4. Github Tag: release-4.6.0_RC2

@gandham-santhosh and @keshavprasadms

Jenkins Logs:

Started by user Admin Deployer
Rebuilds build #2
Checking out git https://github.com/Sunbird-Ed/sunbird-data-products.git into /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts@script to read data-products/build.Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts@script/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/Sunbird-Ed/sunbird-data-products.git # timeout=10
Fetching upstream changes from https://github.com/Sunbird-Ed/sunbird-data-products.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress -- https://github.com/Sunbird-Ed/sunbird-data-products.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC2^{commit} # timeout=10
git rev-parse release-4.6.0_RC2^{commit} # timeout=10
Checking out Revision 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 (release-4.6.0_RC2)
git config core.sparsecheckout # timeout=10
git checkout -f 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 # timeout=10
Commit message: "Merge pull request #505 from manjudr/uci-jobs-changes"
git rev-list --no-walk 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 # timeout=10
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts
[Pipeline] {
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/Sunbird-Ed/sunbird-data-products.git

git init /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts # timeout=10
Fetching upstream changes from https://github.com/Sunbird-Ed/sunbird-data-products.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress -- https://github.com/Sunbird-Ed/sunbird-data-products.git +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url https://github.com/Sunbird-Ed/sunbird-data-products.git # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
git rev-parse origin/release-4.6.0_RC2^{commit} # timeout=10
git rev-parse release-4.6.0_RC2^{commit} # timeout=10
Checking out Revision 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 (release-4.6.0_RC2)
git config core.sparsecheckout # timeout=10
git checkout -f 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 # timeout=10
Commit message: "Merge pull request #505 from manjudr/uci-jobs-changes"
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/Sunbird-Ed/sunbird-data-products.git # timeout=10
Fetching upstream changes from https://github.com/Sunbird-Ed/sunbird-data-products.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress -- https://github.com/Sunbird-Ed/sunbird-data-products.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC2^{commit} # timeout=10
git rev-parse release-4.6.0_RC2^{commit} # timeout=10
Checking out Revision 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 (release-4.6.0_RC2)
git config core.sparsecheckout # timeout=10
git checkout -f 2f5e8b1d633a9c424a0beffa43985a7964bc27a9 # timeout=10
Commit message: "Merge pull request #505 from manjudr/uci-jobs-changes"
[Pipeline] echo
github_release_tag specified, building from github_release_tag: release-4.6.0_RC2
[Pipeline] echo
artifact_version: release-4.6.0_RC2
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] stage
[Pipeline] { (Pre-Build)
[Pipeline] sh
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] sh

• cd data-products
• mvn clean install -DskipTests
  [�[1;34mINFO�[m] Scanning for projects...
  [�[1;33mWARNING�[m]
  [�[1;33mWARNING�[m] Some problems were encountered while building the effective model for org.sunbirded:data-products:jar:1.0
  [�[1;33mWARNING�[m] 'dependencies.dependency.systemPath' for ing.wbaa.druid:scruid_${scala.maj.version}:jar should not point at files within the project directory, ${project.basedir}/lib/scruid_2.11-2.4.0.jar will be unresolvable by dependent projects @ line 290, column 16
  [�[1;33mWARNING�[m]
  [�[1;33mWARNING�[m] It is highly recommended to fix these problems because they threaten the stability of your build.
  [�[1;33mWARNING�[m]
  [�[1;33mWARNING�[m] For this reason, future Maven versions might no longer support building such malformed projects.
  [�[1;33mWARNING�[m]
  [�[1;34mINFO�[m]
  [�[1;34mINFO�[m] �[1m--------------------< �[0;36morg.sunbirded:data-products�[0;1m >---------------------�[m
  [�[1;34mINFO�[m] �[1mBuilding data-products 1.0�[m
  [�[1;34mINFO�[m] �[1m--------------------------------[ jar ]---------------------------------�[m
  [�[1;33mWARNING�[m] The POM for org.sunbird:batch-models:jar:2.0 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
  [�[1;34mINFO�[m]
  [�[1;34mINFO�[m] �[1m--- �[0;32mmaven-clean-plugin:2.5:clean�[m �[1m(default-clean)�[m @ �[36mdata-products�[0;1m ---�[m
  [�[1;34mINFO�[m]
  [�[1;34mINFO�[m] �[1m--- �[0;32mmaven-resources-plugin:2.6:resources�[m �[1m(default-resources)�[m @ �[36mdata-products�[0;1m ---�[m
  [�[1;34mINFO�[m] Using 'UTF-8' encoding to copy filtered resources.
  [�[1;34mINFO�[m] Copying 4 resources
  [�[1;34mINFO�[m]
  [�[1;34mINFO�[m] �[1m--- �[0;32mmaven-compiler-plugin:3.1:compile�[m �[1m(default-compile)�[m @ �[36mdata-products�[0;1m ---�[m
  [�[1;34mINFO�[m] Nothing to compile - all classes are up to date
  [�[1;34mINFO�[m]
  [�[1;34mINFO�[m] �[1m--- �[0;32mscala-maven-plugin:3.2.2:compile�[m �[1m(default)�[m @ �[36mdata-products�[0;1m ---�[m
  [�[1;33mWARNING�[m] Invalid POM for org.sunbird:batch-models:jar:2.0, transitive dependencies (if any) will not be available, enable debug logging for more details
  [�[1;33mWARNING�[m] Expected all dependencies to require Scala version: 2.11.8
  [�[1;33mWARNING�[m] org.sunbirded:data-products:1.0 requires scala version: 2.11.8
  [�[1;33mWARNING�[m] com.redislabs:spark-redis_2.11:2.5.0 requires scala version: 2.11.12
  [�[1;33mWARNING�[m] Multiple versions of scala libraries detected!
  [�[1;34mINFO�[m] /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts/data-products/src/main/scala:-1: info: compiling
  [�[1;34mINFO�[m] Compiling 44 source files to /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts/data-products/target/classes at 1660893573899
  [�[1;31mERROR�[m] /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts/data-products/src/main/scala/org/sunbird/analytics/exhaust/BaseReportsJob.scala:41: error: too many arguments for method getSparkSession: (parallelization: Int, appName: String, sparkCassandraConnectionHost: Option[AnyRef], sparkElasticsearchConnectionHost: Option[AnyRef], readConsistencyLevel: Option[String], sparkRedisConnectionHost: Option[AnyRef], sparkRedisDB: Option[AnyRef], sparkRedisPort: Option[AnyRef])org.apache.spark.sql.SparkSession
  [�[1;31mERROR�[m] val sparkSession = CommonUtil.getSparkSession(JobContext.parallelization, config.appName.getOrElse(config.model), sparkCassandraConnectionHost, sparkElasticsearchConnectionHost, Option(readConsistencyLevel),sparkRedisConnectionHost, sparkUserDbRedisIndex, sparkUserDbRedisPort, writeConsistencyLevel)
  [�[1;31mERROR�[m] ^
  [�[1;31mERROR�[m] /var/lib/jenkins/workspace/Build/DataPipeline/EdDataProducts/data-products/src/main/scala/org/sunbird/analytics/exhaust/util/ExhaustUtil.scala:19: error: value getBlobUrl is not a member of object org.ekstep.analytics.framework.util.CommonUtil
  [�[1;31mERROR�[m] val url = CommonUtil.getBlobUrl(store, file, bucket)
  [�[1;31mERROR�[m] ^
  [�[1;31mERROR�[m] two errors found
  [�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
  [�[1;34mINFO�[m] �[1;31mBUILD FAILURE�[m
  [�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
  [�[1;34mINFO�[m] Total time: 21.034 s
  [�[1;34mINFO�[m] Finished at: 2022-08-19T12:49:47+05:30
  [�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
  [�[1;31mERROR�[m] Failed to execute goal �[32mnet.alchim31.maven:scala-maven-plugin:3.2.2:compile�[m �[1m(default)�[m on project �[36mdata-products�[m: �[1;31mwrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)�[m -> �[1m[Help 1]�[m
  [�[1;31mERROR�[m]
  [�[1;31mERROR�[m] To see the full stack trace of the errors, re-run Maven with the �[1m-e�[m switch.
  [�[1;31mERROR�[m] Re-run Maven using the �[1m-X�[m switch to enable full debug logging.
  [�[1;31mERROR�[m]
  [�[1;31mERROR�[m] For more information about the errors and possible solutions, please read the following articles:
  [�[1;31mERROR�[m] �[1m[Help 1]�[m http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  ERROR: script returned exit code 1
  Finished: FAILURE

[beepdot]

@sowmya-dixit @anandp504 Can you please suggest a solution for this error?

[shounakacharya]

@keshavprasadms and @sowmya-dixit this solution worked.
@keshavprasadms For the ArtifactUpload/dev/DataPipeline/EdDataProducts also I used the release tag: [release-4.4.0_RC2]. Kindly confirm whether this is correct.

[beepdot]

Yes thats fine. There is no change in the ArtifactUpload jobs code from many releases.

[shounakacharya]

@keshavprasadms : Thanks Marking this as answered!

[jaingaurav126]

Getting this error while running AnalyticsCore, CoreDataProducts and EdDataProducts even after configuring everything correctly.

error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[�[1;31mERROR�[m] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:17)
[�[1;31mERROR�[m] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:18)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:53)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:102)
[�[1;31mERROR�[m] at scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:105)
[�[1;31mERROR�[m] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[�[1;31mERROR�[m] at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[�[1;31mERROR�[m] at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[�[1;31mERROR�[m] at scala.tools.nsc.Global$Run.(Global.scala:1242)
[�[1;31mERROR�[m] at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[�[1;31mERROR�[m] at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[�[1;31mERROR�[m] at scala.tools.nsc.Driver.process(Driver.scala:51)
[�[1;31mERROR�[m] at scala.tools.nsc.Driver.main(Driver.scala:64)
[�[1;31mERROR�[m] at scala.tools.nsc.Main.main(Main.scala)
[�[1;31mERROR�[m] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[�[1;31mERROR�[m] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[�[1;31mERROR�[m] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[�[1;31mERROR�[m] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[�[1;31mERROR�[m] at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[�[1;31mERROR�[m] at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
[�[1;34mINFO�[m] �[1mReactor Summary for Sunbird Analytics Framework 2.0:�[m
[�[1;34mINFO�[m]
[�[1;34mINFO�[m] Sunbird Analytics Framework ........................ �[1;32mSUCCESS�[m [ 0.309 s]
[�[1;34mINFO�[m] analytics-core ..................................... �[1;31mFAILURE�[m [ 8.392 s]
[�[1;34mINFO�[m] Analytics Job Driver - Batch and Streaming ......... �[1;33mSKIPPED�[m
[�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
[�[1;34mINFO�[m] �[1;31mBUILD FAILURE�[m
[�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
[�[1;34mINFO�[m] Total time: 8.840 s
[�[1;34mINFO�[m] Finished at: 2022-09-09T16:18:06+05:30
[�[1;34mINFO�[m] �[1m------------------------------------------------------------------------�[m
[�[1;31mERROR�[m] Failed to execute goal �[32mnet.alchim31.maven:scala-maven-plugin:3.2.2:compile�[m �[1m(default)�[m on project �[36manalytics-core�[m: �[1;31mwrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)�[m -> �[1m[Help 1]�[m
[�[1;31mERROR�[m]
[�[1;31mERROR�[m] To see the full stack trace of the errors, re-run Maven with the �[1m-e�[m switch.
[�[1;31mERROR�[m] Re-run Maven using the �[1m-X�[m switch to enable full debug logging.
[�[1;31mERROR�[m]
[�[1;31mERROR�[m] For more information about the errors and possible solutions, please read the following articles:
[�[1;31mERROR�[m] �[1m[Help 1]�[m http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[�[1;31mERROR�[m]
[�[1;31mERROR�[m] After correcting the problems, you can resume the build with the command
[�[1;31mERROR�[m] �[1mmvn -rf :analytics-core�[m
[Pipeline] }

Kindly help

[beepdot]

@jaingaurav126 please can you create a new thread to discuss?

[jaingaurav126]

Hi, Done with creating a new thread (#146) for the error. Pleas check

…

On Fri, 9 Sep 2022, 16:37 Keshav Prasad, ***@***.***> wrote: @jaingaurav126 <https://github.com/jaingaurav126> please can you create a new thread to discuss? — Reply to this email directly, view it on GitHub <#82 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AP3L4IW4AT3FFLBHUMS6WI3V5MK5PANCNFSM57J6PY3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>