Unable to Publish Content – Status Fails with "Error while fetching definition cache"

[siddhishinde0723]

Hello Team,

We are encountering an issue while trying to publish created content. The publish operation fails, and the content status is marked as FAILED. The error message shown is:Error while fetching definition cache.

Error Logs:
Caused by: org.sunbird.job.exception.InvalidEventException: Error while fetching definition cache. at org.sunbird.job.searchindexer.functions.CompositeSearchIndexerFunction.processElement(CompositeSearchIndexerFunction.scala:48) ... Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000) at org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.howToHandleFailure(AdaptiveScheduler.java:1073) ... Caused by: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: FAILED at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:71) ...

Observed Behavior:

1. Content creation completes successfully.
2. When attempting to publish, the process fails with an error related to definition cache.
3. Flink job associated with processing the publish request fails after retrying.

Please help identify the root cause of the definition cache error and advise on a fix or workaround to enable successful publishing of content.

cc: @aimansharief @vinod303 @pallakartheekreddy

[aimansharief]

Hey @siddhishinde0723, can you please refer to this below link and add those required configurations related to flink.

project-sunbird/sunbird-ed-installer@97bfd3d#diff-24379206580d363ffa4dece04d0cd497eb83db8cd8083a58aaebd47958266152

[siddhishinde0723]

Was there any changes done?
Why this impaced all the old deployments?

[aimansharief]

Since the configuration schema.basePath wasn't added in our configurations, our code was picking up a default value. That default was pointed to azure blob, which we have discontinued now. Because of this, we might have to explicitly set the configuration.

[siddhishinde0723]

I am still facing this issue in the search indexer job manager

`Starting standalonejob as a console application on host search-indexer-jobmanager-575745c8b5-hlqmp.
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2025-05-21 14:13:40,568 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2025-05-21 14:13:40,584 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Preconfiguration:
2025-05-21 14:13:40,584 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -

RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456
dynamic_configs: -D jobmanager.memory.off-heap.size=0b -D jobmanager.memory.jvm-overhead.min=335544320b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1073741824b -D jobmanager.memory.jvm-overhead.max=335544320b
logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
INFO [] - Loading configuration property: jobmanager.memory.flink.size, 1024m
INFO [] - Loading configuration property: taskmanager.memory.flink.size, 1024m
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
INFO [] - Loading configuration property: jobManager.numberOfTaskSlots, 1
INFO [] - Loading configuration property: parallelism.default, 1
INFO [] - Loading configuration property: jobmanager.execution.failover-strategy, region
INFO [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
INFO [] - Loading configuration property: scheduler-mode, reactive
INFO [] - Loading configuration property: heartbeat.timeout, 8000
INFO [] - Loading configuration property: heartbeat.interval, 5000
INFO [] - Loading configuration property: taskmanager.memory.process.size, 1700m
INFO [] - Loading configuration property: jobmanager.memory.process.size, 1600m
INFO [] - The Off-Heap Memory size (0 bytes) is derived the configured Total Flink Memory size (1024.000mb (1073741824 bytes)) minus the configured JVM Heap Memory size (1024.000mb (1073741824 bytes)). The default Off-Heap Memory size (128.000mb (134217728 bytes)) is ignored.
INFO [] - Final Master Memory configuration:
INFO [] - Total Process Memory: 1.563gb (1677721600 bytes)
INFO [] - Total Flink Memory: 1024.000mb (1073741824 bytes)
INFO [] - JVM Heap: 1024.000mb (1073741824 bytes)
INFO [] - Off-heap: 0 bytes
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 320.000mb (335544320 bytes)

2025-05-21 14:13:40,587 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2025-05-21 14:13:40,590 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneApplicationClusterEntryPoint (Version: 1.13.6, Scala: 2.12, Rev:b2ca390, Date:2022-02-03T14:54:22+01:00)
2025-05-21 14:13:40,591 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: root
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/flink/lib/asset-enrichment-1.0.0.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2025-05-21 14:13:41,914 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: root
2025-05-21 14:13:41,914 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 11/11.0.13+8
2025-05-21 14:13:41,916 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 989 MiBytes
2025-05-21 14:13:41,917 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-11
2025-05-21 14:13:41,921 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Hadoop version: 2.7.3
2025-05-21 14:13:41,922 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2025-05-21 14:13:41,923 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx1073741824
2025-05-21 14:13:41,925 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms1073741824
2025-05-21 14:13:41,926 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456
2025-05-21 14:13:41,927 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--standalonejob-0-search-indexer-jobmanager-575745c8b5-hlqmp.log
2025-05-21 14:13:41,927 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2025-05-21 14:13:41,929 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2025-05-21 14:13:41,930 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2025-05-21 14:13:41,930 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments:
2025-05-21 14:13:41,931 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --configDir
2025-05-21 14:13:41,931 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /opt/flink/conf
2025-05-21 14:13:41,931 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2025-05-21 14:13:41,934 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=0b
2025-05-21 14:13:41,934 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=335544320b
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=1073741824b
2025-05-21 14:13:41,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2025-05-21 14:13:41,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=335544320b
2025-05-21 14:13:41,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.access-key=AKIATWBJ2JZXV4E2SOWC
2025-05-21 14:13:41,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - ****** (sensitive information)
2025-05-21 14:13:41,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.endpoint=
2025-05-21 14:13:41,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.path.style.access=
2025-05-21 14:13:41,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.access-key=AKIATWBJ2JZXV4E2SOWC
2025-05-21 14:13:41,944 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - ****** (sensitive information)
2025-05-21 14:13:41,944 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.endpoint=
2025-05-21 14:13:41,944 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Ds3.path.style.access=
2025-05-21 14:13:41,944 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --job-classname=org.sunbird.job.searchindexer.task.SearchIndexerStreamTask
2025-05-21 14:13:41,944 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dweb.submit.enable=false
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dmetrics.reporter.prom.class=org.apache.flink.metrics.prometheus.PrometheusReporter
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dmetrics.reporter.prom.port=9250
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Djobmanager.rpc.address=search-indexer-jobmanager
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Djobmanager.rpc.port=6123
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dparallelism.default=1
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dblob.server.port=6124
2025-05-21 14:13:41,945 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dqueryable-state.server.ports=6125
2025-05-21 14:13:41,946 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Djobmanager.memory.heap.size=1024m
2025-05-21 14:13:41,946 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --config.file.path
2025-05-21 14:13:41,946 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /data/flink/conf/search-indexer.conf
2025-05-21 14:13:41,946 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/asset-enrichment-1.0.0.jar:/opt/flink/lib/audit-event-generator-1.0.0.jar:/opt/flink/lib/audit-history-indexer-1.0.0.jar:/opt/flink/lib/auto-creator-v2-1.0.0.jar:/opt/flink/lib/cassandra-data-migration-1.0.0.jar:/opt/flink/lib/content-auto-creator-1.0.0.jar:/opt/flink/lib/content-publish-1.0.0.jar:/opt/flink/lib/csp-migrator-1.0.0.jar:/opt/flink/lib/dialcode-context-updater-1.0.0.jar:/opt/flink/lib/flink-aaa-s3-fs-presto-1.13.5.jar:/opt/flink/lib/flink-azure-fs-hadoop-1.13.5.jar:/opt/flink/lib/flink-csv-1.13.5.jar:/opt/flink/lib/flink-json-1.13.5.jar:/opt/flink/lib/flink-metrics-prometheus_2.12-1.13.5.jar:/opt/flink/lib/flink-s3-fs-hadoop-1.13.5.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-streaming-scala_2.12-1.13.6.jar:/opt/flink/lib/flink-table-blink_2.12-1.13.5.jar:/opt/flink/lib/flink-table_2.12-1.13.5.jar:/opt/flink/lib/live-node-publisher-1.0.0.jar:/opt/flink/lib/live-video-stream-generator-1.0.0.jar:/opt/flink/lib/log4j-1.2-api-2.16.0.jar:/opt/flink/lib/log4j-api-2.16.0.jar:/opt/flink/lib/log4j-core-2.16.0.jar:/opt/flink/lib/log4j-slf4j-impl-2.16.0.jar:/opt/flink/lib/post-publish-processor-1.0.0.jar:/opt/flink/lib/qrcode-image-generator-1.0.0.jar:/opt/flink/lib/scala-library-2.12.11.jar:/opt/flink/lib/search-indexer-1.0.0.jar:/opt/flink/lib/transaction-event-processor-1.0.0.jar:/opt/flink/lib/video-stream-generator-1.0.0.jar:/opt/flink/lib/flink-dist_2.12-1.13.5.jar:::
2025-05-21 14:13:41,946 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --------------------------------------------------------------------------------
2025-05-21 14:13:41,982 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT[]
2025-05-21 14:13:42,014 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.flink.size, 1024m
2025-05-21 14:13:42,014 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.flink.size, 1024m
2025-05-21 14:13:42,014 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2025-05-21 14:13:42,015 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobManager.numberOfTaskSlots, 1
2025-05-21 14:13:42,015 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2025-05-21 14:13:42,015 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2025-05-21 14:13:42,015 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
2025-05-21 14:13:42,016 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: scheduler-mode, reactive
2025-05-21 14:13:42,016 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: heartbeat.timeout, 8000
2025-05-21 14:13:42,016 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: heartbeat.interval, 5000
2025-05-21 14:13:42,016 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1700m
2025-05-21 14:13:42,017 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2025-05-21 14:13:42,199 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneApplicationClusterEntryPoint.
2025-05-21 14:13:42,232 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem.
2025-05-21 14:13:42,393 ERROR org.apache.flink.core.fs.FileSystem [] - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3hadoop.S3FileSystemFactory could not be instantiated
at java.util.ServiceLoader.fail(Unknown Source) ~[?:?]
at java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) ~[?:?]
at java.util.ServiceLoader$ProviderImpl.get(Unknown Source) ~[?:?]
at java.util.ServiceLoader$3.next(Unknown Source) ~[?:?]
at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1077) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1058) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:338) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:225) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.13.5.jar:1.13.5[]
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.(java.lang.String, org.apache.flink.fs.s3hadoop.common.HadoopConfigLoader)'
at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.(S3FileSystemFactory.java:50) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5[]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
... 10 more
2025-05-21 14:13:42,402 ERROR org.apache.flink.core.fs.FileSystem [] - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3hadoop.S3AFileSystemFactory could not be instantiated
at java.util.ServiceLoader.fail(Unknown Source) ~[?:?]
at java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) ~[?:?]
at java.util.ServiceLoader$ProviderImpl.get(Unknown Source) ~[?:?]
at java.util.ServiceLoader$3.next(Unknown Source) ~[?:?]
at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1077) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1058) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:338) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:225) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600) [transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.13.5.jar:1.13.5[]
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.(java.lang.String, org.apache.flink.fs.s3hadoop.common.HadoopConfigLoader)'
at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.(S3FileSystemFactory.java:50) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5[]
at org.apache.flink.fs.s3hadoop.S3AFileSystemFactory.(S3AFileSystemFactory.java:22) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5[]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
... 10 more
2025-05-21 14:13:42,486 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context.
2025-05-21 14:13:42,510 WARN org.apache.flink.runtime.util.HadoopUtils [] - Could not find Hadoop configuration via any of the supported methods (Flink configuration, environment variables).
2025-05-21 14:13:42,685 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE)
2025-05-21 14:13:42,694 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-11710596480317070681.conf.
2025-05-21 14:13:42,708 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services.
2025-05-21 14:13:42,796 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address search-indexer-jobmanager:6123, bind address 0.0.0.0:6123.
2025-05-21 14:13:46,301 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@search-indexer-jobmanager:6123
2025-05-21 14:13:46,410 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
2025-05-21 14:13:46,423 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-0acf56cf-31e5-42d9-8570-9748b8626a06
2025-05-21 14:13:46,428 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2025-05-21 14:13:46,585 WARN org.apache.flink.runtime.metrics.ReporterSetup [] - Multiple implementations of the same reporter were found in 'lib' and/or 'plugins' directories for org.apache.flink.metrics.prometheus.PrometheusReporterFactory. It is recommended to remove redundant reporter JARs to resolve used versions' ambiguity.
2025-05-21 14:13:46,585 WARN org.apache.flink.runtime.metrics.ReporterSetup [] - Multiple implementations of the same reporter were found in 'lib' and/or 'plugins' directories for org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory. It is recommended to remove redundant reporter JARs to resolve used versions' ambiguity.
2025-05-21 14:13:46,685 INFO org.apache.flink.metrics.prometheus.PrometheusReporter [] - Started PrometheusReporter HTTP server on port 9250.
2025-05-21 14:13:46,690 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - Reporting metrics for reporter prom of type org.apache.flink.metrics.prometheus.PrometheusReporter.
2025-05-21 14:13:46,698 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address search-indexer-jobmanager:0, bind address 0.0.0.0:0.
2025-05-21 14:13:46,886 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@search-indexer-jobmanager:41399
2025-05-21 14:13:46,914 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
2025-05-21 14:13:47,097 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
2025-05-21 14:13:47,101 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Upload directory /tmp/flink-web-50835618-ace1-4f82-87f0-3de4e7cbe2d3/flink-web-upload does not exist.
2025-05-21 14:13:47,103 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Created directory /tmp/flink-web-50835618-ace1-4f82-87f0-3de4e7cbe2d3/flink-web-upload for file uploads.
2025-05-21 14:13:47,133 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Starting rest endpoint.
2025-05-21 14:13:47,825 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/flink--standalonejob-0-search-indexer-jobmanager-575745c8b5-hlqmp.log
2025-05-21 14:13:47,826 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/flink--standalonejob-0-search-indexer-jobmanager-575745c8b5-hlqmp.out
2025-05-21 14:13:48,238 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Rest endpoint listening at search-indexer-jobmanager:8081
2025-05-21 14:13:48,242 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://search-indexer-jobmanager:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2025-05-21 14:13:48,248 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Web frontend listening at http://search-indexer-jobmanager:8081.
2025-05-21 14:13:48,327 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_0 .
2025-05-21 14:13:48,387 INFO org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - DefaultDispatcherRunner was granted leadership with leader id 00000000-0000-0000-0000-000000000000. Creating new DispatcherLeaderProcess.
2025-05-21 14:13:48,398 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess.
2025-05-21 14:13:48,401 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Starting the resource manager.
2025-05-21 14:13:48,407 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - ResourceManager akka.tcp://flink@search-indexer-jobmanager:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token 00000000000000000000000000000000
2025-05-21 14:13:48,413 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs.
2025-05-21 14:13:48,415 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 0 persisted job graphs.
2025-05-21 14:13:48,497 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 .
2025-05-21 14:13:48,620 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: true)
2025-05-21 14:13:48,801 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.sunbird.job.searchindexer.compositesearch.domain.Event does not contain a setter for field jobName
2025-05-21 14:13:48,802 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.sunbird.job.searchindexer.compositesearch.domain.Event cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2025-05-21 14:13:48,910 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - class org.sunbird.job.searchindexer.compositesearch.domain.Event does not contain a setter for field jobName
2025-05-21 14:13:48,911 INFO org.apache.flink.api.java.typeutils.TypeExtractor [] - Class class org.sunbird.job.searchindexer.compositesearch.domain.Event cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2025-05-21 14:13:49,484 WARN org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property [transaction.timeout.ms[] not specified. Setting it to 3600000 ms
2025-05-21 14:13:49,507 WARN org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property [transaction.timeout.ms[] not specified. Setting it to 3600000 ms
2025-05-21 14:13:49,509 WARN org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property [transaction.timeout.ms[] not specified. Setting it to 3600000 ms
2025-05-21 14:13:49,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job 16be8a5e7fd7dc6c0a17732f5bb58801 is submitted.
2025-05-21 14:13:49,806 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Submitting Job with JobId=16be8a5e7fd7dc6c0a17732f5bb58801.
2025-05-21 14:13:49,818 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received JobGraph submission 16be8a5e7fd7dc6c0a17732f5bb58801 (search-indexer).
2025-05-21 14:13:49,821 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting job 16be8a5e7fd7dc6c0a17732f5bb58801 (search-indexer).
2025-05-21 14:13:49,981 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_2 .
2025-05-21 14:13:49,998 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801).
2025-05-21 14:13:50,028 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000) for search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801).
2025-05-21 14:13:50,115 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801) under job master id 00000000000000000000000000000000.
2025-05-21 14:13:50,124 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@search-indexer-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
2025-05-21 14:13:50,182 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration
2025-05-21 14:13:50,186 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager [email protected]://flink@search-indexer-jobmanager:6123/user/rpc/jobmanager_2 for job 16be8a5e7fd7dc6c0a17732f5bb58801.
2025-05-21 14:13:50,204 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager [email protected]://flink@search-indexer-jobmanager:6123/user/rpc/jobmanager_2 for job 16be8a5e7fd7dc6c0a17732f5bb58801.
2025-05-21 14:13:50,215 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000.
2025-05-21 14:13:50,221 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 16be8a5e7fd7dc6c0a17732f5bb58801: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=128}]
2025-05-21 14:13:54,002 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 10.0.9.91:6122-209737 (akka.tcp://[email protected]:6122/user/rpc/taskmanager_0) at ResourceManager
2025-05-21 14:13:54,042 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 10.0.9.91:6122-209737 (akka.tcp://[email protected]:6122/user/rpc/taskmanager_0) at ResourceManager
2025-05-21 14:13:54,308 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Running initialization on master for job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801).
2025-05-21 14:13:54,311 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Successfully ran initialization on master in 3 ms.
2025-05-21 14:13:54,337 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 1 ms
2025-05-21 14:13:54,395 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@53d59e91
2025-05-21 14:13:54,399 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Checkpoint storage is set to 'jobmanager'
2025-05-21 14:13:54,428 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No checkpoint found during restore.
2025-05-21 14:13:54,433 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801) switched from state CREATED to RUNNING.
2025-05-21 14:13:54,506 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: search-indexer-consumer (1/1) (8571836500ab9d463bcb75e1aab2c69d) switched from CREATED to DEPLOYING.
2025-05-21 14:13:54,509 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source: search-indexer-consumer (1/1) (attempt #0) with attempt id 8571836500ab9d463bcb75e1aab2c69d to 10.0.9.91:6122-209737 @ 10-0-9-91.search-indexer-taskmanager.sunbird.svc.cluster.local (dataPort=39535) with allocation id a65b7eefeb2c1bb2aa42c71187376913
2025-05-21 14:13:54,521 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - transaction-event-router -> (composite-search-indexer -> Sink: Unnamed, dialcode-external-indexer -> Sink: Unnamed, dialcode-metric-indexer -> Sink: Unnamed) (1/1) (87b1abb2e640e59244d04a441ff56cd0) switched from CREATED to DEPLOYING.
2025-05-21 14:13:54,524 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying transaction-event-router -> (composite-search-indexer -> Sink: Unnamed, dialcode-external-indexer -> Sink: Unnamed, dialcode-metric-indexer -> Sink: Unnamed) (1/1) (attempt #0) with attempt id 87b1abb2e640e59244d04a441ff56cd0 to 10.0.9.91:6122-209737 @ 10-0-9-91.search-indexer-taskmanager.sunbird.svc.cluster.local (dataPort=39535) with allocation id a65b7eefeb2c1bb2aa42c71187376913
2025-05-21 14:13:54,608 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: search-indexer-consumer (1/1) (8571836500ab9d463bcb75e1aab2c69d) switched from DEPLOYING to INITIALIZING.
2025-05-21 14:13:54,652 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - transaction-event-router -> (composite-search-indexer -> Sink: Unnamed, dialcode-external-indexer -> Sink: Unnamed, dialcode-metric-indexer -> Sink: Unnamed) (1/1) (87b1abb2e640e59244d04a441ff56cd0) switched from DEPLOYING to INITIALIZING.
2025-05-21 14:13:54,685 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: search-indexer-consumer (1/1) (8571836500ab9d463bcb75e1aab2c69d) switched from INITIALIZING to RUNNING.
2025-05-21 14:13:55,121 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - transaction-event-router -> (composite-search-indexer -> Sink: Unnamed, dialcode-external-indexer -> Sink: Unnamed, dialcode-metric-indexer -> Sink: Unnamed) (1/1) (87b1abb2e640e59244d04a441ff56cd0) switched from INITIALIZING to RUNNING.
2025-05-21 14:13:55,277 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - transaction-event-router -> (composite-search-indexer -> Sink: Unnamed, dialcode-external-indexer -> Sink: Unnamed, dialcode-metric-indexer -> Sink: Unnamed) (1/1) (87b1abb2e640e59244d04a441ff56cd0) switched from RUNNING to FAILED on 10.0.9.91:6122-209737 @ 10-0-9-91.search-indexer-taskmanager.sunbird.svc.cluster.local (dataPort=39535).
org.sunbird.job.exception.InvalidEventException: Error while fetching definition cache.
at org.sunbird.job.searchindexer.functions.CompositeSearchIndexerFunction.processElement(CompositeSearchIndexerFunction.scala:48) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.CompositeSearchIndexerFunction.processElement(CompositeSearchIndexerFunction.scala:17) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:49) ~[asset-enrichment-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.KeyedProcessOperator$ContextImpl.output(KeyedProcessOperator.java:129) ~[transaction-event-processor-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.TransactionEventRouter.processElement(TransactionEventRouter.scala:31) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.TransactionEventRouter.processElement(TransactionEventRouter.scala:13) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.BaseProcessKeyedFunction.processElement(BaseProcessFunction.scala:119) ~[asset-enrichment-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[transaction-event-processor-1.0.0.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-05-21 14:13:55,334 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Restarting job.
org.sunbird.job.exception.InvalidEventException: Error while fetching definition cache.
at org.sunbird.job.searchindexer.functions.CompositeSearchIndexerFunction.processElement(CompositeSearchIndexerFunction.scala:48) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.CompositeSearchIndexerFunction.processElement(CompositeSearchIndexerFunction.scala:17) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:49) ~[asset-enrichment-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.KeyedProcessOperator$ContextImpl.output(KeyedProcessOperator.java:129) ~[transaction-event-processor-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.TransactionEventRouter.processElement(TransactionEventRouter.scala:31) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.searchindexer.functions.TransactionEventRouter.processElement(TransactionEventRouter.scala:13) ~[search-indexer-1.0.0.jar:?]
at org.sunbird.job.BaseProcessKeyedFunction.processElement(BaseProcessFunction.scala:119) ~[asset-enrichment-1.0.0.jar:?]
at org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[transaction-event-processor-1.0.0.jar:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[transaction-event-processor-1.0.0.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-05-21 14:13:55,344 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801) switched from state RUNNING to CANCELLING.
2025-05-21 14:13:55,347 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: search-indexer-consumer (1/1) (8571836500ab9d463bcb75e1aab2c69d) switched from RUNNING to CANCELING.
2025-05-21 14:13:55,402 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: search-indexer-consumer (1/1) (8571836500ab9d463bcb75e1aab2c69d) switched from CANCELING to CANCELED.
2025-05-21 14:13:55,407 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job search-indexer (16be8a5e7fd7dc6c0a17732f5bb58801) switched from state CANCELLING to CANCELED.
2025-05-21 14:13:55,408 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Stopping checkpoint coordinator for job 16be8a5e7fd7dc6c0a17732f5bb58801.`

[aimansharief]

@siddhishinde0723 As discussed, updating the configuration resolved the issues in one of the instances. Please ensure the same changes are applied to the remaining instances and let us know if you face any issue.