kamon-bundle: None of the Kanela instrumentation modules showing up

[xtrntr]

My build.sbt settings for a fat JAR:

assemblyMergeStrategy in assembly := {
     case "reference.conf" => MergeStrategy.concat
     case anyOther         => (assemblyMergeStrategy in assembly).value(anyOther)
}

libraryDependencies ++= Seq(
     "io.kamon" %% "kamon-bundle" % "2.1.4",
     "io.kamon" %% "kamon-prometheus" % "2.1.4",
     "com.typesafe" % "config" % "1.4.0",
     "com.typesafe.akka" %% "akka-actor" % "2.6.5",
     "com.typesafe.akka" %% "akka-testkit" % "2.6.5" % Test,
     "com.typesafe.akka" %% "akka-stream" % "2.6.5",
     "com.typesafe.akka" %% "akka-stream-testkit" % "2.6.5" % Test,
     "com.typesafe.akka" %% "akka-http" % "10.1.12",
     "com.typesafe.akka" %% "akka-http-testkit" % "10.1.12" % Test,   
     ...
)

If it's relevant, this JAR is deployed in production as a long running Spark application that acts as a web server and occasionally runs Spark operations:

 gosu "${SPARK_USER}" "${SPARK_SUBMIT}" \
     ${SPARK_SUBMIT_ARGS:-} \
     --name "server" \
     --class ...ServerApp \
     --driver-java-options "-Dconfig.file=application.conf ${SPARK_DRIVER_OPTIONS:-}" \
     --files application.conf \
     application.jar

Kamon.init() is run as the first thing of the main application.

Printing out the ConfigFactory values (that i thought were relevant):

kamon.instrumentation.http-server.default.tracing.preferred-trace-id-tag=Quoted("none")
kamon.instrumentation.http-server.default.tracing.tags.url=Unquoted("span")
kamon.instrumentation.http-server.default.tracing.span-metrics=Unquoted("on")
kamon.instrumentation.http-server.default.tracing.response-headers.span-id=Unquoted("none")
kamon.instrumentation.http-server.default.tracing.operations.default=Quoted("http.server.request")
kamon.instrumentation.http-server.default.tracing.tags.method=Unquoted("metric")
kamon.instrumentation.http-server.default.propagation.channel=Quoted("default")
kamon.instrumentation.http-server.default.metrics.enabled=Unquoted("yes")
kamon.instrumentation.http-server.default.tracing.tags.status-code=Unquoted("metric")
kamon.instrumentation.http-server.default.tracing.operations.unhandled=Quoted("unhandled")
kamon.instrumentation.http-server.default.tracing.response-headers.trace-id=Quoted("trace-id")
kamon.instrumentation.http-server.default.tracing.enabled=Unquoted("yes")
kamon.instrumentation.http-server.default.tracing.operations.name-generator=Quoted("default")
kamon.instrumentation.http-server.default.propagation.enabled=Unquoted("yes")

 kanela.modules.kafka-clients.instrumentations=SimpleConfigList(["kamon.instrumentation.kafka.client.ProducerInstrumentation","kamon.instrumentation.kafka.client.ConsumerInst\
  rumentation"])
 kanela.modules.akka-remote.instrumentations=SimpleConfigList(["kamon.instrumentation.akka.remote.MessageBufferInstrumentation","kamon.instrumentation.akka.remote.ShardingInstrumentation","kamon.i\
 nstrumentation.akka.instrumentations.akka_25.remote.RemotingInstrumentation","kamon.instrumentation.akka.instrumentations.akka_26.remote.RemotingInstrumentation"])
 kanela.modules.executor-service.exclude=SimpleConfigList(["^java.*","^sun.*","^com.sun.tools.*","^sbt.internal.*","^com.intellij.rt.*","^scala.concurrent.*","^org.jboss.netty.*","^com.google.comm\
 on.base.internal.Finalizer","^kamon.module.*","^kamon.instrumentation.executor.ExecutorInstrumentation.*","^kanela.agent.*","com.twitter.util.ConstFuture.*"])
 kanela.modules.mongo-driver.instrumentations=SimpleConfigList(["kamon.instrumentation.mongo.MongoClientInstrumentation"])
 kanela.modules.jdbc.instrumentations=SimpleConfigList(["kamon.instrumentation.jdbc.StatementInstrumentation","kamon.instrumentation.jdbc.HikariInstrumentation"])
 kanela.modules.cassandra-driver.instrumentations=SimpleConfigList(["kamon.instrumentation.cassandra.driver.DriverInstrumentation","kamon.instrumentation.cassandra.executors.DriverExecutorInstrume\
 ntation"])
 kanela.modules.annotation.instrumentations=SimpleConfigList(["kamon.annotation.instrumentation.AnnotationInstrumentation"])
 kanela.modules.akka.instrumentations=SimpleConfigList(["kamon.instrumentation.akka.instrumentations.EnvelopeInstrumentation","kamon.instrumentation.akka.instrumentations.SystemMessageInstrumentation","kamon.instrumentation.akka.instrumentations.RouterInstrumentation","kamon.instrumentation.akka.instrumentations.ActorInstrumentation","kamon.instrumentation.akka.instrumentations.ActorLoggingInstrumentation","kamon.instrumentation.akka.instrumentations.AskPatternInstrumentation","kamon.instrumentation.akka.instrumentations.EventStreamInstrumentation","kamon.instrumentation.akka.instrumentations.ActorRefInstrumentation","kamon.instrumentation.akka.instrumentations.akka_25.DispatcherInstrumentation","kamon.instrumentation.akka.instrumentations.akka_26.DispatcherInstrumentation"])
kanela.modules.executor-service.instrumentations=SimpleConfigList(["kamon.instrumentation.executor.ExecutorTaskInstrumentation"])
kanela.modules.logback.instrumentations=SimpleConfigList(["kamon.instrumentation.logback.LogbackInstrumentation"])
kanela.modules.scala-future.instrumentations=SimpleConfigList(["kamon.instrumentation.futures.scala.FutureChainingInstrumentation"])
kanela.modules.play-framework.instrumentations=SimpleConfigList(["kamon.instrumentation.play.PlayServerInstrumentation","kamon.instrumentation.play.PlayClientInstrumentation"])
kanela.modules.executor-service-capture-on-submit.instrumentations=SimpleConfigList(["kamon.instrumentation.executor.CaptureContextOnSubmitInstrumentation"])
kanela.modules.akka-http.instrumentations=SimpleConfigList(["kamon.instrumentation.akka.http.AkkaHttpServerInstrumentation","kamon.instrumentation.akka.http.AkkaHttpClientInstrumentation"])

Status page:

The Kanela agent starts without any visible errors:

  _  __                _        ______
  | |/ /               | |       \ \ \ \
  | ' / __ _ _ __   ___| | __ _   \ \ \ \
  |  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
  | . \ (_| | | | |  __/ | (_| |  / / / /
  |_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

  ==============================
  Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.6)
  [Attach Listener] DEBUG 2020-09-18 08:16:12  Logger : Loaded configuration => {}
  [Attach Listener] INFO 2020-09-18 08:16:12  Logger : Reinstrumenter activated.
  [Attach Listener] DEBUG 2020-09-18 08:16:12  Logger : Reinstrumenter is listening for Reinstrumentation Events.
  [Attach Listener] INFO 2020-09-18 08:16:12  Logger : Startup completed in 595 ms

[xtrntr]

For future reference: the issue was with this:
[Attach Listener] DEBUG 2020-09-18 08:16:12 Logger : Loaded configuration => {}

I ended up attaching the Kanela instrumentation agent with the javaagent option:

# Before
 gosu "${SPARK_USER}" "${SPARK_SUBMIT}" \
     ${SPARK_SUBMIT_ARGS:-} \
     --name "server" \
     --class ...ServerApp \
     --driver-java-options "-Dconfig.file=application.conf ${SPARK_DRIVER_OPTIONS:-}" \
     --files application.conf \
     application.jar

# After
wget -O kanela-agent.jar "https://search.maven.org/remote_content?g=io.kamon&a=kanela-agent&v=${KANELA_AGENT_VERSION}"

...

 gosu "${SPARK_USER}" "${SPARK_SUBMIT}" \
     ${SPARK_SUBMIT_ARGS:-} \
     --name "server" \
     --class ...ServerApp \
     --driver-java-options "-Dconfig.file=application.conf ${SPARK_DRIVER_OPTIONS:-} -javaagent:/app/kanela-agent.jar" \
     --driver-class-path "/app/application.jar" \
     --jars "/app/application.jar" \
     --files application.conf \
     application.jar

The application fat JAR was created with https://github.com/sbt/sbt-assembly, using these configs:

   assemblyMergeStrategy in assembly := {
     case "application.conf" => MergeStrategy.concat
     case "reference.conf" => MergeStrategy.concat
     case anyOther         => (assemblyMergeStrategy in assembly).value(anyOther)
   }

[SimunKaracic]

Thank you for the very detailed bug report 👍 @xtrntr

This was most likely a duplicate of #601
Which means some code got loaded before Kamon was initialized.