Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext #601

Closed
lozenko opened this issue Jul 31, 2019 · 14 comments
Closed

Comments

@lozenko
Copy link

lozenko commented Jul 31, 2019

Added kamon-io to the project and receive the following on every API call.

build.sbt

    lazy val kamonVersion     = "2.0.0"
    ...
    "io.kamon"                   %% "kamon-bundle"            % kamonVersion,
    "io.kamon"                   %% "kamon-akka-http"         % kamonVersion,
    "io.kamon"                   %% "kamon-prometheus"        % kamonVersion,
    "io.kamon"                   %% "kamon-zipkin"            % kamonVersion,

Using only Kamon.init() for now.

[ERROR] [08/01/2019 02:36:01.836] [insights-dashboard-service-akka.actor.default-dispatcher-4] [akka://dashboard-service/system/StreamSupervisor-0/flow-2-0-detacher] Error in stage [akka.http.impl.engine.server.HttpServerBluePrint$RequestTimeoutSupport@3305ab7f]: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
java.lang.ClassCastException: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
	at kamon.instrumentation.akka.http.FastFutureTransformWithAdvice$.transformWith(AkkaHttpServerInstrumentation.scala:267)
	at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala)
	at akka.http.scaladsl.util.FastFuture$.map$extension(FastFuture.scala:23)
	at akka.http.impl.engine.server.HttpServerBluePrint$TimeoutAccessImpl.<init>(HttpServerBluePrint.scala:315)
	at akka.http.impl.engine.server.HttpServerBluePrint$RequestTimeoutSupport$$anon$3$$anon$4.onPush(HttpServerBluePrint.scala:266)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
	at akka.actor.Actor.aroundReceive(Actor.scala:539)
	at akka.actor.Actor.aroundReceive$(Actor.scala:537)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
	at akka.actor.ActorCell.invoke(ActorCell.scala:581)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Any ideas what it could be

@itsnein
Copy link

itsnein commented Aug 3, 2019

Same issue
Just
Kamon.init()
and
bindingServerFuture = Http().bindAndHandle(route, address, port)

And on each request
scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext

	<dependency>
	  <groupId>io.kamon</groupId>
	  <artifactId>kamon-bundle_2.12</artifactId>
	  <version>2.0.0</version>
	</dependency>
	<dependency>
	  <groupId>io.kamon</groupId>
	  <artifactId>kamon-akka_2.12</artifactId>
	  <version>2.0.0</version>
	</dependency>
	<dependency>
	    <groupId>io.kamon</groupId>
	    <artifactId>kamon-akka-http_2.12</artifactId>
	    <version>2.0.0</version>
	</dependency>

@ivantopo
Copy link
Contributor

ivantopo commented Aug 6, 2019

Hey folks, I have seen this happening a couple times in cases where Kamon.init() is not really the very first thing that happens in an application. For example, if you have something like this:

object MyApp extends App with HttpApp {
  Kamon.init()
  // the codez
}

In cases like that, even though it seems like we are initializing Kamon first, there is a lot happening before it gets to Kamon so it could happen that the Future class was loaded before the init and it didn't get instrumented because of that. Could it be the case for you?

If it is the case indeed, you could do one of two things: reorder the initialization code so that Kamon goes first (e.g. moving the actual main to a different place that calls your current main) or you could start the app with the -javaagent:kanela.jar option instead, here is some info about how to do it.

By the way, there is kamon-io/kanela#87 which I just reported. It should be a lot easier to figure out where the problem is, but for now, please let me know what you find!

@itsnein
Copy link

itsnein commented Aug 6, 2019

No, it does not help.

Kamon inited first.

Here my code:

class FrontServerKamon(
implicit val system: ActorSystem,
implicit val materializer: ActorMaterializer) {

val log = org.log4s.getLogger
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val myRoutes: Route =
path("check") {
get {
complete(StatusCodes.OK -> "OK")
}
}
private var bindingServerFuture: Future[Http.ServerBinding] = _

def startServer(address: String, port: Int): Unit = {
bindingServerFuture = Http().bindAndHandle(myRoutes, address, port)
bindingServerFuture.onComplete {
case Success(_) => log.info(s"server started on address: $address and port: $port")
case Failure(ex) =>
log.error(ex)(s"Failed to bind to $address:$port!")
system.terminate()
}
}

def stopServer(): Unit = {
bindingServerFuture
.flatMap(.unbind()) // trigger unbinding from the port
.onComplete(
=> system.terminate()) // and shutdown when done
}

}

object FrontServerKamon {
Kamon.init()
def main(args: Array[String]): Unit = {
val server = buildFrontServer(ActorSystem("rest-actor-system"))
server.startServer("localhost", 8081)
}

def buildFrontServer(actorSystem: ActorSystem): FrontServerKamon = {
implicit val system = actorSystem
implicit val materializer = ActorMaterializer()

new FrontServerKamon()

}

}

it starts, and then I run get
http://localhost:8081/check

and get an exception

@ivantopo
Copy link
Contributor

ivantopo commented Aug 7, 2019

I'm not sure of what is the underlying issue that makes it have a different behavior when running via -javaagent vs attaching, but explicitly targeting the classes did the trick for now. Thanks for reporting the issue! The fix will be included in the Kamon Bundle 2.0.1.

In the meantime, if you start the application with -javaagent it will work fine.

@itsnein
Copy link

itsnein commented Aug 7, 2019

For me it does not help. Don't know why.

Added kanela via javaagent:
-javaagent:kanela-agent-1.0.0.jar
removed even Kamon.init()

started


| |/ / | | \ \ \
| ' / __ _ _ __ | | __ _ \ \ \
| < / | '_ \ / _ \ |/ _ | ) ) ) )
| . \ (
| | | | | __/ | (
| | / / / /
|
|__
,|| ||___||_,| ///_/

==============================
�[32;1mRunning with Kanela, the Kamon Instrumentation Agent :: (v1.0.0)�[0m

but then for check method:

ERROR] [08/07/2019 20:52:09.213] [rest-actor-system-akka.actor.default-dispatcher-61] [akka://rest-actor-system/system/StreamSupervisor-0/flow-1-0-detacher] Error in stage [akka.http.impl.engine.server.HttpServerBluePrint$RequestTimeoutSupport@54137cc3]: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
java.lang.ClassCastException: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
at kamon.instrumentation.akka.http.FastFutureTransformWithAdvice$.transformWith(AkkaHttpServerInstrumentation.scala:261)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala)

so:
for this error, I just need:

VM option:
-javaagent:kanela-agent-1.0.0.jar
and kamon dependency.

io.kamon
kamon-bundle_2.12
2.0.0

and error on check method. If no kamon dependency in classpath - kanela started, and check - works OK

wil wait 2.0.1 hope it will help, thanks!

@itsnein
Copy link

itsnein commented Aug 24, 2019

2.0.1 - the same:
�[32;1mRunning with Kanela, the Kamon Instrumentation Agent :: (v1.0.1)�[0m

[INFO ] 2019-08-24 12:50:52.718 FrontServerRoutes - server started on address: 0.0.0.0 and port: 8081
[ERROR] [08/24/2019 12:50:57.962] [rest-actor-system-akka.actor.default-dispatcher-61] [akka://rest-actor-system/system/StreamSupervisor-0/flow-1-0-detacher] Error in stage [akka.http.impl.engine.server.HttpServerBluePrint$RequestTimeoutSupport@40510360]: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
java.lang.ClassCastException: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
at kamon.instrumentation.akka.http.FastFutureTransformWithAdvice$.transformWith(AkkaHttpServerInstrumentation.scala:276)

Just added dependency:

io.kamon
kamon-bundle_2.12
2.0.1

and javaagent
-javaagent:kanela-agent-1.0.1.jar

@rciccone91
Copy link

rciccone91 commented Oct 1, 2019

Hi, I'm also having this problem. Using kamon bundleio.kamon:kamon-bundle_2.12:2.0.1 and javagent avaagent:kanela-agent-1.0.1.jar.
I am getting the following message:

[mocking-system-akka.actor.default-dispatcher-2] ERROR 2019-10-01 14:40:35  Logger : An error occurred while trying to apply an advisor: java.lang.ClassCastException: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
	at kamon.instrumentation.futures.scala.CallbackRunnableRunInstrumentation$.enter(FutureChainingInstrumentation.scala:74)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:63)
	at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:67)
	at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:82)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:59)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:110)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:537)
	at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:93)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:54)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
[mocking-system-akka.actor.default-dispatcher-2] ERROR 2019-10-01 14:40:35  Logger : An error occurred while trying to apply an advisor: java.lang.NullPointerException
	at kamon.instrumentation.futures.scala.CallbackRunnableRunInstrumentation$.exit(FutureChainingInstrumentation.scala:84)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:67)
	at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:82)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:59)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:110)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:537)
	at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:93)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:54)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
[mocking-system-akka.actor.default-dispatcher-2] ERROR 2019-10-01 14:40:35  Logger : An error occurred while trying to apply an advisor: java.lang.ClassCastException: scala.util.Success cannot be cast to kamon.instrumentation.context.HasContext
	at kamon.instrumentation.futures.scala.CallbackRunnableRunInstrumentation$.enter(FutureChainingInstrumentation.scala:74)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:63)
	at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:67)
	at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:82)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:59)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:110)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:537)
	at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:93)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:54)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
[mocking-system-akka.actor.default-dispatcher-2] ERROR 2019-10-01 14:40:35  Logger : An error occurred while trying to apply an advisor: java.lang.NullPointerException
	at kamon.instrumentation.futures.scala.CallbackRunnableRunInstrumentation$.exit(FutureChainingInstrumentation.scala:84)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:67)
	at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:82)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:59)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:874)
	at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:110)
	at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:107)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:872)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:537)
	at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:93)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.event.slf4j.Slf4jLogger.aroundReceive(Slf4jLogger.scala:54)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:653)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

Could someone please help us with this?
Thanks

@rciccone91
Copy link

@ivantopo I'm still having this issue, could you please help me solve it? Is it something I'm missing or it's just not working?
Thanks

@ivantopo
Copy link
Contributor

@itsnein @lozenko are you still having this issue? Something similar to this just came up on our Gitter channel and I just tried again the same example that @itsnein shared above and it worked as expected. Just for reference, here are the exact dependencies and code I used:

libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.26"
libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.26"
libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.1.10"
libraryDependencies += "io.kamon" %% "kamon-bundle" % "2.0.4"
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.StatusCodes
import akka.http.scaladsl.server.{Directives, Route}
import akka.stream.ActorMaterializer
import kamon.Kamon
import org.slf4j.LoggerFactory

import scala.concurrent.Future
import scala.util.{Failure, Success}

class FrontServerKamon(
  implicit val system: ActorSystem,
  implicit val materializer: ActorMaterializer) {

  import Directives.{path, get, complete}
  val log = LoggerFactory.getLogger("FrontServerKamon")


  // needed for the future flatMap/onComplete in the end
  implicit val executionContext = system.dispatcher
  val myRoutes: Route =
    path("check") {
      get {
        complete(StatusCodes.OK -> "OK")
      }
    }
  private var bindingServerFuture: Future[Http.ServerBinding] = _

  def startServer(address: String, port: Int): Unit = {
    bindingServerFuture = Http().bindAndHandle(myRoutes, address, port)
    bindingServerFuture.onComplete {
      case Success(_) => log.info(s"server started on address: $address and port: $port")
      case Failure(ex) =>
        log.error(s"Failed to bind to $address:$port!", ex)
        system.terminate()
    }
  }

  def stopServer(): Unit = {
    bindingServerFuture
      .flatMap(_.unbind()) // trigger unbinding from the port
      .onComplete(_ => system.terminate()) // and shutdown when done
  }

}


object FrontServerKamon {
  Kamon.init()

  def main(args: Array[String]): Unit = {
    val server = buildFrontServer(ActorSystem("rest-actor-system"))
    server.startServer("localhost", 8081)
  }

  def buildFrontServer(actorSystem: ActorSystem): FrontServerKamon = {
    implicit val system = actorSystem
    implicit val materializer = ActorMaterializer()

    new FrontServerKamon()
  }

}

@ramyamagham
Copy link

@ivantopo Thanks, your tip about initializing at the very beginning worked for me.

@lustefaniak
Copy link
Contributor

Hey @ivantopo
I'm running Kamon with kanela-agent 1.0.7, I also moved initialization to the main and removed slf4j/logback logger initialization but I keep getting

java.lang.ClassCastException: class scala.util.Success cannot be cast to class kamon.instrumentation.context.HasContext (scala.util.Success and kamon.instrumentation.context.HasContext are in unnamed module of loader 'app')
	at kamon.instrumentation.futures.scala.CleanContextFromSeedFuture$.$anonfun$exit$1(FutureChainingInstrumentation.scala:134)
	at kamon.instrumentation.futures.scala.CleanContextFromSeedFuture$.$anonfun$exit$1$adapted(FutureChainingInstrumentation.scala:134)
	at scala.Option.foreach(Option.scala:437)
	at kamon.instrumentation.futures.scala.CleanContextFromSeedFuture$.exit(FutureChainingInstrumentation.scala:134)
	at scala.concurrent.Future$.<clinit>(Future.scala:514)
	at kamon.instrumentation.system.process.ProcessMetricsCollector$MetricsCollectionTask.schedule(ProcessMetricsCollector.scala:61)
	at kamon.instrumentation.system.process.ProcessMetricsCollector$$anon$1.run(ProcessMetricsCollector.scala:40)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

This started happening after I added kamon-cassandra 🤔 Usually after restart of the pod (killed by healthcheck) it works normally.

What is interesting after initialization and those errors being printed I get from the Java11 a warning

WARNING: Illegal reflective access by kamon.instrumentation.executor.ExecutorInstrumentation$ (file:/opt/chronicle/lib/io.kamon.kamon-executors_2.13-2.1.8.jar) to field java.util.concurrent.Executors$DelegatedExecutorService.e

Looks like ExecutorInstrumentation is loaded later than the code which relies on Futures being instrumented 🤔

@leszekgruchala
Copy link

leszekgruchala commented Dec 21, 2020

I have kamon-bundle 2.0.9 (akka 2.6.10, akka-http 10.1.12) and I get this issue even though I am 100% sure I start the Kamon as the first thing in my app

object Server extends App {

  Kamon.init()

The only way to overcome this is to use -javaagent VM option.

@asnare
Copy link

asnare commented Dec 21, 2020

The underlying issue is that these classes are loaded (inadvertently) prior to the agent attaching, which means they're not instrumented like the agent subsequently assumes. Performing Kamon.init() as early as possible helps, but a whole bunch of things run prior to this due to (recursive) class loading and initialisation.

One way to debug the issue (if you hit it) is to start the application with -verbose:class which will cause the class loaders to log the names of classes as they're loaded. This is normally enough to figure out the (probably transitive!) loading/initialisation path leading to the problem.

Often this problem is intractable, which is why the agent approach works because then the agent is already present for the whole initialisation sequence.

@mrt181
Copy link

mrt181 commented Mar 10, 2021

Happens for me too:

jdk: adopt-openjdk-11.0.10
scala: 2.13.5
sbt: 1.4.7
akka: 2.6.13
kamon: 2.1.12

starting only with -javaagent:kanela-agent.jar, no sbt-kanela-runner, no sbt-javaagent

and this Main

object Main extends App {
  Kamon.init()
  println(Banner.text)
  val userSystem: ActorSystem[Protocol] =
    ActorSystem[Protocol](Protocol(), "mySystem")
}

The moment I initialize the userSystem I get these errors.

 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/
                              
==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.8)

2021-03-10T15:49:12.596Z INFO  [              main]         kamon.status.page.StatusPage  - Status page started on http://0.0.0.0:5266/
2021-03-10T15:49:12.897Z DEBUG [              main] re.os.windows.WindowsOperatingSystem  - Debug privileges not enabled.
2021-03-10T15:49:13.002Z INFO  [              main]  kamon.prometheus.PrometheusReporter  - Started the embedded HTTP server on http://0.0.0.0:9095/metrics
java.lang.ClassCastException: class scala.util.Success cannot be cast to class kamon.instrumentation.context.HasContext (scala.util.Success and kamon.instrumentation.context.HasContext are in unnamed module of loader 'app')

Adding a Thread.sleep(20000) before starting the userSystem fixes the error messages. waiting 10 sec is not enough

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants