Dependency on Aleph and Netty causes issues deploying to wildfly and jboss

[ckirkendall]

The dependency on Netty and Aleph to manage the connection classpath collisions with Wildfly on some deployments. Wildfly uses an older version of netty that is not compatible with Aleph and will throw method not found exceptions.

2017-02-06 22:06:39,104 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-25) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./: Failed to start service
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]
    at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Application initialization failed
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:222)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:87)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.start(UndertowDeploymentService.java:72)
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
    ... 3 more
Caused by: java.lang.RuntimeException: Application initialization failed
    at org.projectodd.wunderboss.as.ServletListener.contextInitialized(ServletListener.java:155)
    at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:173)
    at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:193)
    ... 7 more
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: io.netty.channel.ChannelMetadata.<init>(ZI)V
    at org.projectodd.wunderboss.clojure.ClojureLanguage.eval(ClojureLanguage.java:73)
    at org.projectodd.wunderboss.ApplicationRunner.start(ApplicationRunner.java:52)
    at org.projectodd.wunderboss.as.ServletListener$1.start(ServletListener.java:104)
    at org.projectodd.wunderboss.as.ServletListener$2.run(ServletListener.java:116)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_102]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_102]
    ... 3 more
Caused by: java.lang.NoSuchMethodError: io.netty.channel.ChannelMetadata.<init>(ZI)V
    at io.netty.channel.epoll.AbstractEpollServerChannel.<clinit>(AbstractEpollServerChannel.java:34)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at aleph.netty$start_server.<clinit>(netty.clj:668)
    at aleph.netty__init.load(Unknown Source)
    at aleph.netty__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:482)
    at aleph.tcp$loading__5569__auto____26946.invoke(tcp.clj:1)
    at aleph.tcp__init.load(Unknown Source)
    at aleph.tcp__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:512)
    at rethinkdb.core$loading__5569__auto____26944.invoke(core.clj:1)
    at rethinkdb.core__init.load(Unknown Source)
    at rethinkdb.core__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:457)
    at rethinkdb.query$loading__5569__auto____21532.invoke(query.cljc:1)
    at rethinkdb.query__init.load(Unknown Source)
    at rethinkdb.query__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:619)
    at lego.components.rethinkdb$loading__5569__auto____21530.invoke(rethinkdb.clj:1)
    at lego.components.rethinkdb__init.load(Unknown Source)
    at lego.components.rethinkdb__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:3894)
    at epiphany.system$loading__5569__auto____40.invoke(system.clj:1)
    at epiphany.system__init.load(Unknown Source)
    at epiphany.system__init.<clinit>(Unknown Source)
    at java.lang.Class.forName0(Native Method) [rt.jar:1.8.0_102]
    at java.lang.Class.forName(Class.java:348) [rt.jar:1.8.0_102]
    at clojure.lang.RT.classForName(RT.java:2168)
    at clojure.lang.RT.classForName(RT.java:2177)
    at clojure.lang.RT.loadClassForName(RT.java:2196)
    at clojure.lang.RT.load(RT.java:443)
    at clojure.lang.RT.load(RT.java:419)
    at clojure.core$load$fn__5677.invoke(core.clj:5893)
    at clojure.core$load.invokeStatic(core.clj:5892)
    at clojure.core$load.doInvoke(core.clj:5876)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5697)
    at clojure.core$load_one.invoke(core.clj:5692)
    at clojure.core$load_lib$fn__5626.invoke(core.clj:5737)
    at clojure.core$load_lib.invokeStatic(core.clj:5736)
    at clojure.core$load_lib.doInvoke(core.clj:5717)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$load_libs.invokeStatic(core.clj:5774)
    at clojure.core$load_libs.doInvoke(core.clj:5758)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:648)
    at clojure.core$require.invokeStatic(core.clj:5796)
    at clojure.core$require.doInvoke(core.clj:5796)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at immutant.internal.util$require_resolve.invokeStatic(util.clj:58)
    at immutant.internal.util$require_resolve.invoke(util.clj:55)
    at immutant.wildfly$init_deployment.invokeStatic(wildfly.clj:90)
    at immutant.wildfly$init_deployment.invoke(wildfly.clj:83)
    at clojure.core$eval70.invokeStatic(NO_SOURCE_FILE:0)
    at clojure.core$eval70.invoke(NO_SOURCE_FILE)
    at clojure.lang.Compiler.eval(Compiler.java:6927)
    at clojure.lang.Compiler.eval(Compiler.java:6917)
    at clojure.lang.Compiler.eval(Compiler.java:6917)
    at clojure.lang.Compiler.eval(Compiler.java:6890)
    at clojure.core$eval.invokeStatic(core.clj:3105)
    at clojure.core$eval.invoke(core.clj:3101)
    at clojure.lang.Var.invoke(Var.java:379)
    at org.projectodd.wunderboss.c

[danielcompton]

Can you upgrade the version of Netty that Wildly is using?

[ckirkendall]

Daniel, No, I don't have control over that environment. Creighton

…

On Tue, Feb 7, 2017 at 2:31 PM, Daniel Compton ***@***.***> wrote: Can you upgrade the version of Netty that Wildly is using? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIlY_pMbS3_I5B4BHhZKUfJUu31KrBAks5raMaPgaJpZM4L52wv> .

[danielcompton]

Can you share as much of your dependency tree as possible? It's a bit hard to debug further without more details on the Netty versions. Also, is it possible that there are two Netty JARs on the classpath under different artifact ID's (e.g. netty-handler and netty-all)? Looking around at this error, it seems like that sometimes is the cause (though I can imagine other reasons for the problem too).

[ckirkendall]

The issue come from Jboss and wildfly providing Netty and a specific version of netty. The only netty files are coming in from the rethinkdb.clj deps. The version of Widfly that I am trying to deploy to has 4.0.15-Final being provided. This normally wouldn't be an issue but aleph is highly coupled with specific versions of netty and rethinkdb is highly coupled with current release of aleph due to ssl support. Note: widlfly and jboss both use Netty as base module.

deps tree root for rethinkdb:

[com.apa512/rethinkdb "0.15.24" :exclusions [[cljsjs/react]]]
   [aleph "0.4.1"]
     [byte-streams "0.2.2"]
       [clj-tuple "0.2.2"]
       [primitive-math "0.1.5"]
     [io.aleph/dirigiste "0.1.3"]
     [io.netty/netty-all "4.1.0.CR3"]
     [manifold "0.1.4"]
       [riddley "0.1.12"]
     [potemkin "0.4.3"]
   [com.google.protobuf/protobuf-java "3.0.0-alpha-3.1"]
   [gloss "0.2.5"]
   [org.clojure/tools.logging "0.3.1"]
   [rethinkdb-protobuf "2.2.0.1"]

[ckirkendall]

@danielcompton - Is there a reason you don't just wrap the java api? The RQL api seems like it matches your setup very closely. It feels like you could support almost everything you have with very little code around the official java api.

[danielcompton]

The last time I looked, the Java driver didn't have any async support which means each changefeed would tie up a whole thread (I think). It's not uncommon to have thousands of changefeeds open, so this is a problem. I'd love to switch to using the Java driver, and it would make a lot of things a lot easier, but we need some sort of async support.

Have you tried excluding io.netty/netty-all from the rethinkdb dependency?

I don't have time to look into switching to the Java driver at the moment, or adding async support to the official Java driver, but if you wanted to provide a patch, I'd be happy to review it.

Alternatively, if you're able to provide a patch to clj-rethinkdb which make the problematic netty code path optional, I'd be happy to remove that too. Are you trying to use SSL in your cluster?

[ckirkendall]

@danielcompton - thanks for the reply. That is a bummer. It would simplify so many things. I notice there is suggestion to use Netty. Probably should link this issue there also. I don't know how to reference this on that thread. rethinkdb/rethinkdb#4802

[danielcompton]

Totally agree, it's a bit of a bummer. Now that there's a bit more certainty around RethinkDB, people (including me) may be able to invest more time into the Java driver.

If you can find a way to exclude the rethinkdb.ssl namespace unless it is needed, that could be a good option, but I'm not sure how (without doing dirty hacks).

[ckirkendall]

@danielcompton - I wrote a version of query that wraps the java driver to get around the issue. I did my best to make it a drop in replacement of query that way when async comes to the java api it can be added without issue. Do you think it would be good to open up a light version like this?

[danielcompton]

Not quite sure what you mean by open up, but I'd be happy to see it as a PR, or as a separate project?