Devicehive server crash after runing a long time #540
Description
I am running a devicehive server 3.5.0 on CentOS 7. and I am getting the following error after runing a long time and have to restart the server. Could you please help to have a look and advice for possible cause and solution? Thank you,
2019-08-05 09:49:34.148 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/info ]
2019-08-05 09:49:34.177 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/info ]
2019-08-05 09:49:34.180 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/info ]
2019-08-05 09:49:34.204 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/info ]
2019-08-05 09:49:34.214 [XNIO-2 task-5] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/token ]
2019-08-05 09:49:34.231 [XNIO-2 task-5] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/token ]
2019-08-05 09:49:34.233 [XNIO-2 task-6] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ POST /dh/rest/token ]
2019-08-05 09:49:34.243 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL -
select
user0_.id as id1_3_,
user0_.all_device_types_available as all_devi2_3_,
user0_.data as data3_3_,
user0_.intro_reviewed as intro_re4_3_,
user0_.last_login as last_log5_3_,
user0_.login as login6_3_,
user0_.login_attempts as login_at7_3_,
user0_.password_hash as password8_3_,
user0_.password_salt as password9_3_,
user0_.role as role10_3_,
user0_.status as status11_3_
from
"dh_user" user0_
where
user0_.login=?
and user0_.status<>3
2019-08-05 09:49:34.247 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL -
select
configurat0_.name as name1_0_,
configurat0_.entity_version as entity_v2_0_,
configurat0_.value as value3_0_
from
configuration configurat0_
where
configurat0_.name=?
2019-08-05 09:49:34.249 [XNIO-2 task-6] DEBUG anonymousPrincipal org.hibernate.SQL -
update
"dh_user"
set
all_device_types_available=?,
data=?,
intro_reviewed=?,
last_login=?,
login=?,
login_attempts=?,
password_hash=?,
password_salt=?,
role=?,
status=?
where
id=?
2019-08-05 09:49:34.272 [XNIO-2 task-6] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ POST /dh/rest/token ]
2019-08-05 09:49:34.280 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/token/create ]
2019-08-05 09:49:34.285 [XNIO-2 task-7] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/token/create ]
2019-08-05 09:49:34.287 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ POST /dh/rest/token/create ]
2019-08-05 09:49:34.332 [XNIO-2 task-8] DEBUG org.hibernate.SQL -
select
user0_.id as id1_3_0_,
user0_.all_device_types_available as all_devi2_3_0_,
user0_.data as data3_3_0_,
user0_.intro_reviewed as intro_re4_3_0_,
user0_.last_login as last_log5_3_0_,
user0_.login as login6_3_0_,
user0_.login_attempts as login_at7_3_0_,
user0_.password_hash as password8_3_0_,
user0_.password_salt as password9_3_0_,
user0_.role as role10_3_0_,
user0_.status as status11_3_0_
from
"dh_user" user0_
where
user0_.id=?
2019-08-05 09:49:34.343 [XNIO-2 task-8] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission MANAGE_TOKEN
2019-08-05 09:49:34.383 [XNIO-2 task-8] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ POST /dh/rest/token/create ]
2019-08-05 09:49:35.651 [XNIO-2 task-9] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/user/current ]
2019-08-05 09:49:35.655 [XNIO-2 task-9] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/user/current ]
2019-08-05 09:49:35.657 [XNIO-2 task-10] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/user/current ]
2019-08-05 09:49:35.661 [XNIO-2 task-10] DEBUG org.hibernate.SQL -
select
user0_.id as id1_3_0_,
user0_.all_device_types_available as all_devi2_3_0_,
user0_.data as data3_3_0_,
user0_.intro_reviewed as intro_re4_3_0_,
user0_.last_login as last_log5_3_0_,
user0_.login as login6_3_0_,
user0_.login_attempts as login_at7_3_0_,
user0_.password_hash as password8_3_0_,
user0_.password_salt as password9_3_0_,
user0_.role as role10_3_0_,
user0_.status as status11_3_0_
from
"dh_user" user0_
where
user0_.id=?
2019-08-05 09:49:35.683 [XNIO-2 task-10] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission GET_CURRENT_USER
2019-08-05 09:49:35.693 [XNIO-2 task-10] DEBUG admin org.hibernate.SQL -
select
user0_.id as id1_3_0_,
network2_.id as id1_4_1_,
user0_.all_device_types_available as all_devi2_3_0_,
user0_.data as data3_3_0_,
user0_.intro_reviewed as intro_re4_3_0_,
user0_.last_login as last_log5_3_0_,
user0_.login as login6_3_0_,
user0_.login_attempts as login_at7_3_0_,
user0_.password_hash as password8_3_0_,
user0_.password_salt as password9_3_0_,
user0_.role as role10_3_0_,
user0_.status as status11_3_0_,
network2_.description as descript2_4_1_,
network2_.entity_version as entity_v3_4_1_,
network2_.name as name4_4_1_,
networks1_.user_id as user_id2_7_0__,
networks1_.network_id as network_1_7_0__
from
"dh_user" user0_
left outer join
user_network networks1_
on user0_.id=networks1_.user_id
left outer join
network network2_
on networks1_.network_id=network2_.id
where
user0_.id=?
2019-08-05 09:49:35.712 [XNIO-2 task-10] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/user/current ]
2019-08-05 09:49:35.750 [XNIO-2 task-11] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/network ]
2019-08-05 09:49:35.754 [XNIO-2 task-11] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/network ]
2019-08-05 09:49:35.756 [XNIO-2 task-12] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/network ]
2019-08-05 09:49:35.792 [XNIO-2 task-12] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission GET_NETWORK
2019-08-05 09:49:35.806 [XNIO-2 task-12] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/network ]
2019-08-05 09:49:35.813 [pool-1-thread-2] ERROR c.l.disruptor.FatalExceptionHandler - Exception processing: 11 com.devicehive.model.ServerEvent@22db5968
java.lang.NullPointerException: null
at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[disruptor-3.3.6.jar!/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Exception in thread "pool-1-thread-2" java.lang.RuntimeException: java.lang.NullPointerException
at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:165)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83)
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80)
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
... 3 more
2019-08-05 09:49:37.346 [XNIO-2 task-13] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ OPTIONS /dh/rest/user ]
2019-08-05 09:49:37.350 [XNIO-2 task-13] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ OPTIONS /dh/rest/user ]
2019-08-05 09:49:37.352 [XNIO-2 task-14] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Bound request context to thread: HttpServletRequestImpl [ GET /dh/rest/user ]
2019-08-05 09:49:37.362 [XNIO-2 task-14] INFO admin c.d.auth.JwtPermissionEvaluator - Successfully checked for permission MANAGE_USER
Exception in thread "pool-1-thread-1" java.lang.RuntimeException: java.lang.NullPointerException2019-08-05 09:49:37.367 [pool-1-thread-1] ERROR c.l.disruptor.FatalExceptionHandler - Exception processing: 12
com.devicehive.model.ServerEvent@1d0f3cfc
java.lang.NullPointerException: null
at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39) ~[devicehive-proxy-ws-kafka-impl-3.5.0.jar!/:3.5.0]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[disruptor-3.3.6.jar!/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:165)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at com.devicehive.proxy.client.WebSocketKafkaProxyClient.push(WebSocketKafkaProxyClient.java:83)
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:80)
at com.devicehive.proxy.ProxyResponseHandler.onEvent(ProxyResponseHandler.java:39)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
... 3 more
2019-08-05 09:49:37.369 [XNIO-2 task-14] DEBUG o.s.b.w.f.OrderedRequestContextFilter - Cleared thread-bound request context: HttpServletRequestImpl [ GET /dh/rest/user ]