Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] readOffload NoClassDefFoundError #23736

Open
2 of 3 tasks
KannarFr opened this issue Dec 16, 2024 · 2 comments · May be fixed by #24031
Open
2 of 3 tasks

[Bug] readOffload NoClassDefFoundError #23736

KannarFr opened this issue Dec 16, 2024 · 2 comments · May be fixed by #24031
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@KannarFr
Copy link
Contributor

Search before asking

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

4.0.1

Minimal reproduce step

It can't reproduce on demand, but when a broker has the issue, it always has it.

What did you expect to see?

Nothing

What did you see instead?

Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,810+0000 [offloader-OrderedScheduler-0-0] ERROR org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader - Failed readOffloaded:
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,811+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Unknown exception for ManagedLedgerException.
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        ... 9 more
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0] Error opening ledger for reading at position 162128641:0 - org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: 2024-12-16T11:17:18,812+0000 [BookKeeperClientWorker-OrderedExecutor-18-0] WARN  org.apache.bookkeeper.mledger.impl.OpReadEntry - [orga_00575eaa-d956-4b65-90bb-7f87706b3d5a/accesslogs/persistent/app_280d8925-a61e-49b4-8537-1d5748e545e0-partition-0][reader-9bd1511927] read failed from ledger at position:162128641:0
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: org.apache.bookkeeper.mledger.ManagedLedgerException: Other exception
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.NoClassDefFoundError: org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]: Caused by: java.lang.ClassNotFoundException: org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader.lambda$readOffloaded$4(BlobStoreManagedLedgerOffloader.java:558) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) ~[org.apache.bookkeeper-bookkeeper-common-4.17.1.jar:4.17.1]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
Dec 16 11:17:18 yo-pulsar-broker-c3-n8 pulsar[2705940]:        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Anything else?

Offloaders nar is present. When restarting the broker, the topic is assigned to another broker and it doesn't have the issue now. Also, this error is mixed in many exceptions related to #23735.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@KannarFr KannarFr added the type/bug The PR fixed a bug or issue reported a bug label Dec 16, 2024
@KannarFr
Copy link
Contributor Author

I still have the issue occasionally. Restarting the brokers fixes the problem, (the topic is assigned to another broker), but the fresh topics on the restarted broker don't have the issue. It looks like a race or GC issue.

@lhotari lhotari linked a pull request Feb 26, 2025 that will close this issue
Open
4 tasks
@lhotari
Copy link
Member

lhotari commented Feb 26, 2025

The JVM doesn't trigger class loading until a specific class is referenced. That's why it happens later. However, it should work.

#13589 and #13501 are other changes where a similar issue has been addressed. I have created #24031 as an attempt. Although it seems to me that the problem could be elsewhere, such as having the same root cause as #23294.

I have noticed that the nar files duplicate most of the Pulsar dependencies. There's an old issue about this, #22218.
I recently noticed that it also applies to the tiered storage nar files (discussion: apache/pulsar-helm-chart#571 (comment)). The dependency duplication could also cause classloading issues.
That's why #24031 might not be needed. There are multiply messy things to be sorted out. We need more contributors that would have the energy to clean up such things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[WIP][fix][offload] Fix classloading issue when using tiered storage lhotari/pulsar
2 participants
@lhotari @KannarFr