Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OutOfMemoryError #9

Open
brentp opened this issue Oct 10, 2018 · 2 comments
Open

OutOfMemoryError #9

brentp opened this issue Oct 10, 2018 · 2 comments

Comments

@brentp
Copy link

brentp commented Oct 10, 2018

hi, I was able to use bazam on a BAM file to realign and it was chugging along smoothly but it eventually errored with the message below.
I ran as:

java -Xmx4G -jar bazam.jar -L regions.bed -bam $bam -n 3 \
   | bwa mem ...

where regions.bed is:

1	0	249250621
20	0	63025520
21	0	48129895
22	0	51304566

the final status message was:

gngs.pair.PairScanner   [1]     INFO    |2:41:56 Wed Oct 10 14:41:56 MDT 2018 :  Processed 650989000 @ 18663.56/s 21:10917077, loc: 651m,71.1m,240.5k chimeric: 803.2k formatted: 71.1m, written: 68.3m

traceback:

An exception occurred in the Actor thread Actor Thread 132
An exception occurred in the Actor thread Actor Thread 128

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Actor Thread 132"

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.HashSet.<init>(HashSet.java:144)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:216)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:214)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:36)
        at org.codehaus.groovy.reflection.CachedClass.getInterfaces(CachedClass.java:266)
        at org.codehaus.groovy.reflection.CachedClass.<init>(CachedClass.java:252)
        at org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:369)
        at org.codehaus.groovy.reflection.ClassInfo.access$300(ClassInfo.java:46)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:423)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:414)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
        at org.codehaus.groovy.reflection.ClassInfo.getCachedClass(ClassInfo.java:135)
        at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:110)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:223)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:214)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:36)
        at org.codehaus.groovy.reflection.CachedClass.getInterfaces(CachedClass.java:266)
        at org.codehaus.groovy.reflection.CachedClass.<init>(CachedClass.java:252)
        at org.codehaus.groovy.reflection.stdclasses.CachedClosureClass.<init>(CachedClosureClass.java:30)
        at org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:365)
        at org.codehaus.groovy.reflection.ClassInfo.access$300(ClassInfo.java:46)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:423)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:414)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
   at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
        at org.codehaus.groovy.reflection.ClassInfo.getCachedClass(ClassInfo.java:135)
        at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:110)
        at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:182)
        at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:218)
An exception occurred in the Actor thread Actor Thread 129
java.lang.OutOfMemoryError: GC overhead limit exceeded
An exception occurred in the Actor thread Actor Thread 126
java.lang.OutOfMemoryError: GC overhead limit exceeded

java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.NullObject
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:891)
        at groovyx.gpars.actor.Actor.sweepQueue(Actor.java:393)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:316)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:65)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.NullObject
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:891)
        at groovyx.gpars.actor.Actor.sweepQueue(Actor.java:393)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:316)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:65)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
An exception occurred in the Actor thread Actor Thread 125
java.io.IOException: Stream closed
        at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:45)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:118)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
        at java.io.Writer.write(Writer.java:157)
        at java.io.Writer.append(Writer.java:227)
        at gngs.pair.PairWriter$_act_closure1$_closure2.doCall(PairWriter.groovy:60)
        at gngs.pair.PairWriter$_act_closure1$_closure2.call(PairWriter.groovy)
        at groovyx.gpars.actor.DefaultActor.onMessage(DefaultActor.java:94)
        at groovyx.gpars.actor.DefaultActorClosure.call(DefaultActorClosure.java:38)
        at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
        at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
        at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
@ssadedin
Copy link
Owner

I think that 4G is probably not enough RAM (depending on the structure of your BAM file Bazam will cache a lot of reads in memory). So I'd suggest trying with

java -Xmx24g -jar bazam.jar ...

You might not need all of that 24g but it's a reasonably safe value to figure out what's right.

The alternative to use less memory per-instance is to apply sharding, but then you need to run multiple Bazams (either in series or parallel if you can do that).

@brentp
Copy link
Author

brentp commented Oct 11, 2018

I think this was a problem with using the regions. I was able to do the full bam with 4G but it errored on regions. Doesn't bazam write reads to disk to avoid high memory use?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brentp @ssadedin