Skip to content

"GC overhead limit exceeded" from v7.6.2 #245

Open
@miura

Description

@miura

Hi JY @tinevez

I encountered the following error starting from the v7.6.2-snapshot, while the v7.6.1-snapshot or earlier versions do not show this memory handling problem.

Exception in thread "main" java.lang.IllegalArgumentException: Invalid service: net.imagej.animation.DefaultAnimationService
        at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:278)
        at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:229)
        at org.scijava.service.ServiceHelper.loadService(ServiceHelper.java:192)
        at org.scijava.service.ServiceHelper.loadServices(ServiceHelper.java:164)
        at org.scijava.Context.<init>(Context.java:285)
        at org.scijava.Context.<init>(Context.java:234)
        at org.scijava.Context.<init>(Context.java:130)
        at org.scijava.Context.<init>(Context.java:117)
        at org.scijava.Context.<init>(Context.java:106)
        at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at ij.IJ.runUserPlugIn(IJ.java:239)
        at ij.IJ.runPlugIn(IJ.java:204)
        at ij.IJ.runPlugIn(IJ.java:193)
        at fiji.plugin.trackmate.util.TMUtils.getContext(TMUtils.java:714)
        at fiji.plugin.trackmate.detection.DetectionUtils.findLocalMaxima(DetectionUtils.java:355)
        at fiji.plugin.trackmate.detection.DogDetector.process(DogDetector.java:127)

The above error is just an example and can be different as memory exhaustion causes this:

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:141)
        at org.bushe.swing.event.ThreadSafeEventService.subscribe(ThreadSafeEventService.java:346)
        at org.scijava.event.DefaultEventService.subscribe(DefaultEventService.java:184)
        at org.scijava.event.DefaultEventService.subscribe(DefaultEventService.java:203)
        at org.scijava.event.DefaultEventService.subscribe(DefaultEventService.java:137)
        at org.scijava.service.Service.registerEventHandlers(Service.java:66)
        at org.scijava.service.ServiceHelper.createServiceRecursively(ServiceHelper.java:353)
        at org.scijava.service.ServiceHelper.createExactService(ServiceHelper.java:267)

To reproduce this error, please try the following code with two different versions. I start to see the slowing down around 500th iteration, and often stops by 750th iteration. This is not affected by the VM setting such as -xmx.

I will add a download link to the sample image (hardcoded in the main ) below the code.

import java.util.List;

import fiji.plugin.trackmate.Spot;
import fiji.plugin.trackmate.detection.DogDetector;
import ij.IJ;
import ij.ImagePlus;
import net.imglib2.img.Img;
import net.imglib2.img.display.imagej.ImageJFunctions;
import net.imglib2.type.numeric.integer.UnsignedByteType;

public class TrackMateTest20230224 {

    void testMemoryDogDetector(ImagePlus pmimp, int radius,
            int threshold, boolean doSubpixel, boolean doMedian) {

        // Set the parameters for LogDetector
        Img<UnsignedByteType> img = ImageJFunctions.wrap(pmimp);
        Img<UnsignedByteType> interval = img;
        double[] calibration = { 1, 1, 1 };
        DogDetector<UnsignedByteType> detector = new DogDetector<UnsignedByteType>(
                img, interval, calibration, radius, threshold,
                doSubpixel, doMedian);
        // detector.setNumThreads(1);
        detector.process();
        List<Spot> peaks = detector.getResult();
        if (peaks.size() > 0) {
            IJ.log("=== Detected FOCI coordinates ===");
            for (Spot peak : peaks) {
                double xm = peak.getDoublePosition(0);
                double ym = peak.getDoublePosition(1);
                IJ.log("... " + Double.toString(xm) + ", "
                        + Double.toString(ym));
            }
        }
    }

    public void spot2DTestMinimal(String pp) {

        ImagePlus ccimpG = new ImagePlus(pp);
        for (int i = 0; i < 1000; i++) {
            IJ.log("=== count:" + Integer.toString(i));
            int radius = 3;
            int threshold = 5;
            boolean doSubpixel = false;
            boolean doMedian = true;
            testMemoryDogDetector(ccimpG, radius, threshold,
                    doSubpixel, doMedian);

        }
    }
    public static void main(String[] args) throws Exception {
        TrackMateTest20230224 tmt = new TrackMateTest20230224();
        String pp = "/Users/miura/samples/test5dots.tif";
        tmt.spot2DTestMinimal( pp );
    }
}

The sample image:
https://www.dropbox.com/s/deu2vc6ftkp9wle/test5dots.tif?dl=0

Just as additional information: when similar processing is done using Jython, there is no problem. Interesting...

from ij import IJ, ImagePlus
from fiji.plugin.trackmate.detection import LogDetector, DogDetector
from net.imglib2.img.display.imagej import ImageJFunctions

def spotDetectionTrackMate(pmimp, Param_Dot):
	# Set the parameters for LogDetector
	img = ImageJFunctions.wrap( pmimp )
	interval = img
	calibration = [ 1, 1, 1 ]
	radius = Param_Dot[0]  			
	threshold = Param_Dot[1] 
	doSubpixel = Param_Dot[2]
	doMedian = Param_Dot[3] 
		
	# Spot Detection
	#detector = LogDetector(img, interval, calibration, radius, threshold, doSubpixel, doMedian)
	detector = DogDetector(img, interval, calibration, radius, threshold, doSubpixel, doMedian) 
	detector.process()
	peaks = detector.getResult()
	print peaks
	print "=== Detected FOCI coordinates ==="
	for peak in peaks:
		xm = peak.getDoublePosition(0)
		ym = peak.getDoublePosition(1)
		print "x: ", xm, "y: ", ym #, "z: ", zm

pp = "/Users/miura/samples/test5dots.tif"
imp = ImagePlus(pp)
Param_DotG = [3, 5, False, True]
for i in range(1000):
	print "=== count ", str(i)
	spotDetectionTrackMate(imp, Param_DotG)

I hope this can be somehow fixed, as this is causing crashes when many spots are analyzed. Until this is fixed, I will use v7.6.1 to avoid the problem.

... and last but not least, big thanks to your masterpiece plugin!
Cheers,
Kota

ps: following versions were tested.

7.1.1 OK
7.5.1 OK
7.6.1 OK
7.6.2 failed
7.7.1 failed
7.10.2 failed
7.10.3 failed

Metadata

Metadata

Assignees

No one assigned

    Labels

    investigateAssignee should verify the issue and try to reproduce it.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions