Run processCcd for all DR1 stacks

[wmwv]

Presently only 150 of 700 stacks successfully make it through processCcd.py. Get this up to at least 80%.

1. Debug astrometric matching
2. Debug reference catalogs
3. Identify additional failure modes.

[wmwv]

I processed all of the DR1 images on lsst-dev01 last night:

710 images

[wmwv@lsst-dev01 wiyn_lsst]$ sh success_processCcd.sh processCcd.log
Succeeded:   495
Failed   :   215
----
  Aperture Correction:     80
  Photometric no matches:  54
  No viable PSF candidates:       55
  No passed PSF candidates:       23

212 failure modes accounted for. That leaves 3.

[wmwv]

The 3 errors were LengthError. Each was "Too many CR pixels (max 1000000)".

processCcd INFO: Processing {'field': 'LSQ12gef', 'seq': 'A', 'filter': 'J', 'night': 20121122, 'expnum': 259}
processCcd.isr INFO: Fetch stack image on sensor {'field': 'LSQ12gef', 'seq': 'A', 'filter': 'J', 'night': 20121122, 'expnum': 259}
processCcd.charImage.repair INFO: Identified 0 cosmic rays.
processCcd.charImage.detection INFO: Detected 80 positive peaks in 23 footprints and 2 negative peaks in 1 footprints to 50 sigma
processCcd.charImage.detection INFO: Resubtracting the background after object detection
processCcd.charImage INFO: Processing {'field': 'LSQ12gef', 'seq': 'A', 'filter': 'J', 'night': 20121122, 'expnum': 259}
processCcd.charImage.measurement INFO: Measuring 23 sources (23 parents, 0 children)
processCcd.charImage.measurePsf INFO: Measuring PSF
processCcd.charImage.measurePsf.reserve INFO: Reserved 0/5 sources
processCcd.charImage.measurePsf INFO: PSF star selector found 5 candidates
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd.charImage.measurePsf.psfDeterminer WARN: You only have 3 eigen images (you asked for 4): reducing number of eigen components
processCcd.charImage.measurePsf INFO: PSF determination using 3/5 stars.
processCcd.charImage INFO: iter 1; PSF sigma=2.74, dimensions=(35, 35); median background=-0.10
processCcd.charImage WARN: Source catalog detected and measured with placeholder or default PSF
processCcd FATAL: Failed on dataId={'field': 'LSQ12gef', 'seq': 'A', 'filter': 'J', 'night': 20121122, 'expnum': 259}: LengthError:
  File "src/CR.cc", line 435, in std::vector<std::shared_ptr<lsst::afw::detection::Footprint> > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::pex::policy::Policy&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage<float>]
    Too many CR pixels (max 1000000) {0}
lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 1000000)'

Traceback (most recent call last):
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/cmdLineTask.py", line 413, in __call__
    result = task.run(dataRef, **kwargs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/processCcd.py", line 189, in run
    doUnpersist=False,
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/characterizeImage.py", line 351, in run
    background = background,
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/characterizeImage.py", line 409, in characterize
    background = background,
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/characterizeImage.py", line 475, in detectMeasureAndEstimatePsf
    self.repair.run(exposure=exposure, keepCRs=True)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-4-g654b129+5/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/repair.py", line 213, in run
    self.cosmicRay(exposure, keepCRs=keepCRs)
  File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/15.0-5-g389937dc+5/python/lsst/pipe/tasks/repair.py", line 262, in cosmicRay
    pexConfig.makePolicy(self.config.cosmicray), keepCRs)
lsst.pex.exceptions.wrappers.LengthError:
  File "src/CR.cc", line 435, in std::vector<std::shared_ptr<lsst::afw::detection::Footprint> > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::pex::policy::Policy&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage<float>]
    Too many CR pixels (max 1000000) {0}
lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 1000000)'

[wmwv]

The P525-E image looks fine, but it only has three stars. This is of course true for all of the P525-E images, so I'm not sure why this one fails in this particular manner. 35 P525-E reductions fail, 40 succeed. The ones that fail are due to not finding suitable PSF stars or aperture correction stars.

[wmwv]

The SN2012fk reasonable seems like it should fail with that bright star, but the process of 6 other images of this field succeeded. The location of the bright star was further form the well-covered center of the stack.

[wmwv]

LSQ12gef_A_J_20121122_259 looks perfectly fine. I have no real idea what could have gone wrong.