Densification Error

[svenborgstrom]

Hi Andreas,
at step 4 (sarvey -f config.json 4 4) I got the error shown in the attached file (SARVEY_Densification_Error).
For completeness I include also the log file and the "config.json" file, as I suppose I have to set differently this in the "densification" section.
Plese give also a look at this file if you have suggestions to set it differently, not only for densification, as I'm quite new to PS processing.
As you can see, I used the default densification parameters.
With regard to my AOI:

• currently strong deformation (about 18 cm/year at the center of the coastline close to the sea);
• a lot of water around but I filtered the sea with an appropriate "waterMask.h5" file (the waterMask.rdr file from MintPy processing converted with "generate_mask.py waterMask.rdr --max 0.5 -o waterMask.h5 --revert".
  Let me know if you need more info from my side.
  Best,
  Sven.

P.S.: I filled out the questionnaire :)
config.json
sarvey_2025-06-16-13-09-50.log

[svenborgstrom]

Hi Andreas,
any news on my question?
Best,
Sven.

[Andreas-Piter]

Dear Sven,
thanks for your patience and for the detailed description of the issue. Thanks a lot for participating in the user poll!
To me this error message looks like there could be no second-order points selected during the densification step. Could you please check the figure selected_pixels_temp_coh_0.8.png and step_3_mask_p2_coh80.png? Does it contain pixels within your mask? Could you please post the figures here?
Best, Andreas

[svenborgstrom]

Hi Andreas,
thanks for answering.
In the attached files you'll find the figures you asked me (selected_pixels_temp_coh_0.8.png & step_3_mask_p2_coh80.png), besides the waterMask.png (from "view.py waterMask.h5"), I used for masking water.
I got this from the waterMask.rdr file in MintPy processing, converted with "generate_mask.py waterMask.rdr --max 0.5 -o waterMask.h5 --revert": I don't know if the error arises from this, b.t.w I tried to exclude completely the contribution from water which could result afterwards in an issue.
Let me know if you need more info from my side.
Best,
Sven.

[Andreas-Piter]

Dear Sven,
thanks for sharing the figures. It looks all good. Also the water mask seems fine.
I have two ideas what the problem could be. Please try the following:

1. It might be a bug in the splitting of the dataset for parallel processing. Please change the number of cores to any other value in the configuration file and simply run again.
2. Alternatively, it could be related to the integer overflow recently reported by @esrivas17 in Issue with splitDatasetForParallelProcessing for large data #104. Please try a higher coherence_p2 threshold to have less points in the final dataset.

Best, Andreas

[Andreas-Piter]

Hi @esrivas17,
thanks for checking it out. Could you please point to the branch where the changes are applied? Alternatively, @svenborgstrom, you could apply the proposed change (#104) in your files locally to make it work and we will fix the bug in the next version.
Best, Andreas

[svenborgstrom]

Hi Andreas and Erik,
I'm a bit confused now and afraid in some way to damage my installation.
Erik told me about pulling changes: is there the possibility to run somewhere a "git pull" command to get the updates?
Best,
Sven.

[Andreas-Piter]

Hi Sven,
I do understand your concern. Let us first make sure what the error source is. Please try the two suggestions I posted above and afterwards we will see how we can fix it together.
Best, Andreas

[esrivas17]

Hi @svenborgstrom , sorry for the confusion, it seems that the changes proposed in #104 have not been merged yet.
But you can always add and try these changes yourself without messing your original installation.

You can create another branch in your sarvey repository and edit utils.py
on line 532:

sarvey/sarvey/utils.py

Line 532 in 7abc29a

num_samples_per_core = np.zeros((num_cores,), dtype=np.int16)

Here, you can try to change np.int16 to np.uint32 or np.uint64, and give it a go again to your step 4. It hopefully solves the issue.

You can always go back to the original source code with git checkout main

Erik

[svenborgstrom]

Hi Andreas and Erik,
sorry for late reply but I was quite busy in these day.
At the end I was able to conclude successfully my processing following you suggestions.
I changed the number of cores from 10 to 12, then I changed line 532 in sarvey/utils.py from dtype=np.int16 into np.uint.32 (I had a problem with np.uint.64).
Best,
Sven.