FMRC logfile errors: fmrc:SABGOM_ROMS_Forecast_Model_Run_Collection

[jbzambon]

FMRC is throwing errors referencing some file "fmrc:SABGOM_ROMS_Forecast_Model_Run_Collection" instead of pulling variables listed in the catalog (sabgom_ROMS_nf_catalog.xml) and getting errors when it can't find some variables (e.g. AKt)...

[2016-08-31T19:33:16.585+0000] ERROR ucar.nc2.ft.fmrc.FmrcDataset: buildDataset2D: cant find ugrid variable AKt in collection fmrc:SABGOM_ROMS_Forecast_Model_Run_Collection
result dataset fmrc:SABGOM_ROMS_Forecast_Model_Run_Collection
ntimes
ndtfast
dt
dtfast
dstart
....

[rsignell-usgs]

@jbzambon, you are running thredds on oceanus "the old way", not using docker, right?
I see one potential problem is that there is a mixture of files owned by root and tomcat under:
/usr/local/tds/apache-tomcat-8.0.32
which is usually caused by someone running tomcat as root by mistake.

When I just logged in and checked, tomcat is running under user tomcat as it should be, but perhaps it's having trouble reading or writing certain files. Can you please shutdown, change all files under tomcat to be owned by user tomcat and restart?
Should be:

sudo -u tomcat /usr/local/tds/apache-tomcat-8.0.32/bin/shutdown.sh
sudo chown -R tomcat:tomcat /usr/local/tds/apache-tomcat-8.0.32
sudo -u tomcat /usr/local/tds/apache-tomcat-8.0.32/bin/startup.sh

[jbzambon]

@rsignell-usgs Thanks! I did this and restarted it.

On a similar topic, we've been having trouble with our tomcat server leaking memory through the java server. A currently top reveals >10GB of memory usage. We have a similar issue with omgsrv1 and I usually will just restart when the usage gets too obscene. Any hints on solving this besides with a restart? Thanks!

3113 tomcat 20 0 10.134g 1.603g 20132 S 0.0 5.1 0:58.53 java

[kwilcox]

THREDDS does leak memory and there is no way around it. We run over 10 thredds servers seeing high volume and restart them every 2 days.

[rsignell-usgs]

@lesserwhirls, do you guys restart http://thredds.ucar.edu/thredds/catalog.html regularly (e.g. every night or every few days?)

[rsignell-usgs]

BTW, @cwardgar bet me $5 you don't. 😺

[lesserwhirls]

@rsignell-usgs, we do not do regular restarts of our TDS, so I guess you need to pay up 💵? @kwilcox we do not see memory leaks on our systems, but we do not have any FMRCs, so that might be why. Do you have any more info on the leaks?

[kwilcox]

Do you host any NetCDF aggregations on http://thredds.ucar.edu/thredds/catalog.html? From what I gather that has always been the issue. In the ocean world most of our aggregations are NetCDF3/4, not GRIB.

[lesserwhirls]

We do not host any netCDF aggregations. I admit it's shameful that our support for GRIB is better than our support for netCDF3/4 👎

We have been talking quite a bit about creating a grid feature collection, much like the grib feature collections we have now (grid feature collections would replace FMRC under the hood, but would be more generic than that). This would help in the area of aggregating netCDF files, but I suspect there would still be a need for using NcML aggregations. We are getting the new GOES-16 data in real-time, which are netCDF-4 files. I'll set up a test aggregation collection on our test server and keep an eye on the memory usage to see what is going on. Are you using just simple time aggregations?

[rsignell-usgs]

Fascinating. @lesserwhirls, I can't think of anything else that would be more useful for the IOOS community than to have NetCDF aggregations (both joinExisting and FMRC-type) working efficiently and without leaking memory.

And I'll see @cwardgar in a few short weeks to pay up. 😺