"Seeking Help: Using ROMS HIS Files with Parcels Package"

[pphu5]

Question

Question

Hello everyone,
I'm currently working on a project using the ROMS model to simulate a 60-day scenario. I have a total of 60 his files (with dimensions 60,20,300,500), where each his file represents a 24-hour period (24,20,300,500) and the data is recorded at an hourly interval. Additionally, I have one avg file that averages the data over a day.
When I use the Parcel package to calculate, everything works fine with the avg file as the flow field. However, I encountered a problem when I attempted to use the flow velocities from the his files to create the flow field. Specifically, when I tried to use all the his files together, the operation failed and an error message related to dimension issues was displayed. I suspect that the problem lies in the way Fieldset.from_nemo reads the files.
Initially, I thought of reading the U or V components from each his file separately and then merging them into a single fieldset based on the time dimension. Unfortunately, this approach also failed as it seems that multiple fieldsets cannot be combined in this manner.
I'm wondering if anyone has encountered a similar issue or has any suggestions on how to resolve this. I would be extremely grateful for any help.
Here is a part of the code for reference:

Thank you in advance for your time and assistance.
Best regards,

Supporting code/error messages

# Paste your code within this block

Successful run:
ufiles = sorted(glob('/public/home/work/model/case5/out/full_avg_result.nc'))
vfiles = ufiles
tfiles = ufiles
...
fieldset = Fieldset.from_nemo(filenames, variables, dimensions, indices, allow_time_extrapolation=True)


Failed run:
ufiles = sorted(glob('/public/home/work/model/case5/out/full_his_*.nc'))
vfiles = ufiles
tfiles = ufiles
...
fieldset = Fieldset.from_nemo(filenames, variables, dimensions, indices, allow_time_extrapolation=True)

# Paste your error message within this block
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (60,) + inhomogeneous part.

[erikvansebille]

Thanks for reaching out @pphu5, good to hear that you can run with the avg files. But I can see why you would also be able to run with the his files. I haven't come across these his files myself, so can't provide any guidance. Perhaps some of the other Parcels/ROMS/CROSO users have? @andrew-s28? @nvogtvincent?

[pphu5]

Thank you so much for your help!

[andrew-s28]

Hey @pphu5, I have run into similar issues when using multiple input files from CROCO model fields.

Is there any chance that one of your full_his_*.nc files has different dimensions from the rest? For example, does running xr.concat(sorted(glob('/public/home/work/model/case5/out/full_his_*.nc'))) give you what you expect? How does it compare to xr.concat(sorted(glob('/public/home/work/model/case5/out/full_his_*.nc'))[1:]) (without the first file)?

This would be the first thing I would check. Depending on the model configs, the first file may be output with additional diagnostic coordinates and dimensions that make concatenating the files require a bit of preprocessing (e.g., removing unnecessary dimensions in files) before running with Parcels.

[pphu5]

Thank you very much for your response. We have attempted to run the following code: xr.concat(sorted(glob('/public/home/work/model/case5/out/full_his_*.nc'))[1:]) (ignoring the first "his" file), but we are still encountering errors. The issue lies in the fact that glob returns a list of file path strings, not datasets. The xr.concat() function requires xarray.Dataset or xarray.DataArray objects, not file paths.

From the documentation of FieldSet.from_nemo, it appears that the filenames parameter is supposed to support a list of files. However, based on my debugging experience, it seems that this function can only read a list of his files with a single time dimension, where each his file contains U (or V) data for one specific time step. I’m not entirely sure if this understanding is correct.

Similar to "avg" files, it would be feasible if all the U (or V) data from the his files could be stored together in a single file. Therefore, another approach we considered was to use xr.open_dataset to read all the his files, then concatenate them using xr.concat along the 'ocean_time' dimension, and finally save the result as a single .nc file. We could then use FieldSet.from_nemo to construct the flow field from this combined file. While this method is feasible, it results in a very large file size and a lengthy concatenation runtime.

Given these challenges, I would like to ask if there are any alternative ways to construct a flow field from "his" files that contain multiple time dimensions. Any suggestions would be greatly appreciated!

[JamiePringle]

Are you successful in opening them with xr.open_dataset()? I suspect you will not be, because of some internal inconsistency in the data files. That is what @andrew-s28 was trying to get you to do. Jamie

[andrew-s28]

That is indeed what I had intended, @JamiePringle (or xr.open_mfdataset()).

Sorry for the confusion, @pphu5. Either way, I suspect from the initial error that this has something to do with the model files you are using and not necessarily with Parcels, which makes it hard to say for sure what the issue is without seeing the files myself.