What libraries could and perhaps should be considered

[toqduj]

Please add thoughts below.

So far, we're considering:

• XArray (with Dask),
• attrs
• pint (only on ingest and final display)
• Numexpr for speed?
• PyTorch perhaps?
• Tiled

Uncertainties packages:

• uncertainties: linear, first-order propagation for quick estimates
• soerp: second-order for moderately accurate results
• mcerp: MC, LHC sampling for precise results at higher computational cost

Scipp was considered, but given the complexity of adapting it (core is written in C++) it may make life complicated very quickly.
Sciline... maybe?

[toqduj]

For caching functionality, we might need a caching library that can work across multiple threads..

[toqduj]

For Pint, these are good tips:

https://pint.readthedocs.io/en/stable/advanced/performance.html
and
https://pint.readthedocs.io/en/stable/advanced/wrapping.html#wrapping

[gudlot]

Maybe worth checking out:

PyMC is a probabilistic programming library for Python that allows users to build Bayesian models with a simple Python API and fit them using Markov chain Monte Carlo (MCMC) methods.

In this context:
PyTensor PyTensor is a Python library that allows you to define, optimize/rewrite, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.

PyZMQ (https://github.com/zeromq/pyzmq) for messaging for distributed systems.

[gudlot]

Question about xarray:

A HDF5/NeXus file can easily contain up to 300 datasets within a single file, encompassing a mix of integers, strings, floats, and arrays of varying dimensions (1D, 2D, and 3D). How can this data be effectively structured in xarray, ensuring metadata is retained? Additionally, what is the best approach to link and organise multiple HDF5 files within xarray to facilitate efficient access and analysis?

[toqduj]

Maybe worth checking out:

PyMC
PyTensor
PyZMQ

Hey. Thanks for the suggestions, I'll add them to the list (ZMQ is hopefully not necessary).

As for the XArray question and data access.. I've played around more with Dask itself, and I think we may not need XArray at all, which would simplify matters. XArray is powerful but unless there's a good reason, we don't need to include it. Dask can take care of sequencing and parallelisation of operations, and has good visualisation tools and backends for local and cluster-based operation.

The data access across streams, files and filesystems is fortunately a (somewhat) solved problem with Tiled. My preliminary experiments show that Tiled forms a reasonable option to abstract, access and load data, both locally and from larger repositories. Tiled can remain optional, however, in favour for more traditional approaches.

[gudlot]

Hi, thanks for the reply.

As there is nothing more detailed yet on the requirements, I will only mention that I came across TensorFlow Probability (TFP)(https://www.tensorflow.org/probability) for Bayesian neural networks. But there are probably also other options that will work for your idea.

Thanks for sharing your experience with Xarray. I have currently a similar project with structured data, but very heterogeneous data. I need to keep it together per file, but organise it over many more files as well. I will check out again dask, but maybe again Xarray.
I did not know tiled and chose the traditional way via h5py to handle loading etc.

[toqduj]

New considerations and requests are to initially keep this library as simple as possible, so for now we'll hold off on including Dask until there is a proven speed improvement

[gudlot]

Hi,
have you published your considerations and requirements in a central location where interested contributors can follow and align with them?
If open community engagement is a real goal, a dedicated document outlining the project's requirements, design principles, and roadmap might help make the project more accessible and inviting to external contributors.

[toqduj]

Hey,

That is in the plan, at the moment it is in a high state of flux so I am just keeping notes here for now. Once these things are crystallising, they will be committed to paper.

[toqduj]

scipy was added to support sparse matrix operations (and likely future operations as well) It's not overly large. uncertainties is added in testing to double-check.

[toqduj]

auto_uncertainties is a nice one to keep an eye on. small dev team, but performant for uncertainties propagation.