refactor suggestion: modular index, spi tags, softdep management by spi

[fkiraly]

Thanks for the great presentation today, @benfulcher!

Inspired, I looked in greater detail into the repository, to my shame perhaps for the first time at that level of detail.

What I understood is that your SPI are actually not all manually implemented, but there is a wealth of them, some using external dependencies in turn. As such, pyspi is, morally, very much similar to sktime, being a mix of de-novo implementations, direct interfaces to external algorithms, and implementations that use components with soft dependencies.

I also noticed that you have tags for the different SPI, which again is very similar to sktime.

Further, when trying to interface SPI individually, I noticed that this is currently not intended to be possible - only batch feature sets can be obtained? Which seems to be a shame, you have collected so many useful pairwise transformations! Unless of course you use the yaml, and the process of discovery if you want that is tedious, and currently cannot be automated, so composability with other frameworks is severely limited.

Based on this, I had a number of ideas if you would like to hear me out:

• move the SPIs to a strategy object orientation pattern, using the tag system provided by scikit-base. This would give you for free runtime discoverability - no need to use config yaml, or the webpage.
  • at runtime, it would look like all_estimators in sktime: https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.registry.all_estimators.html
  • on the webpage, you can use this as a backend for selection with tags: https://www.sktime.net/en/latest/estimator_overview.html
• change SPIs from private API to public API. Define a public API, and add a test suite for testing individual SPI for interface conformance.
• isolate SPI specific dependencies to the particular SPI. Then you can treat, I believe, all dependencies as soft dependencies.
  • users could then say: compute all SPI for which I have all dependencies installed. Or, get me all SPI for this dependency set. SPIs would also tell the user directly which dependency to install.
  • pyproject would look like this in sktime: https://github.com/sktime/sktime/blob/main/pyproject.toml - minimal core dependency set; and dependencies are managed via tags like this: https://www.sktime.net/en/latest/api_reference/tags.html#general-tags-packaging

What do you think? I'd be happy to devote some time to shift the code base gradually towards this schema. As a side effect, it would also easily allow to interface all SPI as time seires distances in sktime, and would make it easier to add SPI for multivariate or unequal length time series.

FYI @jmoo2880

[fkiraly]

Here are concrete suggestions for the base class API design.

For context, the current strategy pattern is internally defined in base.py (in package root) and the classes therein.

I would simplify the design into a flat class (no inheritance hierarchy) with tags, as follows:

• BaseSPI inheriting from skbase BaseObject only
• two pairs of public/private method spi / _spi, and spi_mat / _spi_mat. This covers bivariate and multivariate, but slightly different as current - as in the next bullet point.
• the extender contract is restricted to _spi, _spi_mat, and object tags
• the signature is spi(data: Data) -> float, and spi_mat(data, data2=None, i=None, j=None). The calculation in spi_mat is always "bivariate", i.e., pairwise. spi_mat(data, data2) returns a 2D np.ndarray of float type, with (m,n)-th entriy corresponding to pairwise interaction of data[m] and data[n]. If i or j are provided it subsets the calculation to that row and/or column, resulting in an 1D or 0D np object. If data2 is not provided, it is identical to data.
• the decorators starting with parse move into the boilerplate, e.g., parse_multivariate happens in spi before it calls the internal _spi
• attributes such as name, identifier, labels move into skbase tags
• "property"-like tags such as issigned also move into skbase tags
• the logic for the default multivariate from Directed and Undirected moves into the default implementation of _spi
• optionally, we can provide methods bivariate and multivariate for downwards compatibility, if yes, it should raise a deprecation warning. Probably this is not needed, because the interface is not public.

[fkiraly]

FYI @Spinachboul, @benfulcher

[fkiraly]

oh, hi, @joshuabmoore!

So someone has started maintaining the package again?

See some suggestions above for API improvement.

[joshuabmoore]

Hi @fkiraly,

Thank you for the API improvement suggestions! We appreciate the feedback and we'll look into incorporating some of these ideas in a future release.

Yes, I'm currently maintaining the package, but only passively. There aren't any major overhauls planned at the moment, just minor updates and fixes as needed to keep things running smoothly.

[joshuabmoore]

Regarding integration with sktime:

We agree that making pyspi more accessible through its integration with sktime would be a great outcome. That said, I won't be playing an active role in the integration process myself, but fully support you or @Spinachboul taking the lead on this if interested.

If you'd like to proceed, you're welcome to create a fork of the repository and work from there. I'm happy to provide occasional pointers along the way if that would be helpful, but just to set expectations, I'm not currently in a position to offer active mentorship.

[Spinachboul]

@joshuabmoore @fkiraly @benfulcher
Yes! I would be glad to get started on this
Franz has already proposed an initial idea to get started with the integration , so I will be following that.
If in any doubt, I would be then asking the doubts here on Github (or maybe in the draft PR) where anyone could provide their insights which would be much appreciated.
Thanks everyone for getting involved in this.., really means a lot!!