Might need to update this to fit with the updated roadmap.

@adamkells I updated the description for this issue, are you up for taking this

Yeah I can take this. I think the updated version of the issue is much cleaner than previously.

Summary of Findings

I looked at the four packages suggested (sklearn, spaCy, langchain, hugging-face).

These package all introduce a concept of pipelines. However each is slightly different.

• sklearn: A pipeline has fit + predict/transform methods so it can be applied flexibly to input data of an expected form and return model predictions.
• spaCy: Converts a text input to a Document and then applies operations which add metadata to the Document.
• hugging-face: Similar to spaCy but has more variety in pipelines for different tasks which means it has more flexible pipeline outputs (e.g. a list of dictionaries containing predictions).
• langchain: Doesn't have a direct concept of a pipeline but instead has an LLMChain which takes an llm and a prompt and passes it through a chain of tasks to provide a flexible output (usually text).

If I had to rank these in terms of how neatly they fit into our framework:

1. spaCy: Almost identical in concept and naming convention to our existing pipelines.
2. hugging-face: Very similar but as it can handle a wider range of tasks it will need to be restricted to a narrower list of tasks which fit our framework.
3. langchain: Task and language is a bit different but can be wrapped in a way that makes it compatible with ours.
4. sklearn: Very different as it is by far the most flexible and is not typically applied to nlp tasks. I also think this is relatively easy for an experienced sklearn user to integrate on their own.

Proposal

To add methods to instantiate pipelines from spaCy, langchain and hugging-face and to omit sklearn for the moment.

The api can be as simple as from_spacy etc and to check that the inputs and outputs of the pipeline conform to our constraints. We can throw errors like from_hf expects a pipeline which accepts text as input and returns either string or list etc.

This is great 🌟 Let's focus on text pipelines and park the others for future. Are you happy to also create an issue and work on the implementation @adamkells ?

Also what did you mean by:

I also think this is relatively easy for an experienced sklearn user to integrate on their own.

Yeah happy to work on it.

With sklearn, I just meant that the components of an sklearn pipeline which a user may want to use make more sense to be just added manually as a part of the sandbox than to be an initialiser for the healthchain pipeline.