Add retrieval RequestProcessor and end-to-end RAG examples

[frreiss]

This PR adds a RequestProcessor that performs the retrieval phase of the RAG pattern. There is a basic implementation that uses an in-memory vector database and an extension point for adding support for other vector databases in the future.

The PR also includes tests for the new functionality.
I have refactored the RequestProcessor for hallucinations so that it can also be used to perform query rewrite.

This PR also includes a notebook that shows several end-to-end RAG examples that use different combinations of intrinsics.

[hickeyma]

@frreiss Do you mind fixing the gate issues?

[frreiss]

Linter issues fixed.

@hickeyma and @markstur I'm seeing test failures because the test cases can't download the data files they need. Is there a way to get around that limitations?

[frreiss]

CI problems with base data fixed now, but am seeing issues with using an embedding model from SentenceTransformers:

@staticmethod
      def load(input_path) -> Pooling:
  >       with open(os.path.join(input_path, "config.json")) as fIn:
  E       FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.cache/huggingface/hub/models--sentence-transformers--multi-qa-mpnet-base-dot-v1/snapshots/4633e80e17ea975bc090c97b049da[260](https://github.com/ibm-granite/granite-io/actions/runs/14499640836/job/40676006992?pr=148#step:8:262)62b054d3/1_Pooling/config.json'

@hickeyma @markstur is there a special trick to get models into the Hugging Face cache directory for CI?

[hickeyma]

@frreiss I pushed commit ce00ef3 and that fixes the download of embedding model issue.

There are now the following issues:

• Test fail: AssertionError: assert [-0.110380716...87530518, ...] == approx([-0.11...17 ± 1.2e-08])
• There are also issues with Nvidia GPUs not available: RuntimeError: Found no NVIDIA driver on your system. Can potentially use the following check: if torch.cuda.is_available():

Do you mind addressing those issues?

[frreiss]

Tests are passing now.

Did an internal review of the notebook rag.ipynb with the researchers who created the models involved this morning; some additional changes recommended before we merge this PR.

[hickeyma]

Thanks @frreiss for the PR.

Overall, it looks good. Small nit inline and maybe could squash the 2 notebooks into 1 as rag.ipynb incorporates retrieval.ipynb.

You mentioned that you wanted to update the notebooks. I am going to merge for now and lets do that in a follow up PR.

[hickeyma]

Ok for the moment. However, need to find a repo which is part of the community.