Skip to content

Add IO processor for certainty intrinsic #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Apr 10, 2025
Merged

Conversation

frreiss
Copy link
Collaborator

@frreiss frreiss commented Apr 4, 2025

This PR adds an IO processor for the Granite certainty intrinsic, AKA the Granite 3.2 8B Instruct Uncertainty LoRA.

In addition to the IO processor, I've also added some support functionality to the library:

  • A new base class for IO processors that mutate generation arguments, i.e. for constrained decoding
  • An implementation of Add back support for captive vLLM server #41 to support testing the new IO processor with the target LoRA adapter

I also include a notebook and test cases.

@frreiss frreiss requested review from markstur and hickeyma April 4, 2025 21:23
frreiss added 9 commits April 4, 2025 14:47
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
Signed-off-by: Fred Reiss <[email protected]>
@frreiss
Copy link
Collaborator Author

frreiss commented Apr 4, 2025

Notebook CI tests failing due to disk space issue. Not sure how to fix that.

@hickeyma
Copy link
Collaborator

hickeyma commented Apr 7, 2025

Notebook CI tests failing due to disk space issue. Not sure how to fix that.

Try adding disk cleanup (https://github.com/ibm-granite/granite-io/blob/main/.github/workflows/test.yml#L59C7-L60C48) before setting up Python (https://github.com/ibm-granite/granite-io/blob/main/.github/workflows/test_examples.yml#L58)

Copy link
Collaborator

@hickeyma hickeyma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a shallow review not but noticed vLLM backend and also potential fix for the ci disk issue. Will return later to review further.

Signed-off-by: Fred Reiss <[email protected]>
@frreiss
Copy link
Collaborator Author

frreiss commented Apr 7, 2025

Notebook CI tests failing due to disk space issue. Not sure how to fix that.

Try adding disk cleanup (https://github.com/ibm-granite/granite-io/blob/main/.github/workflows/test.yml#L59C7-L60C48) before setting up Python (https://github.com/ibm-granite/granite-io/blob/main/.github/workflows/test_examples.yml#L58)

Thanks, adding that step to the workflow has cleared up the CI issues.

Copy link
Collaborator

@markstur markstur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see questions inline

Copy link
Collaborator

@hickeyma hickeyma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@frreiss Some small things

@hickeyma hickeyma self-requested a review April 9, 2025 16:39
Signed-off-by: Fred Reiss <[email protected]>
@markstur markstur dismissed hickeyma’s stale review April 10, 2025 01:15

Martin said on internal slack that this was okay

Looks good to go. Just some small comments. @mark.sturdevant I'll leave it to you to decide

... and I feel like we need to merge it and iterate

Copy link
Collaborator

@markstur markstur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge this and then iterate on things like supportability, usability, docs, test limitations...

@markstur markstur merged commit b0f5d8e into ibm-granite:main Apr 10, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants