SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls #64

paulhdk · 2024-08-27T10:09:23Z

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls

♻️ Current situation & Problem

This PR replaces the MacPaw/OpenAI package with generated API calls by the swift-openapi-generator package.
Calls are generated from OpenAI's official OpenAPI spec.
As discussed with @PSchmiedmayer, this marks the first step in adding the ability to send local image content to the OpenAI API.

This PR does not add any new features but simply replicates the existing feature set with the generated API calls.

I've tried my best to keep track of any known issues in-code with FIXMEs as well as in the following list.

Current Issues

Sources/SpeziLLMOpenAI/LLMOpenAISession+Generation.swift does not handle the "[DONE]" message sent by the API to conclude a stream. There is currently a hacky workaround that catches the error that is thrown in that case. I'm not quite sure yet how to handle that case elegantly.
There are several do ... catch blocks that catch OpenAI package-specific errors, which I had to commented out. I have not found a semantically equivalent solution for the generated API calls yet.
The LLMFunctionParameterItemSchema type does not use a generated type yet.
The convenience initialisers in SpeziLLMOpenAI/FunctionCalling should be, if possible, refactored, as they currently have a lot of optional bindings.
Correct error handling
Currently, openapi-generator-swift expects and an openapi.yaml and a configuration file in the TestApp, which is why there are duplicate openapi specs and configuration files in this PR. I'm not quite sure why it's expecting them in the TestApp, but I suspect it has something to do with the generated types being used in the TestApp's model selection mechanism.
The SpeziLLMTests are currently not passing. Because the test errors are realted to the above issues, I’ll update the tests once I’ve addressed all of the issues above.

⚙️ Release Notes

Replace the MacPaw/OpenAI package with Apple/swift-openapi-generator, which is able to generate API calls directly from OpenAI's official OpenAPI spec.

📚 Documentation

As no new functionality is added, nothing should change here (unless I missed something).

✅ Testing

This PR passes the existing tests. Since no new functionality has been added, I believe this should suffice.

📝 Code of Conduct & Contributing Guidelines

By submitting creating this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

I agree to follow the Code of Conduct and Contributing Guidelines.

code

endpoint This enables swift-openapi-generator to generate streamed responses. See: openai/openai-openapi#311

outgoing requests

See apple/swift-openapi-generator#622

API response errors

Swiftlint type_contents_order_violation warning

Fixes: a1fa636

…emSchema`

codecov · 2024-10-04T22:03:05Z

Codecov Report

Attention: Patch coverage is 52.01342% with 286 lines in your changes missing coverage. Please review.

Project coverage is 31.24%. Comparing base (e53bc15) to head (974449b).

Files with missing lines	Patch %	Lines
...peziLLMOpenAI/LLMOpenAISession+Configuration.swift	0.00%	77 Missing ⚠️
...ources/SpeziLLMOpenAI/LLMOpenAISession+Setup.swift	0.00%	51 Missing ⚠️
...s/SpeziLLMOpenAI/LLMOpenAISession+Generation.swift	0.00%	39 Missing ⚠️
Sources/SpeziLLMOpenAI/LLMOpenAIError.swift	0.00%	22 Missing ⚠️
...tionCalling/LLMFunctionParameterWrapper+Enum.swift	70.59%	20 Missing ⚠️
...ng/LLMFunctionParameterWrapper+OptionalTypes.swift	86.56%	16 Missing ⚠️
...SpeziLLMOpenAI/Helpers/LLMOpenAIStreamResult.swift	0.00%	16 Missing ⚠️
...nCalling/LLMFunctionParameterSchemaCollector.swift	0.00%	11 Missing ⚠️
...lling/LLMFunctionParameterWrapper+ArrayTypes.swift	88.74%	8 Missing ⚠️
...g/LLMFunctionParameterWrapper+PrimitiveTypes.swift	83.68%	8 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #64      +/-   ##
==========================================
+ Coverage   31.18%   31.24%   +0.07%     
==========================================
  Files          67       68       +1     
  Lines        3012     3198     +186     
==========================================
+ Hits          939      999      +60     
- Misses       2073     2199     +126

Files with missing lines	Coverage Δ
...penAI/Configuration/LLMOpenAIModelParameters.swift	`100.00% <100.00%> (ø)`
...iLLMOpenAI/Configuration/LLMOpenAIParameters.swift	`100.00% <ø> (ø)`
Sources/SpeziLLMOpenAI/Helpers/Chat+OpenAI.swift	`0.00% <ø> (ø)`
...I/Onboarding/LLMOpenAIAPITokenOnboardingStep.swift	`100.00% <ø> (ø)`
...enAI/Onboarding/LLMOpenAIModelOnboardingStep.swift	`97.88% <100.00%> (-2.12%)`	⬇️
Sources/SpeziLLM/Models/LLMContextEntity.swift	`33.34% <0.00%> (-1.66%)`	⬇️
.../FunctionCalling/LLMFunctionParameterWrapper.swift	`60.00% <33.34%> (+22.50%)`	⬆️
...ling/LLMFunctionParameterWrapper+CustomTypes.swift	`92.00% <91.31%> (-8.00%)`	⬇️
Sources/SpeziLLMOpenAI/LLMOpenAISession.swift	`22.86% <0.00%> (ø)`
...urces/SpeziLLMOpenAI/LLMOpenAIAuthMiddleware.swift	`0.00% <0.00%> (ø)`
... and 10 more

... and 5 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e53bc15...974449b. Read the comment docs.

PSchmiedmayer

Thank you for all the work here @paulhdk; very important to improve this setup and build on the OpenAPI specification!

It would be amazing to get a first insight from @philippzagar to get a good round of feedback.

Package.swift

PSchmiedmayer · 2024-10-15T14:14:17Z

Sources/SpeziLLMOpenAI/Configuration/LLMOpenAIModelParameters.swift

@@ -51,7 +50,7 @@ public struct LLMOpenAIModelParameters: Sendable {
    ///   - logitBias: Alters specific token's likelihood in completion.
    ///   - user: Unique identifier for the end-user, aiding in abuse monitoring.
    public init(
-        responseFormat: ChatQuery.ResponseFormat? = nil,
+        responseFormat: Components.Schemas.CreateChatCompletionRequest.response_formatPayload? = nil,


I am wondering if we should add compact type aliases for this?

I’ve added an LLMOpenAIRequestType alias. Does that work for you?

Should we also introduce an alias for Components.Schemas in general? This won’t make the types shorter, but something like LLMOpenAIGeneratedTypes could improve readability, maybe?

I think we can introduce well defined and named typealias for the specific types that we use in our API surface; we should see if we can make them compact and focus on them.

PSchmiedmayer · 2024-10-15T14:15:31Z

Sources/SpeziLLMOpenAI/FunctionCalling/LLMFunctionParameterArray.swift

+///                 "firstName": [
+///                     "type": "string",
+///                     "description": "The first name of the person")
+///                 ],


I am wondering if we can add a nicely typed type for this instead of a dictionary; it can always map to a dictionary under the hood. Would be cool to avoid loosing that type-safe element?

Previously, SpeziLLMOpenAI wrapped around the Swift types provided by the OpenAI package, which would then eventually be passed to the API.
With the OpenAI OpenAPI spec, such types aren't generated, but the JSON schemas are instead validated for correctness as they're being encoded in the OpenAPIObjectContainer type.

Introducing such wrapper types again would require precise alignment with the OpenAI, which would make it, I could imagine, harder to maintain over time.
I could imagine that’s one reason why the official OpenAI Python package, which is also generated from the OpenAI OpenAPI specification, does not offer wrapper types either, AFAICT.

What do you think?

I think adding an extension initializer/function that takes the well-typed arguments of one wants to use them would be beneficial and would avoid issues with string keys that are not correct or malformatted. Still allowing to pass in a dictionary might be an escape hatch that we can still provide. The OpenAPI surface is quite stable and if we use e.g. an enum for the type of the parameter can also have an other case with an associated string value.

Sources/SpeziLLMOpenAI/FunctionCalling/LLMFunctionParameterSchemaCollector.swift

Sources/SpeziLLMOpenAI/LLMOpenAIError.swift

Sources/SpeziLLMOpenAI/LLMOpenAISession.swift

See: apple/swift-openapi-runtime#115

PSchmiedmayer

Thank you for continuing to work on this @paulhdk!

I had a quick sync with @philippzagar and he will take a closer look at the PR to provide insights on the different changes; would be great to update the PR to the latest version of main to resolve the conflicts; I think after the feedback from @philippzagar we should be ready to get this merged 🚀

PSchmiedmayer · 2024-10-28T17:54:54Z

Sources/SpeziLLMOpenAI/LLMOpenAISession+Configuration.swift

+                .Input(body: .json(LLMOpenAIRequestType(
+                    messages: openAIContext,
+                    model: schema.parameters.modelType,
+                    frequency_penalty: schema.modelParameters.frequencyPenalty,
+                    logit_bias: schema.modelParameters.logitBias.additionalProperties.isEmpty ? nil : schema
+                        .modelParameters
+                        .logitBias,
+                    max_tokens: schema.modelParameters.maxOutputLength,
+                    n: schema.modelParameters.completionsPerOutput,
+                    presence_penalty: schema.modelParameters.presencePenalty,
+                    response_format: schema.modelParameters.responseFormat,
+                    seed: schema.modelParameters.seed,
+                    stop: LLMOpenAIRequestType.stopPayload.case2(schema.modelParameters.stopSequence),
+                    stream: true,
+                    temperature: schema.modelParameters.temperature,
+                    top_p: schema.modelParameters.topP,
+                    tools: functions.isEmpty ? nil : functions,
+                    user: schema.modelParameters.user
+                )))


Might be nice to format this similar to our other code bases; might be applicable to other parts as well:

Suggested change

.Input(body: .json(LLMOpenAIRequestType(

messages: openAIContext,

model: schema.parameters.modelType,

frequency_penalty: schema.modelParameters.frequencyPenalty,

logit_bias: schema.modelParameters.logitBias.additionalProperties.isEmpty ? nil : schema

.modelParameters

.logitBias,

max_tokens: schema.modelParameters.maxOutputLength,

n: schema.modelParameters.completionsPerOutput,

presence_penalty: schema.modelParameters.presencePenalty,

response_format: schema.modelParameters.responseFormat,

seed: schema.modelParameters.seed,

stop: LLMOpenAIRequestType.stopPayload.case2(schema.modelParameters.stopSequence),

stream: true,

temperature: schema.modelParameters.temperature,

top_p: schema.modelParameters.topP,

tools: functions.isEmpty ? nil : functions,

user: schema.modelParameters.user

)))

.Input(body:

.json(

LLMOpenAIRequestType(

messages: openAIContext,

model: schema.parameters.modelType,

frequency_penalty: schema.modelParameters.frequencyPenalty,

logit_bias: schema.modelParameters.logitBias.additionalProperties.isEmpty ? nil : schema

.modelParameters

.logitBias,

max_tokens: schema.modelParameters.maxOutputLength,

n: schema.modelParameters.completionsPerOutput,

presence_penalty: schema.modelParameters.presencePenalty,

response_format: schema.modelParameters.responseFormat,

seed: schema.modelParameters.seed,

stop: LLMOpenAIRequestType.stopPayload.case2(schema.modelParameters.stopSequence),

stream: true,

temperature: schema.modelParameters.temperature,

top_p: schema.modelParameters.topP,

tools: functions.isEmpty ? nil : functions,

user: schema.modelParameters.user

)

)

)

paulhdk added 6 commits August 27, 2024 12:00

SpeziLLMOpenAI: openapi-generator infrastructure

09214b2

SpeziLLMOpenAI: set "public" as the default access modifier generated

57a7fd8

code

SpeziLLMOpenAI: set a text/event-stream schema for the chat/completions

c0a5960

endpoint This enables swift-openapi-generator to generate streamed responses. See: openai/openai-openapi#311

SpeziLLMOpenAI: add a ClientMiddleware that injects the API key into

95bb39c

outgoing requests

SpeziLLMOpenAI: replace uses of MacPaw/OpenAI with generated API calls

99e0e6d

UITests: update SpeziLLMOpenAI tests for generated API calls

2b94bba

paulhdk requested a review from PSchmiedmayer August 27, 2024 10:09

PSchmiedmayer removed their request for review August 30, 2024 18:51

PSchmiedmayer assigned paulhdk Aug 30, 2024

PSchmiedmayer added the enhancement New feature or request label Aug 30, 2024

paulhdk force-pushed the generate-api-calls branch from 29be03d to d5a4eb3 Compare September 13, 2024 16:31

paulhdk added 13 commits September 13, 2024 09:32

LLMOpenAI: remove redundant marker

3d20437

LLMOpenAI: filter out "[DONE]" event in streamed responses

ccdb1d6

See apple/swift-openapi-generator#622

LLMOpenAI: remove FIXIT re: Swiftformat bug

b355e91

LLMOpenAI: address FIXMEs re: @parameter

6fba2ff

LLMOpenAI: Refactor setup() to reduce its length

cacfc09

LLMOpenAI: remove redundant FIXME

e3745e3

LLMOpenAI: fix error handling in LLMOpenAISession+Configuration.swift

268d335

LLMOpenAI: introduce LLMOpenAISession+ResponseHandler.swift for handling

d4fbb81

API response errors

LLMOpenAI: refactor getChatMessage() to reduce its length

cd6946a

LLMOpenAI: reorder LLMOpenAISession+Configuration.swift to fix

6ffee95

Swiftlint type_contents_order_violation warning

LLMOpenAI: move logger into global scope

0f04dfe

LLMOpenAI: fix error handling in FunctionCalling

b1d8691

TestApp: remove redundant OpenAPI spec + generator config

3ded304

paulhdk force-pushed the generate-api-calls branch from d5a4eb3 to 3ded304 Compare September 13, 2024 16:33

paulhdk added 5 commits September 13, 2024 09:34

Merge branch 'main' into generate-api-calls

e7257e1

LLMOpenAI: address some remaining Swiftlint warnings

a1fa636

LLMOpenAI: add licence header to LLMOpenAIAuthMiddleware.swift

5f61643

LLMOpenAI: add license header to openapi.yaml

b91b6d4

LLMOpenAI: add license header to openapi-generator-config.yaml

65c36d6

LLMOpenAI: comments

24abcfc

paulhdk marked this pull request as ready for review September 13, 2024 17:49

paulhdk added 4 commits September 13, 2024 11:41

LLMOpenAI: fix "functionCalls" assignment

ab9951d

Fixes: a1fa636

Use generated type for function calling

85f0eeb

Refactor error handling

6c6b784

Pass SpeziLLM tests

97b12ae

paulhdk force-pushed the generate-api-calls branch from 645ec81 to 97b12ae Compare October 4, 2024 20:30

paulhdk added 3 commits October 4, 2024 13:49

Move LLMFunctionParameterPropertySchema and `LLMFunctionParameterIt…

f5f48c9

…emSchema`

Remove redundant closing brackets

14ede91

Swiftlint

e2b435d

paulhdk force-pushed the generate-api-calls branch from ab4f137 to e2b435d Compare October 4, 2024 20:49

Fix tests

974449b

paulhdk force-pushed the generate-api-calls branch from 48f3c0b to 974449b Compare October 4, 2024 21:20

PSchmiedmayer requested a review from philippzagar October 15, 2024 14:11

PSchmiedmayer reviewed Oct 15, 2024

View reviewed changes

paulhdk changed the title ~~SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls~~ S Oct 24, 2024

paulhdk changed the title S SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls Oct 24, 2024

paulhdk added 3 commits October 28, 2024 09:54

Remove OpenAI dependency from SpeziLLMOpenAI in Package.swift

ecd470f

Rename local var in LLMFunctionParameterSchemaCollector.swift

ee76967

Introduce LLMOpenAIRequestType type alias

1f52a7f

paulhdk force-pushed the generate-api-calls branch from bdd7127 to 1f52a7f Compare October 28, 2024 08:54

paulhdk added 2 commits November 4, 2024 14:46

LLMOpenAI: refactor generation call to OpenAI API

a3db281

See: apple/swift-openapi-runtime#115

LLMOPenAI: Remove redundant OpenAI package imports

03e4fd5

PSchmiedmayer reviewed Nov 5, 2024

View reviewed changes

Merge branch 'main' into generate-api-calls

cb3c6e4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls #64

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls #64

paulhdk commented Aug 27, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024

PSchmiedmayer left a comment

PSchmiedmayer Oct 15, 2024

paulhdk Oct 28, 2024

PSchmiedmayer Nov 6, 2024

PSchmiedmayer Oct 15, 2024

paulhdk Nov 5, 2024

PSchmiedmayer Nov 6, 2024

PSchmiedmayer left a comment

PSchmiedmayer Oct 28, 2024

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls #64

Are you sure you want to change the base?

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls #64

Conversation

paulhdk commented Aug 27, 2024 • edited Loading

SpeziLLMOpenAI: Repalce MacPaw/OpenAI With Generated API Calls

♻️ Current situation & Problem

Current Issues

⚙️ Release Notes

📚 Documentation

✅ Testing

📝 Code of Conduct & Contributing Guidelines

codecov bot commented Oct 4, 2024

Codecov Report

PSchmiedmayer left a comment

Choose a reason for hiding this comment

PSchmiedmayer Oct 15, 2024

Choose a reason for hiding this comment

paulhdk Oct 28, 2024

Choose a reason for hiding this comment

PSchmiedmayer Nov 6, 2024

Choose a reason for hiding this comment

PSchmiedmayer Oct 15, 2024

Choose a reason for hiding this comment

paulhdk Nov 5, 2024

Choose a reason for hiding this comment

PSchmiedmayer Nov 6, 2024

Choose a reason for hiding this comment

PSchmiedmayer left a comment

Choose a reason for hiding this comment

PSchmiedmayer Oct 28, 2024

Choose a reason for hiding this comment

paulhdk commented Aug 27, 2024 •

edited

Loading