-
Notifications
You must be signed in to change notification settings - Fork 661
WIP test runner #4210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
WIP test runner #4210
Conversation
4f5324a to
da3abdc
Compare
e3f2a2c to
821975d
Compare
|
Happy to be convinced otherwise, but this does not seem like a path we should pursue. For starters, you define a whole new test syntax without explaining why we need it. Should we go that path, we should write some amazingly good test cases to demonstrate why we need that. Also, I am not at all convinced we need different backends for the tests. I can imagine we need test runners in different languages to run an unified test suite, but a gRPC thing makes no sense to me. If you really, really want to take this path, you need to demonstrate the feasibility of the approach by implementing some insanely nice test cases that would end up looking like crap in TCL. You should then implement a minimal test runner, not one-shotted AI slop that nobody is even going to read. |
026351a to
14c0d55
Compare
…y when sending sql commands
85b278e to
eb343fc
Compare
Description
Depends on #4208 #4207
The idea is to have a test runner that can run tests in parallel and we can just define one file to run in different
backends. For now we only have the CLI backend is implemented, but we could create separate backends for our SDKs, so that we can always test them. One way we could integrate more backends, is by just having the backend print to stdout, but we could also have some GRPC backend that is easier for other languages to interact with as well.Also, the dsl here can be expanded so that we won't even need to define the expected value if we don't want to, and always compare with SQLite's output
Motivation and context
Test SDKs with our existing test suite without duplicating a gazillion tests in different languages
AI Disclosure
Vibecoded the foundations of the runner almost in one shot. Made some manual adjustments. Really impressed.
Tests were migrated with Claude, need to check if all the functionality is there