Skip to content

Conversation

@pedrocarlo
Copy link
Contributor

@pedrocarlo pedrocarlo commented Dec 14, 2025

Description

Depends on #4208 #4207

The idea is to have a test runner that can run tests in parallel and we can just define one file to run in different backends. For now we only have the CLI backend is implemented, but we could create separate backends for our SDKs, so that we can always test them. One way we could integrate more backends, is by just having the backend print to stdout, but we could also have some GRPC backend that is easier for other languages to interact with as well.

Also, the dsl here can be expanded so that we won't even need to define the expected value if we don't want to, and always compare with SQLite's output

Motivation and context

Test SDKs with our existing test suite without duplicating a gazillion tests in different languages

AI Disclosure

Vibecoded the foundations of the runner almost in one shot. Made some manual adjustments. Really impressed.
Tests were migrated with Claude, need to check if all the functionality is there

@penberg
Copy link
Collaborator

penberg commented Dec 15, 2025

Happy to be convinced otherwise, but this does not seem like a path we should pursue. For starters, you define a whole new test syntax without explaining why we need it. Should we go that path, we should write some amazingly good test cases to demonstrate why we need that. Also, I am not at all convinced we need different backends for the tests. I can imagine we need test runners in different languages to run an unified test suite, but a gRPC thing makes no sense to me.

If you really, really want to take this path, you need to demonstrate the feasibility of the approach by implementing some insanely nice test cases that would end up looking like crap in TCL. You should then implement a minimal test runner, not one-shotted AI slop that nobody is even going to read.

@pedrocarlo pedrocarlo marked this pull request as draft December 15, 2025 14:34
@pedrocarlo pedrocarlo force-pushed the test-runner branch 8 times, most recently from 026351a to 14c0d55 Compare December 20, 2025 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants