obtain the expanded schema without importing it to fauna

[zvictor]

TLDR for the @fauna team: Compiled suggestions on how to improve Fauna and fix this issue can be found here.

---

Currently, in order to generate TS types (faugra generate-types and faugra build-sdk), faugra makes some compromises that won't be acceptable to the general audience.

The problem

Given a schema file, faugra uploads its content to fauna in order to:

• have any missing "base" schema appended to it (it adds e.g. scalar Date and directive @resolve)--> primitive values are being hardcoded into base.gql instead.
• have the schema expanded with the basic CRUD methods (e.g. findAll<Type>ByID)

Without uploading the schema to the cloud the TS types would be incomplete , lacking the content that fauna adds to it.

Current solution

Putting all together, in order to generate the TS types, faugra needs to:

1. prepare the schema (which potentially comes from a merge of multiple files)
2. upload the basic schema (requires credentials and can have serious unintended consequences)
3. download the expanded one (requires credentials)
4. run graphql-codegen

As modularisation is a core principle of faugra we need to repeat this process for each file individually (--> We had to give up on that because of this issue). But, if we do not reset the database before pushing the new schema in, fauna will merge the content of the files. The last schema uploaded will in practice extend the content of all schema files pushed before. Therefore, importing the schema in override mode is a must.

Considerations

Considering the performance and side effects of steps $2 and $3 I believe that I can't have the TS types being generated "on save", as I initially planned. And, after all considered, I wonder if anyone would actually bother going through the hassle of setting up such tool that requires credentials and mess with your data.

So, we need to find a way to kill steps $2 and $3: we need to programmatically add the missing content in the basic schema instead of publishing it to the cloud.
Where do we start? 🙃

[zvictor]

If we could programatically expand the schemas ourselves, we would not need to upload the schema to fauna.

As it looks like that fauna is running a loosely compliant version of openCRUD, I believe that we could run a script that would make any initial schema expand to add the openCRUD components. That would give us pretty much the same result as uploading the schema to fauna (we could then customize it to be fully compliant).

Sadly, building such script would a big project on itself.

I managed to find only 2 packages that are doing that already, but they have a lot of added flavours on top:
@graphback/codegen-schema and prisma-generate-schema.

They seem to achieve what we need, but they are wrapped in lots of extra complexities that serve the specific needs of their frameworks and simplifying them would be as hard as starting from scratch.

Therefore, we are still sadly dependent on fauna's team goodwill to help us moving forward.

@erickpintor @lregnier @n400 is there any chance you could advocate inside fauna to open source the part of the code that handles the schema expansion? it's virtually impossible to build any extension like faugra having to rely on unaccessible parts

[n400]

Hi @zvictor, Leo and Erick are pretty swamped right now with the upcoming release. I've added this to our roadmap and will discuss it with the Product and Engineering teams to see how feasible it is (being tracked internally as ROAD-245). Feel free to email or DM me on Slack for an update. Sorry that the project has been blocked :/

[zvictor]

Hi @n400! It looks like you are not working with Fauna anymore, which is very sad to hear 🥹
Do you know who I could contact to help me fighting for the OSS community inside Fauna?

[n400]

Hi @zvictor! It’s good to hear from you. I would try reaching out to @rts-rob, Head of DA. He’s passionate about the OSS community and started Fauna Labs .

[rts-rob]

Thanks @n400 !

Hi @zvictor - looping in @Shadid12 who has done some related work in tooling, all available in Fauna Labs.

You can also find us in our forums and Discord server!

[zvictor]

Thanks @n400!

@rts-rob @Shadid12 would you be able to check and report on ROAD-245? Is it still an open issue or is the work done there?

[zvictor]

To be very clear and make it easier for the Fauna team to help us here, I would like to present 2 proposals of solutions.

A) Publishing and Maintaining a Package

It would be great to have the Fauna code open sourced, for obvious reasons, but that is something out of our reach of influence apparently. However, that does not mean that relevant pieces of code could not be open sourced as separated packages, right?

Somewhere inside Fauna the schema we upload gets expanded to include the basic CRUD methods, either while it's still just a GraphQL schema or somewhere else when it's already being processed and being converted into Collections and so on. Regardless of when/where that happens, we need to split parts of it out and publish them as independent packages.

Libraries like fauGRa and it's sort can only be built if we have an environment that is predictable and reliable. Otherwise, "magic features" like the creation of the basic CRUD methods become unreachable, breaking the composability principle.

Having a package maintained by Fauna that we can call from our code in order to expand the schema ourselves would be great!

B) Adding a New Import Mode

POST /import?mode=dry-run

A new mode that would take the schema, validate it, and return the expanded/final schema from Fauna.
Currently we achieve that result by pulling and then pushing the schema from Fauna:
https://github.com/zvictor/faugra/blob/6e6ac349605bedb64c8aef1ca8e0dea43f1de80b/commands/generate-types.js#L35-L37

Needless to say, that's a terrible solution 🥲

The dry-run endpoint/mode must be publicly accessible (i.e. with no need to present a secret key). Otherwise, users wouldn't feel comfortable delegating the secret keys of (possibly) production databases to tools that are supposed to run things in dry-mode. Plus, it would add extra work in the CI/CD and whatever other environments that would need to keep an extra key just for dry-run operations.

[zvictor]

Some reflections on the proposals:

• Option A is definitely my favorite because it does not require network access. Option B would break the rules of some CI/CD environments that, for security reasons, discourage the use of network in automated operations.

• Solving this issue would allow us to easily fix new CLI interface #10. The UX would improve drastically.