Exposing flatbuffer compilation internals

[CasperN]

The maintainers had some conversations and discussions around a significant refactoring of flatc to expose more internals somehow.

However, we're not sure if there are enough use cases to justify such an effort, or even guide its technical direction. So, I'm now crowd sourcing ideas. So... Where does flatbuffers tooling fall short, and can these be addressed by exposing more information that currently exists in flatc? And also how do people feel about the priority of this effort relative to some other large projects such as implementing the ideas in #5875 or #6053

Some use cases:

• U1. Implementing new code generators for new languages in those languages
• U2. Easier front end for projects like flattools and dflats
• U3. Generated validators: example
• U4. Generated fieldmasks (like this but not based on strings)
• U5. Might make an XML to/from flatbuffers tool easier to implement Converting XML to Flatbuffers  #6033
• U6. Generated custom flatbuffer -> string methods
• U7. Invoke code generators owned by other codebases, e.g. swift-grpc

Some methods

• M1: dynamic linking
  • This does not improve how many languages flatc can be developed in and forces us to deal with cross platform dynamic linking and does not seem to provide enough marginal benefits per cost
• M2: scripting languages
  • We will move forward with M3 since M2 does not support U7 and since M3 does not actually prevent M2, should we choose to pursue it in the future.
• M3: exposing a json/bfbs intermediate representation (We currently have reflection.fbs which may be extended or replaced)

Some positive side effects

• P1: It'll make flatbuffers internals easier to understand (idl_parser pretty organically grown and messy right now).

Some preferences / potential requirements

• R1: If we go with M3, we should try to integrate with reflection.fbs rather than starting from scratch
• R2: If we go with M2, we should reimplement our code generators in the scripting language.
• R3: We should support be able to specify multiple levels of information stored in reflection.fbs; optionally include attributes, comments, etc. This can be configured to "give enough for reflection" and "give enough for compilation / everything"
• R4: The IR should be JSON compatible, in case a generator is to be implemented in a language without current flatbuffers support

@dbaileychess @mikkelfj @aardappel @krojew @paulovap @adsharma

(I intend to keep this top comment up to date with discussion below)

[CasperN]

fwiw, I'm totally biased towards M3. protbuffers and captnproto do something like that: There's a CodeGenerator(Request|Response) exchange between the compiler and generator.

[adsharma]

A combination of M3 and M2 might have a few extra benefits in addition to what you outline. * Make flatbuffers internals easy to understand * Modularization - parser, buffer manipulation, template based codegen * Integrate with projects like pyserde

…

On Fri, Jan 29, 2021, 7:39 AM Casper ***@***.***> wrote: fwiw, I'm totally biased towards M3. protbuffers <https://developers.google.com/protocol-buffers/docs/reference/other> and captnproto <https://capnproto.org/otherlang.html> do something like that: There's a CodeGenerator(Request|Response) exchange between the compiler and generator. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6428 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFA2A2AXT5N3AHGXG3GWPDS4LJCDANCNFSM4WZBV7XA> .

[krojew]

Exposing intermediate representation looks like a good idea - it's essentially what modern compilers do.

[mikkelfj]

I'm leaning towards M3 from a flatcc perspective. I believe M2 is useful, but it will not happen for flatcc for portability reasons - scripting engines are not as portable as native. Except if the scripting engine runs in an external process on top of M2. The would also allow easier interop between flatc and flatcc. Even with out of process scripting, flatcc will maintain an internal code generator for C for portability, but it might be rewritten to rely on M3. Except there is a chicken and egg problem for accessing the bfbs interface, so I am not fully convinced here even if attractive.

To clarify. For flatcc and any language other than C, M3 is certainly of interest, and out of process M2 also, via M3.

[CasperN]

Except there is a chicken and egg problem for accessing the bfbs interface, so I am not fully convinced here even if attractive.

I mean, that's only true if you want to bootstrap the code-generator for language X in that language, and i don't think that's such an important use case that we'd design around it. Even still, we can make the schema json-compatible and optionally output the IR in that form.

[adsharma]

For flattools/python case I'm not sure if a json representation is super useful. Parsing the source is easier (there is a ply-python parser that works well). But a side effect of such a refactoring is that a standalone serde independent of the parser is generated and that sounds interesting.

…

On Sat, Jan 30, 2021, 3:05 PM Casper ***@***.***> wrote: Except there is a chicken and egg problem for accessing the bfbs interface, so I am not fully convinced here even if attractive. I mean, that's only true if you want to bootstrap the code-generator for language X in that language, and i don't think that's such an important use case that we'd design around it. Even still, we can make the schema json-compatible and optionally output the IR in that form. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6428 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFA2A3LP46ILDXXQOEDB3LS4SGEDANCNFSM4WZBV7XA> .

[mikkelfj]

Parsing the source is not easier because parsing is only 10% of what a Flatbuffer compiler does before generating code.

[adsharma]

Sure - there is more work to do beyond pure parsing. What I was getting at is the somewhat sorry world of json schema and validating that the input json is well formed. I much prefer parsing a fbs file vs parsing json and then validating that it conforms to some json schema.

[CasperN]

Sure - there is more work to do beyond pure parsing. What I was getting at is the somewhat sorry world of json schema and validating that the input json is well formed. I much prefer parsing a fbs file vs parsing json and then validating that it conforms to some json schema.

I think this thread is a little off topic. One can assume that flatc would only output valid bfbs/json/$favorite_serde_format, and also the resulting schema has been type checked, name resolved, etc checked, fits a predefined schema, and is good for code generation in all respects.

• Integrate with projects like pyserde

How would this work? It seems like pyserde defines data types in python which may conflict with the flatbuffer schema. Which would be the source of truth?

[adsharma]

On Mon, Feb 1, 2021 at 10:32 AM Casper ***@***.***> wrote: - Integrate with projects like pyserde How would this work? It seems like pyserde defines data types in python which may conflict with the flatbuffer schema. Which would be the source of truth?

fbs file will be the source of truth. flatc.py can generate a dataclass from the fbs file, which could be decorated with decorators provided by pyserde.

[aardappel]

I personally would prefer M2, but only if we committed to cleaning up / converting all the C++ generators we have into said scripting language.
If the C++ generators are going to continue being a thing, then M1/M3 are better.

M1 is kinda useless, since it still restricts the languages a generator can be written in a great deal. It also requires we deal with cross-platform dynamic linking, and making sure there is a dll/so for each platform.. not much of an improvement over just static linking it into flatc, imho.

M3 is definitely useful. One thing I wouldn't like is if we ended up with reflection.fbs and generator.fbs and they looked pretty similar. We have already enhanced reflection.fbs a few times in the past for the sake of external generators. I'd be in favor of extending reflection.fbs further, with a command-line flag that controls the level of information in it, as opposed to starting from scratch for the purpose of generators.

An extension of M3/M3 is if we took care of invoking scripts from flatc. Imagine you say flatc -gen foo.py schema.fbs, and flatc would first write out the .bfbs, found if python was installed and ran it for you. This saves the user having to write a script that runs both flatc and python, and possibly doing so for multiple platforms. Then again, not sure if we want to be in the business of running commands.

[mikkelfj]

Then again, not sure if we want to be in the business of running commands.

I imagine you pipe bfbs or json output from flatc into a script. And possible add a wrapper script to simplify things.

[mustiikhalil]

One of the things that would actually be worth it, is allowing the flatc integrate with other already existing generators (GRPC) as an example. This is one of the points I talked with the guys over at swift-grpc. Since this would allow us to keep our code base consistent and not break each time there is a new version of GRPC

fwiw, I'm totally biased towards M3. protbuffers and captnproto do something like that: There's a CodeGenerator(Request|Response) exchange between the compiler and generator.

which is what you suggested in M3 I believe

[CasperN]

One of the things that would actually be worth it, is allowing the flatc integrate with other already existing generators (GRPC) as an example

I think you mean M3 will allow the swift-grpc codebase to own the flatbuffers-swift-grpc generator, which will be invoked by flatc. Is that correct?