Feasibility of registering external entities like pipelines in gravitino

[vishnu-chalil]

Hi Gravitino team,

Iam looking at the possibilites of integrating external entities say for example, Iam running spark jobs to ingest iceberg tables into gravitino. Would it be somehow possible to register those spark jobs as well as part of apache gravitino? Does gravitino support this at the moment? If not how can we manage it?

[mchades]

You may be looking for #7154

[vishnu-chalil]

Awesome. Could this be used to just register the pipelines and probably not execute them? What Iam expecting is something similar to registering custom entities in other open source data catalogs. Also when is the expected release for these features?

[mchades]

@jerryshao is working on this feature; maybe he can answer whether the job system can handle your scenario.

The Gravitino job system is expected to be released in version 1.0.0.

[jerryshao]

The job system is different from what you mentioned "pipeline" here. The thing you wanted is to register the custom entities to Gravitino, then you can fetch it when needed, am I right?

Currently, we don't support such custom entities. And for job system, we support registering job templates (kinda of pipeline) to Gravitino, then you can get it. It may not strictly align to your requirement. Maybe you canreview the code first. Also, it is welcomed if you can propose an idea.

[vishnu-chalil]

The job system is different from what you mentioned "pipeline" here. The thing you wanted is to register the custom entities to Gravitino, then you can fetch it when needed, am I right?

Yes this is exactly what I was looking for.

And for job system, we support registering job templates (kinda of pipeline) to Gravitino, then you can get it.

This should suffice as of now.

Maybe you canreview the code first. Also, it is welcomed if you can propose an idea.

Sure. Will go through it first

Thanks a lot for replying promptly. @jerryshao @mchades