Support Internal Catalog

[melin]

Support Internal Catalog, gravitino stores metadata

[liunaijie]

Hi, I have a question about this.
What does the metadata of the Internal catalog represent?

In my understanding, gravitino is a metadata management system, and it does not store data.
If an internal catalog is supported, what content does it manage?

[melin]

To create a catalog, the gravitino needs to be bound to hms or jdbc. We can call it an external catalog. It is hoped that creating a catalog does not require specifying an external metastore; it is referred to as an internal catalog.

When creating a catalog in UnityCatalog, there is no need to specify a metastore. Table metadata stored in unitycatalog:
https://github.com/unitycatalog/unitycatalog/blob/b141f19665c0c88ed6b72a04149881690ac29dcd/server/src/main/java/io/unitycatalog/server/persist/TableRepository.java#L108

[liunaijie]

To create a catalog, the gravitino needs to be bound to hms or jdbc. We can call it an external catalog. It is hoped that creating a catalog does not require specifying an external metastore; it is referred to as an internal catalog.

When creating a catalog in UnityCatalog, there is no need to specify a metastore. Table metadata stored in unitycatalog: https://github.com/unitycatalog/unitycatalog/blob/b141f19665c0c88ed6b72a04149881690ac29dcd/server/src/main/java/io/unitycatalog/server/persist/TableRepository.java#L108

Thanks for your reply, but it still hasn't cleared up my confusion.

For the external catalog, every entity—such as a Table or Topic—has a corresponding counterpart in other systems (e.g., a Hive table or Kafka topic). Gravitino serves as a unified system to manage all these entities collectively.

So, regarding the internal catalog: what do a Entity (Table) represent?

Technically, an HMS is required. If the customer requests a cloud-native solution, they may question why an HMS is still needed (believing that only Hadoop would require an HMS). It is hoped to find an alternative solution to HMS. It is believed that more people have such demands.

what's the relationship between cloud-native and metadata?

As someone who's just starting to learn about these concepts, I'd really appreciate it if you could explain this to me in a way that's easy for a novice to grasp—thank you!

[melin]

As someone who's just starting to learn about these concepts, I'd really appreciate it if you could explain this to me in a way that's easy for a novice to grasp—thank you!

Some enterprises have upgraded their technical architectures to cloud-native solutions and do not want to use the hadoop ecosystem software. HMS is part of the hadoop ecosystem. An explanation of the reason is required.

[jerryshao]

The managed catalog can only manage the delta table for UnityCatalog. Foreign catalog in UnityCatalog is the same as we do in Gravitino (like a proxy catalog). If you want to use Iceberg, you don't have to deploy any other service; a Gravitino server and a RDBMS is enough, like Delta table in UC (just a UC server and a RDBMS).

[melin]

The UnityCatalog OSS catalog does not distinguish whether it is a managed catalog. However, it can create tables in parquet, orc, delta, and icberg formats. Only delta can be a managed table. The main purpose is to automatically optimize the table in conjunction with the Datarbicks platform.

https://github.com/unitycatalog/unitycatalog/pull/557/files#diff-43434307380b8612b5bd964c4698281c6cee09583781b79814864404ed91afc8

[jerryshao]

What you checked is the open source version, you should check their enterprise version. These two projects have different focuses, and I really don't want to do apple-to-apple comparison.

[melin]

In China, data development personnel generally do not pay much attention to whether to use a data lake. They are more focused on real-time data being written to the data warehouse. A data lake is needed. There is no such high degree of substitution in the foreign market as using iceberg to replace hive. Only some ODS tables use Paimon or iceberg. The main ones still use Parquet format tables. Using a data lake will also increase maintenance costs and storage costs. It is relatively difficult to implement using iceberg or Paimon for all tables.

Technically, an HMS is required. If the customer requests a cloud-native solution, they may question why an HMS is still needed (believing that only Hadoop would require an HMS). It is hoped to find an alternative solution to HMS. It is believed that more people have such demands.

[melin]

apache polaris support Internal Catalog