feat(collections): add annotations for handling collections

[jsdonn]

Summary

Add @gma.collection. annotation for supporting collections. There are 3 items to specify:

1. isCollection - should be true for collections. default false.

2. primaryKeys - struct (essentially a map) representing which fields to use as primary keys for collection-typed fields. for example, given:

upstream: {
    dataset_urn: string
    audit_stamp: {
        time: long
        actor: string
    }
}

I can specify that I want the primary key for this field to be the dataset_urn and actor fields. i.e. "upstream": ["dataset_urn", "audit_stamp.actor"]

3. defaultUpdateBehavior - enum indicating how to handle updates to existing entries in the collection. default REPLACE_BY_KEY

Next step is to add logic to metadata models plugin to convert proto annotations to this pdl annotation

Testing Done

Add a unit test.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Docs related to the changes have been added/updated (if applicable)

[ZihanLi58]

how do you plan to support delete?

[jsdonn]

I don't think we can do it here at the model level, since all AssetLabel instances will have to have the same UpdateBehavior. Instead, I'm thinking to use the IngestionMode field of IngestionParams to specify what kind of update is to be performed. So something like:

LabelService
    -> update(new label)
             -> call assetService.update(mode=LIVE) w/ default UpdateBehavior
    -> delete(existing label)
             -> call assetService.update(mode=DELETE_FROM_COLLECTION) which will override the UpdateBehavior

[jsdonn]

or even more generic than the LabelService, maybe we can have a CollectionsService just for handling Collections-type aspects

[ZihanLi58]

if we do the collection, let's not have a separate service for it then. Probably you can explore the existing delete api we already have in MMG. And saying that we have default update behavior and default delete behavior etc.

[jsdonn]

Sounds good, I can also update the delete API to include IngestionParams

[ZihanLi58]

I think we have another annotation defined in the plugin, how will that work? can we consolidate the two here? I will prefer us to deprecate the ppl annotation here at all.

[jsdonn]

The annotation on the internal model will get translated to this pdl annotation (there will be logic in metadata-models-plugin to do this). We cannot fully deprecate the pdl annotation without either moving datahub-gma into a LI MP or starting to rely on proto in datahub-gma.

Essentially, the proto annotation (i.e. (.proto.mg.collection={...}) will get translated into this pdl annotation @gma.collection = {...}

[ZihanLi58]

If we need to process the annotation and do some customization on the annotation, can we just do it in MGA (i.e. a layer up of DAO). I think you will anyway need to check the update method and the collection type before we enter DAO layer?

[jsdonn]

I guess the main tradeoff for where we put the annotation processing logic is how many db operations need to be done?

if we put in MGA, we need to do

1. get() to get the latest value of the collection so we may manipulate it before calling dao.add
   2a) getLatest() as part of the dao.add
   2b) saveLatest()

if we put in DAO, we just don't need the first get() so it will just be 1 read, 1 write.

but agree with you that in the second case, we will have to process the annotation in both MGA and in DAO.

I admit that the performance gained by going with this approach is insignificant, let's go just put the annotation and processing in MGA instead. thanks for the comment.

[ZihanLi58]

Oh I just saw this message, thanks for taking the suggestion. But for the part where we process the annotation on MGA, I would suggest you to call the add with preprocessFunction method to avoid the duplicate get call and make sure the update is atomic.