Managed YAML Repositories

[shangyian]

At the moment, we support using YAML to do semantic modelling, and these YAML files can be tracked in a repository and synced with the DJ server via command line or the python client. We should turn this setup into a properly governed, declarative source of truth, with clear ownership and change control.

What we need to make it better:

• Ability to register a YAML repository, which serves as a source of truth for a subset of DJ nodes. This would store metadata like the repo URL, owner, type (Git, S3, etc.), and sync method.
• Prevent unauthorized adhoc changes to nodes and namespaces if it is associated with a YAML repo
• Keep the system synced with repo changes, perhaps with the reflection service

New Table + Registration API

The table to track yaml repositories:

Repository {
  id
  name
  repo_url
  repo_type (git, s3, etc)
  sync_strategy (manual / periodic / webhook)
  owner_team
  reserved_namespaces: [namespace1, namespace2]
}

POST /repositories/{repo_id}/webhook

[samredai]

Isn't this all possible with a pretty standard ci/cd flow? It makes sense to allow clients to add scm metadata and enabling auhorization controls generally also makes sense, but I'm wondering if the server should be cloning down git repos to pull the changes in. I would expect that changes are pushed to the server instead.

[shangyian]

Hmm but a standard ci/cd flow doesn't really cover the prevention of unauthorized changes or track which repos map to which nodes more centrally right?

Or are you just referring to the webhook part, which I agree does not need to be done by the DJ server. However, if it does pull the changes down, we can arguably provide a more centralized view into the state of the sync etc, rather than relying on clients or an unstable ci/cd process

[shangyian]

In thinking about it more - the syncing process may also make sense as something that's managed by the reflection service, which can basically use the same client setup today to reflect changes into the server

[samredai]

Hmm but a standard ci/cd flow doesn't really cover the prevention of unauthorized changes or track which repos map to which nodes more centrally right?

Or are you just referring to the webhook part, which I agree does not need to be done by the DJ server. However, if it does pull the changes down, we can arguably provide a more centralized view into the state of the sync etc, rather than relying on clients or an unstable ci/cd process

I was mostly referring to the webhook part. I agree there's work to be done to get authorization fully functional, but once that's done and users can lock down a namespace/node, then I think their ci/cd process (external to the DJ server) should be where everything else is taken care of, including the flow that keeps the server in sync with their repo.

In thinking about it more - the syncing process may also make sense as something that's managed by the reflection service, which can basically use the same client setup today to reflect changes into the server

Makes sense! This also would likely make enforcing authorization much easier.