Production and Polaris credentials

[lefebsy]

Hello,

I've read many pages of Polaris documentation and code, but still not sure to uderstand how in Production, Polaris itself should be configured to retrieve his own credentials (AWS by example) to be allowed to ask token to STS for a catalog subScope request...

• Principals seems related to internal catalogs security model, not Polaris itself

• In Polaris Helm chart there is an extraEnv section where the classics AWS_ACCESS_KEY_ID could be retrieved

#- extraEnv:
#  - name: AWS_STORAGE_BUCKET
#    value: s3://xxxxx/
#  - name: AWS_ACCESS_KEY_ID
#    valueFrom:
#      secretKeyRef:
#        name: aws-secret
#        key: access_key_id
#  - name: AWS_SECRET_ACCESS_KEY
#    valueFrom:
#      secretKeyRef:
#        name: aws-secret
#        key: secret_access_key

• in documentation and "polaris-server.yml" there is settings related to oauth2 or authenticator, but is it for catalogs inside Polaris or Polaris deployement itself ? Not clear... seems more related to catalogs than Polaris

• in AZURE and GCP core/storage credentials classes, there is comments about using default Env Var credentials to communicate with the service related to subscop tokens

• in storageConfiguration implementations there is no secrets stored for AWS, GCP or AZURE (good, because debug logs show the properties, and the REST GetCatalog response also)

Is it obvious and I miss the point ?

• Polaris have no dedicated setting "file" or "envVar" for his own access to cloud environnement ?
• If the credentials are globals to the Polaris instance, all the catalogs inside will inherit of them (and limited by) to do some actions ?

Thank you

[MonkeyCanCode]

@lefebsy so the AWS credential example above is to show how you can pass in your AWS credential for Polaris service to use to communicate with AWS. Polaris is using AWS SDK which will loop through a set of auth methods and reading off from env variables if one of them. You can avoid this with annotations for K8S SA with IAM role/policy.

[lefebsy]

Thank you @MonkeyCanCode

It is clear. I think. I will try below to say it with my words, just to avoid misunderstanding.

So Polaris service credentials for AWS, GCP, and AZURE are coming from environment variables, or in case of Kubernetes deployment, serviceAcount linked to cloud provider IAM hosting Polaris is a possible alternative.

Maybe it should be mentioned in a chapter in documentation : configuring-polaris-for-production.md ?

Is it possible to draw a scenario to configure (a self hosted) Polaris service, with credentials provided to allow creation of catalogs working with more than one cloud storage ?

It could help users/ops to do distinction between the deployment requirements and Polaris catalogs usage/configuration...

By example something like this :

• How to configure a Polaris service hosted in kubernetes somewhere (AWS or self hosted) with credentials variables required to manage catalogs with storage configuration in S3:// AWS & GS:// GCP and/or Azure...
• In extension, if the deployment is done through the kubernetes service account link to cloud provider IAM hosting Polaris service, the catalogs targeting other clouds will require credentials sourced from additional environment variables

[MonkeyCanCode]

Anytime @lefebsy. See my response below:

"Maybe it should be mentioned in a chapter in documentation : configuring-polaris-for-production.md ?" -> I think this will apply for non-prod as well as Polaris only support Azure, GCP, AWS, and File (local) and nothing for self-managed object storage such as minio. But feel free to raise a PR if this is not clear for documentation update.

"Is it possible to draw a scenario to configure (a self hosted) Polaris service, with credentials provided to allow creation of catalogs working with more than one cloud storage ?" -> I am assuming people may have use cases to use different cloud providers for object storage as diff backends (DR purpose or data segregation etc.). The storage layer is tie to individual catalog object within Polaris and there can be more than one catalog within a Polaris service.

"How to configure a Polaris service hosted in kubernetes somewhere (AWS or self hosted) with credentials variables required to manage catalogs with storage configuration in S3:// AWS & GS:// GCP and/or Azure..." -> In terms of deployment, I only used AWS and File (for local testing etc.) so far. But yeah, it may be good to add how to connect to other clouds provider storage if not already documented. Unfortunally, I don't have much experience with other cloud providers. The auth to other cloud providers may be similar as those are done via SDK from various providers (e.g. AWS SDK).

"In extension, if the deployment is done through the kubernetes service account link to cloud provider IAM hosting Polaris service, the catalogs targeting other clouds will require credentials sourced from additional environment variables" -> that is correct if that is what other cloud provider supported. For AWS, it is preferred to use annotation as oppose to env variables then assume role and STS.

[lefebsy]

Thank you @MonkeyCanCode for the quick response.

You are right, many informations are not limited to Production and are ok for lower environments deployment too.

I agree to raise a PR about doc, but I need first to be sure that information is correct :-)

• About the kubernetes service account and your explanation, it seems it is clearly specific to AWS implementation. To be able to link with annotations external stuff like AWS IAM... I don't know if GCP or Azure did similar implementation in their kubernetes distribution. In vanilla kubernetes, service account scope is limited to kubernetes itself and image registries credentials, nothing about external object storage or external IAM to add privileges related to object storage outside kubernetes.
  So if Polaris want to be easily deployed on many cloud providers it could may be safer to not rely on Kube Service Account, otherwise documentation should be enriched with specific flavor and customization like this AWS annotations... and maybe extended with specific examples for more than one cloud provider...

• Polaris service requirements in terms of privileges seems happily limited to read and write to the object storage type defined in catalog, and access to the subScope token delivery service.
  Even kubernetes service account did not add specific kubernetes privileges for Polaris, in Helm chart I've seen classic HPA so no pod creation privilege required...

• Polaris catalogs are today limited to 3 object storage provider, but Polaris service itself could be deployed elsewhere with network access to those storages. It seems not limited to 3 cloud providers.
  I am digging in these questions also because I've raised a PR (under review, reviewed once by Eric-Maynard) adding S3 compatible storage (MinIO, Dell ECS, NetApp StorageGRID, Scality, etc..). In this implementation the credentials are declared at catalog level, not "shadow" inherited from Polaris service, even if at the end variables have still to be available in service running context.
  So now I am not sure what is best approach about this: keep it at catalog level, or be aligned with 3 others storage type and relying to correct but almost hidden setup of Polaris service credentials.

Last stuff, if I have correctly understood current AWS implementation: I need that Polaris service have in only one access-secret key pair, all the habilitations and roles to all the buckets of all the AWS S3 catalogs ? No segregation possible. All privileges are inherited from this unique (default named to be found by AWS sdk) AWS S3 Access Key... Like... All eggs in one basket ?
In case of revoke the impact can be huge and not limited to 1 catalog...

[MonkeyCanCode]

"About the kubernetes service account and your explanation, it seems it is clearly specific to AWS implementation. To be able to link with annotations external stuff like AWS IAM... I don't know if GCP or Azure did similar implementation in their kubernetes distribution. In vanilla kubernetes, service account scope is limited to kubernetes itself and image registries credentials, nothing about external object storage or external IAM to add privileges related to object storage outside kubernetes." -> not really right as the k8s manifest doesn't have anything to do with AWS. The support to auth with AWS can be done via different ways (such as env or assume role via annotation and SA). Quick checks shows Aazure supports something similar: https://learn.microsoft.com/en-us/samples/azure-samples/azure-ad-workload-identity-mi/azure-ad-workload-identity-mi/). The intension for the chart is to be generic and shows the support of what is being present in the demo notebook (which is using AWS). Feel free to PR the examples for other cloud provides via docker compose or helm.

"Polaris service requirements in terms of privileges seems happily limited to read and write to the object storage type defined in catalog, and access to the subScope token delivery service. Even kubernetes service account did not add specific kubernetes privileges for Polaris, in Helm chart I've seen classic HPA so no pod creation privilege required" -> so access for Polaris to cloud provider will be controlled by the auth info you put (take AWS with assume role as an example, i can put a generic annotation on the pod which maps to the IAM roles with custom IAM policies. thus, to change the privileges, only changes will be needed on IAM policies side instead of Polaris...also, if the object storage is using custom encryption key, then additional access to KMS will be needed as well).

"Polaris catalogs are today limited to 3 object storage provider, but Polaris service itself could be deployed elsewhere with network access to those storages. It seems not limited to 3 cloud providers." -> yes, the service itself can be anywhere (local, lab, public/private clouds, etc.). Currently the only 4 official supported storage backends are AWS, GCP, Azure, and File (local...not for prod). It may work with other storage backend (e.g. Minio) but I haven't test those. There are couple threads around this.

"Last stuff, if I have correctly understood current AWS implementation: I need that Polaris service have in only one access-secret key pair, all the habilitations and roles to all the buckets of all the AWS S3 catalogs ? No segregation possible. All privileges are inherited from this unique (default named to be found by AWS sdk) AWS S3 Access Key... Like... All eggs in one basket ?
In case of revoke the impact can be huge and not limited to 1 catalog..." -> as far as I know, that is the case, the credential used by polaris to cloud provider (e.g. AWS) only allowed one (e.g. env variable for AWS client key and client secret). There is no way to said catalog A in Polaris service will be using auth 1 and catalog B in Polaris service will be using auth 2. However, to enforce segregation, there are a couple of ways for you to do so such as one department per catalog and map to one s3 path (s3 path can be from different account as well...you will just need to do cross account assume role in AWS). Then for each of these catalogs, you can create diff principle (users in Polaris) to enforce additional access control. By doing so, your AWS credential used by Polaris service will have access to all the allowed S3 buckets from a set of AWS accounts and the end-users can use the principals created for their departments to ensure they don't access other people's data as well. This is one way to enforce segregation. Alternative, if u want complete segregation, you will then need to create multiple polaris deployment and each with their own cloud provider auth info.