Support for Microsoft Entra authentication

[jeremydelacruz]

My team is interested in the synapse.ml.cognitive package to write from a data frame to an AI Search index. We have been able to use the writeToAzureSearch function by providing an API key (as subscriptionKey). However, we have security policies in our environment requiring us to disable key-based authentication on AI Search and leverage Microsoft Entra authentication. We also have additional security policies that prevent us from being able to use a service principal. As a result, we need to use managed identity for authentication when connecting to AI Search from our notebook.

Looking through the documentation and the source code (e.g., AzureSearch.scala), it seems there is only support for Search API key authentication and no support for providing a DefaultAzureCredential to authenticate via a managed identity. This is something that the official Python Azure AI Search client library supports.

Is this accurate, or am I missing something?

If it is not currently supported, can this be raised as a feature request to also support managed identity for authentication to AI Search when using the synapse.ml.cognitive package?