[ISSUE] Cannot delete token: unable to parse response

[DiogoGeada]

We want to provision a service principal from databricks account into a databricks workspace and then generate a personal access token for it. However, when we try to either create or destroy the token, we face this error.

The error is the following:
cannot delete token: unable to parse response. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

Reproduction

provider "databricks" {
  host                        = # [workspace_url]
  azure_workspace_resource_id = # [workspace_id]
  azure_client_id             = # service_principal_id
  azure_client_secret         = # service principal secret
  alias                       = "data"
}

data "databricks_service_principal" "data_collection_sp" {
  application_id = #[application_id]
  provider = databricks.account # databricks account provider
}

resource "databricks_mws_permission_assignment" "provision_sp" {
  workspace_id = # [workspace_id]
  principal_id = data.databricks_service_principal.data_collection_sp.id
  permissions  = ["ADMIN"]

  provider = databricks.account

  depends_on = [ module.dbw_identity ]
}

module "data_collection_sp_pat" {
  source = # [workspace_personal_access_token_module]
  kv = # [kv_id]
  dbw = module.dbw.dbw
  team_config = module.team_config
  sp = {
    id = # [service_principal_id]
    short_name = "data"
  }

  providers = {
    databricks.sp = databricks.data
  }
}

### Module ###

terraform {
  required_providers {
    databricks = {
      source                = "databricks/databricks"
      configuration_aliases = [databricks.sp]
    }
  }
}

resource "databricks_token" "sp_token" {
  comment  = "${upper(var.sp.short_name)} SP's personal access token. Managed by Terraform"
  provider = databricks.sp
}

Expected behavior
A token should be created and destroyed without any issues.

Is it a regression?
Token creation module has worked for the past year or so. We only faced issues last week, with provider 1.53. It was upgraded to 1.63 but we still had the same issue.

Debug Logs

Error: cannot delete token: unable to parse response. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:
│ ```
│ POST /api/2.0/token/delete
│ > * Host: 
│ > * Accept: application/json
│ > * Authorization: REDACTED
│ > * Content-Type: application/json
│ > * Traceparent: 00-796af4854d6938687ab24fd96be9f89f-b0c38516a44e712d-01
│ > * User-Agent: databricks-tf-provider/1.63.0 databricks-sdk-go/0.55.0 go/1.22.10 os/linux terraform/1.9.2 sdk/sdkv2 resource/token auth/azure-client-secret
│ > * X-Databricks-Azure-Sp-Management-Token: [token]
│ > * X-Databricks-Azure-Workspace-Resource-Id: /subscriptions/[subscription_id]/resourceGroups/[workspace_name]/providers/Microsoft.Databricks/workspaces/[workspace_name]
│ > {
│ >   "token_id": "813518eb183f08ddd15ebde42eb9d9e97a1690ad0a6ce277cd5a98243c557c74"
│ > }
│ < HTTP/2.0 403 Forbidden
│ < * Content-Length: 20
│ < * Content-Type: text/html; charset=utf-8
│ < * Date: Mon, 20 Jan 2025 10:31:05 GMT
│ < * Server: databricks
│ < * X-Request-Id: d5a998eb-f4fd-4d14-b5de-d02411f76df6
│ < User not authorized.
│ ```

Other Information

• OS: Mac, Ubuntu

[DiogoGeada]

Seems like the unauthorized error was a dependency issue, but the provider still has trouble parsing the response from the sdk.