Skip to content

Conversation

@vibhansa-msft
Copy link
Member

Description

Customer ICM
Duplicate Hdi_isfolder keys in metadata of directories on HNS account.

  • Feature / Bug Fix:
    When azcopy retreives properties from backend, Hdi_isfolder is camel cased by Golang. Later code (in case of account to account copy or sync) reuses the same and we end up adding "Hdi" in camel case while setting the properties on destination. When backend returns back the ListBlob API responce it add "hdi_isfolder" for all directories (all in lower case). This creates a duplicate key in metadata ("hdi" and "Hdi", one set by azcopy during sync and one added by backend during list on dfs account).

  • Related Links:
    ICM

Type of Change

  • Bug fix
  • New feature
  • Documentation update required
  • Code quality improvement
  • Other (describe):

How Has This Been Tested?

This needs manual validation

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug where duplicate hdi_isfolder metadata keys were being created on HNS (Hierarchical Namespace Storage) accounts. The issue occurred because Go canonicalizes the key as "Hdi_isfolder" when retrieving properties, but the backend always returns "hdi_isfolder" in lowercase during ListBlob API responses. During account-to-account copy or sync operations, AzCopy would set the camel-cased version, while the backend would also add the lowercase version, resulting in duplicate keys.

Key Changes:

  • Normalized hdi_isfolder metadata key to always be set in lowercase on the backend
  • Removed conditional logic that could set either "Hdi_isfolder" or "hdi_isfolder"
  • Ensured the camel-cased variant is deleted before setting the lowercase version

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@vibhansa-msft
Copy link
Member Author

az storage blob list --account-name vikasfusename --container-name testcntgo1 --auth-mode login --include m --prefix "abcd"
....
When hdi_isfolder is set by azcopy
"name": "abcd",
"objectReplicationDestinationPolicy": null,
"objectReplicationSourceProperties": [],
"metadata": {
"hdi_isfolder": "true"
},
.....
When Hdi_isfolder is set by azcopy
"name": "abcd",
"metadata": {
"Hdi_isfolder": "true",
"hdi_isfolder": "true"
},
......

@vibhansa-msft vibhansa-msft added this to the 10.32.0 milestone Dec 3, 2025
}
}

// TODO (gapra): figure out better way to deal with hdi_isfolder metadata key capitalization
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should apply this to is_symlink too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While copying symlinks we are creating the metadata with all lower case explicitly:

func (s *blobSymlinkSender) SendSymlink(linkData string) error {
	err := s.getExtraProperties()
	if err != nil {
		return fmt.Errorf("when getting additional folder properties: %w", err)
	}
	s.metadataToApply["is_symlink"] = to.Ptr("true")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of blob to blob copy with preserve-symlink flags here is my output:

   GET https://vikasfuseblob.blob.core.windows.net/testcntgo1/abcd%2Fa.lnk?...
   Accept: application/xml
   User-Agent: AzCopy/10.32.0~preview.1 azsdk-go-azblob/v1.6.2 (go1.25.1; linux)
   X-Ms-Client-Request-Id: 9602ecc6-11c6-468a-6ec4-373ccafae6e3
   x-ms-version: 2025-07-05
   --------------------------------------------------------------------------------
   RESPONSE Status: 200 OK
   ...
   X-Ms-Lease-Status: unlocked
   X-Ms-Meta-Is_symlink: true
   ...
  --------------------------------------------------------------------------------
   PUT https://vikasfusename.blob.core.windows.net/testcntgo1/abcd%2Fa.lnk?...
  ...
   x-ms-blob-type: BlockBlob
   x-ms-meta-is_symlink: true
   x-ms-version: 2025-07-05
  ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants