[StorageBrowser] integrating existing S3 resources issue

[calebpollman]

Hi, I'm having a bit of trouble to make it work with existing Cognito and S3 resources.
I've referenced the resources on backend configuration (code below), the bucket also has CORS configured according to the documentation: https://ui.docs.amplify.aws/react/connected-components/storage/storage-browser#bucket-cors

And I also set bucket policies to allow access from the Coginito Authenticated role to access and list the bucket.
However, I'm getting on display that there is "No folders or files"
I was able to make it work, if I edit amplify_outputs.json directly and add path configuration access to " * / * "

backend.addOutput({
  auth: {
    aws_region: BACKEND_CONFIG.REGION,
    user_pool_id: BACKEND_CONFIG.COGNITO_USER_POOL_ID,
    user_pool_client_id: BACKEND_CONFIG.COGNITO_USER_POOL_CLIENT_ID,
    identity_pool_id: BACKEND_CONFIG.COGNITO_IDENTITY_POOL_ID,
    username_attributes: ['username'],
    user_verification_types: ['email'],
    mfa_configuration: 'OPTIONAL',
    unauthenticated_identities_enabled: true,
    standard_required_attributes: ['email', 'name'],
    password_policy: {...}
  },
  storage: {
    aws_region: BACKEND_CONFIG.REGION,
    bucket_name: BACKEND_CONFIG.BUCKET
  }
});

Here is my bucket policy

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"<ARN-COGNITO-ROLE-AUTHENTICATED>"
				]
			},
			"Action": [
				"s3:GetObject",
				"s3:PutObject",
				"s3:DeleteObject"
			],
			"Resource": "arn:aws:s3:::<BUCKET-NAME>/*"
		},
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"<ARN-COGNITO-ROLE-AUTHENTICATED>"
				]
			},
			"Action": "s3:ListBucket",
			"Resource": "arn:aws:s3:::<BUCKET-NAME>/*"
		}
	]
}

Originally posted by @alanst32 in #5731 (comment)

[AllanZhengYP]

Regarding this issue of supporting existing S3 resources, I can confirm StoBro supports listing from bucket root.

So I think to support S3 bucket's that's not created by Amplify, but still configured to be accessed by Cognito identity,
the only gap I can see is to have backend.addOutput to support /* path.

[nookale]

Hi, any news? I am also unable to use Amplify Storage with any S3 bucket (https://docs.amplify.aws/react/build-a-backend/storage/use-with-custom-s3/#use-storage-resources-with-an-amplify-backend)

Thanks!

[calebpollman]

Hi @nookale! Can you provide some additional information on how you've configured the StorageBrowser?

[itconsultor]

Hello @calebpollman can you please share how you configured this part: ".....I was able to make it work, if I edit amplify_outputs.json directly and add path configuration access to " * / * "....." Since I was checking the amplify_outputs.json file, but I don't see the "path Configuration" section there.

Thanks advance!

Carlos

[nookale]

Hi @nookale! Can you provide some additional information on how you've configured the StorageBrowser?

From this page: https://docs.amplify.aws/react/build-a-backend/storage/use-with-custom-s3/

I tried the part: Use storage resources with an Amplify backend.

Regarding the bucket, besides the one specified in the article I mentioned above, I also used:

• Bucket Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<s3-bucket-name>",
                "arn:aws:s3:::<s3-bucket-name>/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-number>:role/<amplify-role-associated-with-the-project>"
            },
            "Action": [
                "s3:PutBucketPolicy",
                "s3:GetBucket*",
                "s3:List*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::<s3-bucket-name>",
                "arn:aws:s3:::<s3-bucket-name>/*"
            ]
        }
    ]
}

• CORS:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "x-amz-server-side-encryption",
            "x-amz-request-id",
            "x-amz-id-2",
            "ETag"
        ],
        "MaxAgeSeconds": 3000
    }
]

Which is the configuration that is automatically generated when a bucket is created as an Amplified resource, so I copied it to the "non-amplify" bucket I wanted to use with Storage. However, I tried many different combinations and modifications for both the bucket policy and CORS. None of them worked.

Also, regarding the "Principal," it was unclear which one to use, so I tried all the roles associated with the amplify project: CustomS3AutoDeleteObjects, AmplifyBranchLinkerCustom, amplifyAuthunauthenticate, amplifyAuthauthenticatedU.

For CORS, I have also used this (https://ui.docs.amplify.aws/react/connected-components/storage/storage-browser):

[
  {
    "ID": "S3CORSRuleId1",
    "AllowedHeaders": [
      "*"
    ],
    "AllowedMethods": [
      "GET",
      "HEAD",
      "PUT",
      "POST",
      "DELETE"
    ],
    "AllowedOrigins": [
      "*"
    ],
    "ExposeHeaders": [
      "last-modified",
      "content-type",
      "content-length",
      "etag",
      "x-amz-version-id",
      "x-amz-request-id",
      "x-amz-id-2",
      "x-amz-cf-id",
      "x-amz-storage-class",
      "date",
      "access-control-expose-headers"
    ],
    "MaxAgeSeconds": 3000
  }
]

Regarding the Next Js app, I followed the instructions that can be found here: https://docs.amplify.aws/react/build-a-backend/storage/use-with-custom-s3/

[nookale]

By the way, just for the records and in case it may be of interest:

When the bucket is created as an Amplify Resource, the storage part of the autogenerated amplify_outputs.json file looks like:

  "storage": {
    "aws_region": "<region>",
    "bucket_name": "<bucket-name>",
    "buckets": [
      {
        "name": "<your-additional-bucket-friendly-name>",
        "bucket_name": "<bucket-name>",
        "aws_region": "<region>"
      }
    ]
  }

However, when adding a custom S3 bucket as a resource, this is how it looks like:

  "storage": {
    "aws_region": "<region>",
    "bucket_name": "<bucket-name>"
  }

[sephilink]

Hi :)
Any news on this one ? it seems it should be possible as per the documentation but I couldn't make it work properly, especially while adding groups control

[AllanZhengYP]

@sephilink

Have you tried adding the paths to your configuration(these is a bug with this approach as you can see, and we are fixing it right now):

backend.addOutput({
  storage: {
    bucket_name: 'my-default-bucket',
    aws_region: 'my-default-bucket-region',
    buckets: [{
      name: 'default-bucket-friendly-name',
      aws_region: 'my-default-bucket-region',
      // @ts-expect-error
      paths: {
        'my-root-folder/*': {
          guest: ['get', 'list'] as const,
          authenticated: ['list', 'get', 'delete', 'write'] as const,
        }
      }
    }]
  }
});

You need to use paths to let the StorageBrowser know what to render in the locations view(1st screen). The path name should match your cognito permission.

[tiffanynwyeung]

Hello @sephilink, @nookale, @itconsultor et al, we've updated our documentation on integrating existing S3 buckets with Amplify, including several examples on how to configure your bucket with the Amplify backend. Please let us know if you're still encountering issues with setting up the bucket, thanks!