The functionality you requested is not available in this region.

[ELDiablO59152]

Hi,
I am trying to import an image onto AWS but get this error on eu-west-3 region.
Is it correctly passed as argument during import phase ?

$ /usr/local/bin/bsdec2-image-upload --uefi-preferred --sriov --ena  /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.raw "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS "  "FreeBSD/amd64 hardened/13-stable/master@87e860b522f7"  eu-west-3 <bucket_name> /aws.key

Uploading /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.raw to
http://<bucket_name>.s3.amazonaws.com/f97fc57f7f37571b3cd0f8afed00f6e1/
in 1437 part(s)............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. done.
Uploading volume manifest... done.
bsdec2-image-upload: EC2 API request failed:
HTTP/1.1 400 Bad Request
x-amzn-RequestId: <request_id>
Cache-Control: no-cache, no-store
Strict-Transport-Security: max-age=31536000; includeSubDomains
vary: accept-encoding
Content-Type: text/xml;charset=UTF-8
Date: Thu, 07 Nov 2024 16:13:57 GMT
Connection: close
Server: AmazonEC2

<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>UnsupportedOperation</Code><Message>The functionality you requested is not available in this region.</Message></Error></Errors><RequestID><request_id></RequestID></Response>

bsdec2-image-upload: Failure importing disk image

[cperciva]

The "raw" image import mechanism isn't supported in all regions. Try creating a .vhd file and using the --vhd option.

[ELDiablO59152]

Thank you very much for your fast answer, I really appreciate.

I tried on eu-west-2 and it brings some improvements but doesn't work either.

$ /usr/local/bin/bsdec2-image-upload --uefi-preferred --sriov --ena /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.raw "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS"  "FreeBSD/amd64 hardened/13-stable/master@87e860b522f7" eu-west-2 <bucket_name2> /aws.key

Uploading /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.raw to
http://<bucket_name2>.s3.amazonaws.com/dde8ff88f816ebb532764eafef4294a0/
in 1437 part(s)............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. done.
Uploading volume manifest... done.
Importing volume: Pending: 5%....................................................................................
Importing volume: CLIENT_ERROR : RequestId: <request_id> Error: Task timed out after 900.00 seconds.........................^C

$ aws ec2 describe-conversion-tasks showed a BytesConverted value of 0....

{
    "ConversionTasks": [
        {
            "ConversionTaskId": "import-vol-729b2a1edf93dfb7t",
            "ExpirationTime": "2024-12-09T17:41:17Z",
            "ImportVolume": {
                "AvailabilityZone": "eu-west-2a",
                "BytesConverted": 0,
                "Image": {
                    "Format": "RAW",
                    "ImportManifestUrl": "https://<bucket_name2>.s3.eu-west-2.amazonaws.com/d9a0f4cc6c1348218eb3d1d076f6dc61/manifest.xml?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<creds>&X-Amz-Date=20241202T174116Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature>",
                    "Size": 15066667008
                },
                "Volume": {
                    "Size": 15
                }
            },
            "State": "cancelled",
            "StatusMessage": "CLIENT_ERROR : RequestId: <request_id> Error: Task timed out after 900.00 seconds"
        }
    ]
}

I also tried with --vhd as you suggested without success.

$ /usr/local/bin/bsdec2-image-upload --uefi-preferred --sriov --ena --vhd /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.vhd "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS"  "FreeBSD/amd64 hardened/13-stable/master@87e860b522f7" eu-west-2 <bucket_name2> /aws.key

Uploading /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.vhd to
http://<bucket_name2>.s3.amazonaws.com/5922481baecc4d2189560c21a0a3056e/snap.vhd
bsdec2-image-upload: SSL request failed: Could not write payload: Broken pipe
S3 PUT failed 1 times: /5922481baecc4d2189560c21a0a3056e/snap.vhd
bsdec2-image-upload: SSL request failed: Could not write payload: Connection reset by peer
S3 PUT failed 2 times: /5922481baecc4d2189560c21a0a3056e/snap.vhd
[...]
bsdec2-image-upload: SSL request failed: Could not write payload: Broken pipe
S3 PUT failed 10 times: /5922481baecc4d2189560c21a0a3056e/snap.vhd
bsdec2-image-upload: Failed to upload disk image

Is there something missing in my policies ?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeImages",
                "ec2:DeleteVolume",
                "ec2:ImportVolume",
                "ec2:CreateTags",
                "ec2:RegisterImage",
                "ec2:CreateSnapshot",
                "ec2:ImportSnapshot",
                "ec2:DescribeImportSnapshotTasks",
                "ec2:CopyImage",
                "ec2:ModifyImageAttribute",
                "ec2:DescribeSnapshots",
                "ec2:DescribeConversionTasks",
                "ec2:DescribeImportImageTasks",
                "ec2:DescribeRegions"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>/*",
                "arn:aws:s3:::<bucket_name2>/*"
            ]
        }
    ]
}

[cperciva]

Is there something missing in my policies ?

The policy looks fine. Did you create the 'vmimport' role?

bsdec2-image-upload/SETUP

Line 29 in 6933b15

1. Create a 'vmimport' role which can be assumed by the AWS VM Import/Export

[ELDiablO59152]

Yes I have, please take a look and tell me what seams incorrect:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "vmie.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:Externalid": "vmimport"
                }
            }
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:CopySnapshot",
                "ec2:Describe*",
                "ec2:RegisterImage",
                "ec2:ModifySnapshotAttribute"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name2>"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>/*",
                "arn:aws:s3:::<bucket_name2>/*"
            ]
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:iam::<account_id>:role/vmimport"
            ]
        }
    ]
}

[ELDiablO59152]

It is working barely with the CLI:

$ aws ec2 import-image --region eu-west-3 --description "VultureOS13import" --disk-containers '[{                                                                                       
    "Description": "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS", 
    "Format": "vhd", 
    "UserBucket": { 
        "S3Bucket": "<bucket_name>",
        "S3Key": "Vulture-dev-13-stable-open.vhd"
    }
}]'

But it fails at the end because BSD is not supported ?

{
    "ImportImageTasks": [
        {
            "Description": "VultureOS13import",
            "ImageId": "",
            "ImportTaskId": "<task_id>",
            "SnapshotDetails": [
                {
                    "DiskImageSize": 7429957120.0,
                    "Format": "VHD",
                    "Status": "completed",
                    "UserBucket": {
                        "S3Bucket": "<bucket_name>",
                        "S3Key": "Vulture-dev-13-stable-open.vhd"
                    }
                }
            ],
            "Status": "cancelled",
            "StatusMessage": "CLIENT_ERROR : ClientError: Unknown OS / Missing OS files.",
            "Tags": []
        }
    ]
}

[cperciva]

"Import Image" only works for operating systems the AWS code knows about. My tool doesn't use that; it uses "Import Snapshot" and then registers an AMI from that.

[ELDiablO59152]

Alright, thank you so much for all your informations !

I finally got it working with aws-cli:

aws ec2 import-snapshot --region eu-west-3 --description "VultureOS13import" --disk-container '{
    "Description": "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS",
    "Format": "vhd",
    "UserBucket": {
        "S3Bucket": "<bucket_name>",
        "S3Key": "Vulture-dev-13-stable-open.vhd"
    }
}'

But it doesn't work with bsdec2-image-upload sadly:

$ /usr/local/bin/bsdec2-image-upload --uefi-preferred --sriov --ena --vhd /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.vhd "FreeBSD 13.4-STABLE-HBSD-amd64 UEFI-PREFERRED CLOUD-INIT ZFS"  "FreeBSD/amd64 hardened/13-stable/master@87e860b522f7" eu-west-3 <bucket_name> /aws.key

Uploading /poudriere/releases/amd64/dev/13-stable/BUILD-LATEST/Vulture-dev-13-stable-open.vhd to
http://<bucket_name>.s3.amazonaws.com/5922481baecc4d2189560c21a0a3056e/snap.vhd
bsdec2-image-upload: SSL request failed: Could not write payload: Broken pipe
S3 PUT failed 1 times: /7d505314a717f68899bacd7d76e1d60f/snap.vhd
[...]
bsdec2-image-upload: SSL request failed: Could not write payload: Broken pipe
S3 PUT failed 10 times: /7d505314a717f68899bacd7d76e1d60f/snap.vhd
bsdec2-image-upload: Failed to upload disk image

Is there a debug mode ?

[cperciva]

No debug mode, but you're getting a failure from S3 which makes it easier.

I assume <bucket name> is a placeholder and that's not the problem.

How large is the .vhd file? It may be that we're running into S3 upload size limits and I need to rewrite the code to do a multipart upload.

[ELDiablO59152]

You found the point, good catch !
Around 7.5Go for my vhd and vmdk.

Upload an object in a single operation by using the AWS SDKs, REST API, or AWS CLI – With a single PUT operation, you can upload a single object up to 5 GB in size.

Upload a single object by using the Amazon S3 console – With the Amazon S3 console, you can upload a single object up to 160 GB in size.

Upload an object in parts by using the AWS SDKs, REST API, or AWS CLI – Using the multipart upload API operation, you can upload a single large object, up to 5 TB in size.

The multipart upload API operation is designed to improve the upload experience for larger objects. You can upload an object in parts. These object parts can be uploaded independently, in any order, and in parallel. You can use a multipart upload for objects from 5 MB to 5 TB in size. For more information, see Uploading and copying objects using multipart upload in Amazon S3.

source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

[cperciva]

Ok, I'm afraid we're stuck until I get around to rewriting that code to do multipart uploads. Sorry...

[cperciva]

I guess for now there is still the workaround of "upload to a region which supports raw image imports and then copy to the region you care about". It's not ideal, but it should work.