Go lambda function fails with `GLIBC_2.32' not found

[Bryson14]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v18.14.0

Amplify CLI Version

11.1.1

What operating system are you using?

Windows WSL 2

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Describe the bug

I created a go function, then ran amplify push to compile and create the lambda function. Everything went smoothly until the function was executed by API Gateway and failed with the error:

/var/task/main: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/main)

Expected behavior

The API gateway should return a 200 but instead returns 502 since the lambda function fails.

Reproduction steps

create a api and lambda function with amplify add api.
Create an api with /search path.
create a new go function then replace the contents with 👍

package main

import (
	"context"
	"encoding/json"
	"fmt"

	// "github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/kendra"
)

type QueryRequest struct {
	Search  string `json:"search"`
	IndexID string `json:"indexID"`
}

type QueryResponse struct {
	Results   *kendra.QueryOutput `json:"results`
	Documents []Document          `json:"documents`
	Lawyers   []Lawyer            `json:"lawyers`
}

/
func handler(ctx context.Context, request QueryRequest) (QueryResponse, error) {
	// print out the request
	functionrequest, _ := json.Marshal(request)
	println("Query Request:\n", string(functionrequest))

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))
	svc := kendra.New(sess)
	println("Started the kendra session")

	// 1. Security checks for the requesting user and index
	// Skipped for Now

	// 2. Query the index with the search term

	// Check if the specified index exists
	_, err := svc.DescribeIndexWithContext(ctx, &kendra.DescribeIndexInput{
		Id: &request.IndexID,
	})
	if err != nil {
		return QueryResponse{}, fmt.Errorf("failed to describe index %s: %v", request.IndexID, err)
	}
	println("Index exists")

	input := &kendra.QueryInput{
		QueryText: &request.Search,
		IndexId:   &request.IndexID,
	}

	kendraResp, err := svc.QueryWithContext(ctx, input)
	if err != nil {
		return QueryResponse{}, fmt.Errorf("failed to query index %s: %v", request.IndexID, err)
	}

	KendraJson, _ := json.Marshal(kendraResp)
	println("Query successful:\n", string(KendraJson))

        return QueryResponse {}

• run amplify push

This will cause the error.

However, I also have CICD setup with a github repo with the following build settings:

version: 1
backend:
  phases:
    preBuild:
      commands:
        - yum install golang -y
        - mkdir ~/go_proj
        - export  PATH=$PATH:/usr/local/go/bin
        - export GOPATH=$HOME/go_proj
        - export GOBIN=$GOPATH/bin
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

when I run git push, and the CICD build starts, the error caused by amplify push from my local machine is fixed

Project Identifier

b2a8e45f990b09821ed3bcbcf1838fbc

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey @Bryson14, 👋 thank you for reaching out. Wanted to confirm some information. To understand the push using Amplify CLI was successfully but the call from the API gateway endpoint provided a response with a error message?
Could you check the lambda logs for any additional information on the run?

additionally, I was unable to reproduce the error message this when creating an API with a lambda function and adding the provided code. The lambda function received a validation error but did not experience the error provided.
Could you let us know any additional information that would assist us in reproducing this issue?

[benjamindoe]

@ykethan Can you confirm what local version of GLIBC you are using? I think the problem here is that when you push locally, Amazon Linux does not come with the latest GLIBC version. As such we need to compile with CGO_ENABLED=0. How are we able to add this flag to the amplify build stage?

[ykethan]

Hey @benjamindoe, thank you for the information. For the reproduction I created the Lambda function in my project on MacOS and pushed it to git. Then connected to Amplify hosting which should contain the default linux 2 instance to reproduce.
The local go.mod file contained the version 1.19

in the hosting build settings, i then installed go with the following settings

the backend build did not experience an issue with GLIBC.

additionally, tested using a Amazon linux 2 instance

ldd --version
ldd (GNU libc) 2.26

created a go lambda function and did a push but did not experience a issue.

[benjamindoe]

Hi @ykethan

So the issue is NOT to do with CI CD. The reason your local amplify push worked is because Macos doesn't use GLIBC. The AL2 instance worked because I assume AL1 instances also have GLIBC_2.26. Later GLIBC versions do not work. We need a way of setting CGO_ENABLED=0 during the build stage locally with amplify CLI.

See this issue for lambda go:

aws/aws-lambda-go#340

[ykethan]

Hey @benjamindoe, thank you for the information. noticed Amplify does set this as env variable on windows: https://github.com/aws-amplify/amplify-cli/blob/947b23f1be982d4823df413cbe5595131f101ce7/packages/amplify-go-function-runtime-provider/src/runtime.ts#LL97C1-L97C1

Marking this as bug for further investigation.

[benjamindoe]

Hey @ykethan,

Thanks for the link to the source, really useful. I can see there's no way of actually setting CGO_ENABLED=0 however as it's using execa which does extend process.env meaning we could set CGO_ENABLED=0 globally by running export CGO_ENABLED=0 before running amplify push. Alternatively we could put it in .zshrc/.bashrc.

Although it does detected Windows as you stated, this doesn't work for WSL (which is what @Bryson14 is said to be running) as this is a Ubuntu box under the hood and process.platform reports linux. Personally, I am on Manjaro so my process.plaform reports linux as well.

[ykethan]

Hey @benjamindoe, as you have pointed using a subsystem for linux such as WSL would report the system as linux. The Amplify CLI team has been made aware into the behaviour and will be discussed as part of our triage meetings.
Do let us if setting the CGO_ENABLED=0 globally mitigates the issue as well as I am currently investigating to see using Amazon Linux 2023 would reproduce this behaviour as this does not contain the amazon-extras with golang and then try exporting the CGO_ENABLED=0. If utilizing this does mitigate the issue, we could try utilizing pre-push hook to set the export.

[benjamindoe]

Hi @ykethan, I can confirm export CGO_ENABLED=0 does work. I've added it to a pre-build hook (pre-build-function.sh) until an official solution is found.

# amplify/hooks/pre-build-function.sh

export CGO_ENABLED=0

[Bryson14]

that does appear to work. As stated above, I'm running ubuntu on WSL 2. Running export CGO_ENABLED=0 before running amplify push did fix the issue.

[michaelrios]

I was having this issue intermittently for some reason during the build process (not on amplify push) and added the following to the amplify.yml, which fixed the problem

backend:
  phases:
    preBuild:
      commands:
        - export CGO_ENABLED=0

[gjae]

Hi @ykethan, I can confirm export CGO_ENABLED=0 does work. I've added it to a pre-build hook (pre-build-function.sh) until an official solution is found.

# amplify/hooks/pre-build-function.sh

export CGO_ENABLED=0

that works for me! :)