Performance issues on Bref PHP 8.1 and Later

[yh1224]

I am experiencing a significant performance issue with the AWS SDK for PHP when running on Bref PHP 8.1 and subsequent versions. Specifically, I have observed that accessing AWS with the SDK is approximately three times slower as compared to Bref PHP 8.0.

To further analyze this, I conducted the same processes on an EC2 instance and found no noticeable performance difference between them. The version of the AWS SDK for PHP I am using is 3.269.9.

Moreover, I conducted a test measuring the time it took to execute 100 queries on DynamoDB. Here are the results:

Environment	Specification	Elapsed
EC2 PHP 8.0.28	c5.large	7.5 sec
EC2 PHP 8.1.18	c5.large	7.5 sec
EC2 PHP 8.2.5	c5.large	7.5 sec
Bref PHP 8.0.28 (php-80:44)	Memory: 1024 MB	4 sec
Bref PHP 8.1.18 (php-81:43)	Memory: 1024 MB	11 sec
Bref PHP 8.2.5 (php-82:33)	Memory: 1024 MB	11 sec

[mnapoli]

Thanks for the report and the details!

The one thing that jumps to mind would be OpenSSL 3, that was applied to PHP 8.1 and 8.2: brefphp/aws-lambda-layers#74

A quick search yield these issues for example:

• Massive performance degradation in OpenSsl 3.0 if used in a heavy multi threaded server application openssl/openssl#17064
• 3.0 Performance problems openssl/openssl#17627

Would you be able to run the same test with Bref 2.0.2 (here are the layer versions: https://runtimes.bref.sh/?region=us-east-1&version=2.0.2)? (this is before OpenSSL 3)

cc @GrahamCampbell if that rings a bell to you?

[yh1224]

@mnapoli

I tried using the Layer compatible with Bref 2.0.2 and there was no degradation in performance as shown below.

Environment	Layer	Elapsed
Bref 2.0.2 / PHP 8.0.28	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-80:42	4 sec
Bref 2.0.2 / PHP 8.1.17	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-81:41	4 sec
Bref 2.0.2 / PHP 8.2.4	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-82:31	4 sec
Bref 2.0.4 / PHP 8.0.28	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-80:44	4 sec
Bref 2.0.4 / PHP 8.1.18	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-81:43	11 sec
Bref 2.0.4 / PHP 8.2.5	arn:aws:lambda:ap-northeast-1:534081306603:layer:php-82:33	11 sec

I would like to use this older version for now. Thank you very much.

[GrahamCampbell]

Interesting, I have not noticed any excess latency on my workloads (though my workloads are super spiky, so it's hard to notice), and I have some that send 1000s of requests to dynamo using the symfony curl http client with a parallelism of 64 (a curl parameter). I am reluctant to suggest that we rollback to openssl 1.1 because of the fact that it will be EOL very soon, leaving us without any security patches. I wonder if we should do a build with openssl 3.1, and see if it also have the same performance issues for @yh1224.

[GrahamCampbell]

Moreover, I conducted a test measuring the time it took to execute 100 queries on DynamoDB.

Are you re-using the client? Can you share a minimal re-producer please?

[GrahamCampbell]

Is the same issue present on arm64 - that is, is this just an issue for x86_64?

[yh1224]

@GrahamCampbell

Are you re-using the client? Can you share a minimal re-producer please?

I create the DynamoDbClient each time.

Is the same issue present on arm64 - that is, is this just an issue for x86_64?

Same with arm64. I feel that it is a little more slower than x86_64.

Environment	Layer	Elapsed
arm64 / Bref 2.0.2 / PHP 8.0.28	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-80:43	4s
arm64 / Bref 2.0.2 / PHP 8.1.17	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-81:22	4s
arm64 / Bref 2.0.2 / PHP 8.2.4	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-82:19	4s
arm64 / Bref 2.0.4 / PHP 8.0.28	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-80:45	4s
arm64 / Bref 2.0.4 / PHP 8.1.18	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-81:24	13s
arm64 / Bref 2.0.4 / PHP 8.2.5	arn:aws:lambda:ap-northeast-1:534081306603:layer:arm-php-82:21	13s

Here is the CDK project to check.
https://github.com/yh1224/aws-cdk-lambda-bref-test

The source is below.
https://github.com/yh1224/aws-cdk-lambda-bref-test/tree/main/src/TestFunc

[GrahamCampbell]

I create the DynamoDbClient each time.

This is probably the issue. You should avoid doing that. I think what's happening here is that the TLS handshake has to happen over and over. That would also explain why I haven't noticed this. Not reusing connections is very expensive.

[GrahamCampbell]

openssl/openssl#17064

I think this issue is unrelated. There is only one thread here.

[mnapoli]

This is probably the issue. You should avoid doing that. I think what's happening here is that the TLS handshake has to happen over and over. That would also explain why I haven't noticed this. Not reusing connections is very expensive.

I would argue that:

• most PHP apps do this, so we have to consider it (it's not an edge case)
• this lets us identify the underlying issue

Fact is that OpenSSL 3 has an issue. Reusing connections is just a workaround, and it's not good for most PHP apps.

[GrahamCampbell]

I strongly disagree that it is a workaround. Setting up a new TLS connection is an expensive thing to do, and application developers should always reuse connections, in the same way that it is standard practice to reuse database connections.

[GrahamCampbell]

Most PHP apps that I have seen do re-use connections. Anyone using the Laravel Guzzle wrapper will have connections re-used. Similarly, anyone using any Laravel native AWS implementations like SES or SQS will also have connections re-used.

[GrahamCampbell]

Not re-using connections and not cleaning up can lead to serious issues like exceeding the max open sockets limit on Lambda (I have done this before ).

[yh1224]

@GrahamCampbell

I understand that this is not an issue with PHP/Bref, but is due to OpenSSL 3.

I'm aware that reusing the client is inefficient, but due to the legacy code of our current production being migrated to Lambda, it seems we can't address it immediately. As the difference in execution time directly affects the cost of using Lambda, I still want to hold out with the old version at this time.

If the performance degradation with OpenSSL is due to some "problem", I would like to wait for its resolution.

Thank you very much.