CURL - massive Performance issue

[aeberhard]

Description

The following code:

<?php

$starttime = microtime(true);

$ch = curl_init();
if (0 === curl_errno($ch)) {
    $header = [];
    $header[] = 'Accept: application/vnd.github+json';
    $header[] = 'X-GitHub-Api-Version:2022-11-28';

    $options = [
        CURLOPT_URL => 'https://api.github.com/rate_limit',
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_USERAGENT => 'PHP',
        //CURLOPT_CONNECTTIMEOUT => 0,
        //CURLOPT_TIMEOUT => 5,
        //CURLOPT_HTTPHEADER => $header,
        //CURLOPT_IPRESOLVE => CURL_IPRESOLVE_V4,
        //CURLOPT_ENCODING => '',
        //CURLOPT_DNS_USE_GLOBAL_CACHE => false,
    ];

    $log = fopen(__DIR__. '/curl.log', 'w');
    if (false !== $log) {
        $logoptions = [
            CURLOPT_VERBOSE => true,
            CURLOPT_STDERR => $log,
        ];
    }

    $curloptions = $options + $logoptions;
    curl_setopt_array($ch, $curloptions);

    $response = curl_exec($ch);
    $info = curl_getinfo($ch);

    $httpCode = (string) curl_getinfo($ch, CURLINFO_HTTP_CODE);
    echo '<pre>';
    var_dump([$httpCode, $response, $info]);
    echo '</pre>';
}


$duration = microtime(true) - $starttime;
echo "Runtime: $duration Sek.";

As you can see I tried with several Curl-Options

Resulted in this output:

Runtime: 0.76369309425354 Sek.
Runtime: 0.6199951171875 Sek.
Runtime: 0.6086049079895 Sek.
Runtime: 0.56775498390198 Sek.
Runtime: 0.54858207702637 Sek.

Performance of the Script in other PHP Versions on same Host:

PHP 7.4
Runtime: 0.11398887634277 Sek.
Runtime: 0.22311997413635 Sek.
Runtime: 0.1849901676178 Sek.
Runtime: 0.12297105789185 Sek.
Runtime: 0.16944289207458 Sek.

PHP 8.0
Runtime: 0.1490170955658 Sek.
Runtime: 0.11332702636719 Sek.
Runtime: 0.16415405273438 Sek.
Runtime: 0.19613790512085 Sek.
Runtime: 0.14559698104858 Sek.

PHP 8.1
Runtime: 0.13851690292358 Sek.
Runtime: 0.11393594741821 Sek.
Runtime: 0.10096192359924 Sek.
Runtime: 0.14624786376953 Sek
Runtime: 0.11632513999939 Sek.

In PHP 8.2 the the Curl pretransfer_time is up to 5 times bigger than in other versions.

PHP 8.0
    ["total_time"]=>
    float(0.127489)
    ["namelookup_time"]=>
    float(0.005659)
    ["connect_time"]=>
    float(0.058811)
    ["pretransfer_time"]=>
    float(0.105881)

PHP 8.2
    ["total_time"]=>
    float(0.522058)
    ["namelookup_time"]=>
    float(0.006427)
    ["connect_time"]=>
    float(0.036172)
    ["pretransfer_time"]=>
    float(0.5042)

PHP Version

PHP 8.2.0

Operating System

Windows 10 Enterprise 21H2

[aeberhard]

Curl Version 7.85.0

PHP Versions downloaded from https://windows.php.net/download/

[cmb69]

In PHP 8.2 the the Curl pretransfer_time is up to 5 times bigger than in other versions.

Indeed! It doesn't seem to be related to HTTP/2, since adding CURLOPT_SSL_ENABLE_ALPN => false shows the same performance degradation. Given that apparently the appconnect_time has degredated (please confirm or deny), this is likely an issue with OpenSSL 3 (which is used on Windows as of PHP 8.2.0), so this might not be Windows specific.

[aeberhard]

I confirm!
appconnect_time_us on PHP 8.2 is up to 5 times bigger!

PHP 8.0
    ["appconnect_time_us"]=>
    int(73811)
    ["connect_time_us"]=>
    int(33838)
    ["namelookup_time_us"]=>
    int(1862)
    ["pretransfer_time_us"]=>
    int(73967)

PHP 8.2
    ["appconnect_time_us"]=>
    int(369953)
    ["connect_time_us"]=>
    int(43670)
    ["namelookup_time_us"]=>
    int(16192)
    ["pretransfer_time_us"]=>
    int(370145)

[mwcng]

I experienced the same issue with curl requests becoming extremely sluggish after updating some of the software I was using, but after some testing it doesn't seem to be related to OpenSSL, PHP 8.2 or Windows.

I'm running Ubuntu 22.04 and I tried multiple OpenSSL versions: 3.0.5, 3.0.7 and the master branch version from today (3.2.0-dev), tried multiple PHP versions (8.1.9, 8.1.14 and 8.2.1) but no matter what I tried I couldn't seem to get curl to work fast like it did before with any of them. The problem ended up being curl itself. While updating my system I also installed the latest version of curl (7.87.0) and after reverting back to the previous version I had (7.84.0) curl performance in php went back to normal.

The issue was definitely related to https requests (although I didn't bother checking if it was http/1.1 or http/2 specific) considering that regular http requests were working fine even with the newer curl version. Considering that your problem started with curl 7.85.0 I guess we can assume that the problem starts from that version.

Right now I'm using PHP 8.2.1, OpenSSL 3.0.7 and cURL 7.84.0 and everything is ok, so for the time being if anyone is experiencing these problems you should probably just stick to using cURL 7.84.0 and update at a later point when this issue gets resolved.

[cmb69]

I just tried PHP 8.2 with libcurl 7.84.0 (Windows), and can see the same performance regression as with libcurl 7.85.0.

[mwcng]

I guess I assumed a bit too much :) I suppose the problem is a bit different on Windows but the performance regression is definitely not limited to Windows only.

[exactmultiple3425]

We experienced the same problem when using the official Docker image and upgrading from php:7.4-fpm to php:8.1-fpm. It was discovered that the problem started with a version of curl between 7.74 and 7.88. Downgrading to use the bullseye image with php:8.1 solved the issue.

[repli2dev]

We stumbled upon the same issue when migrating from PHP 8 to PHP 8.2 (using alpine official PHP images...) the version 8.2.8 (with curl 8.12) was OK, yet the issue seems to occasionally go disappear and reappear:

• 8.0.17 (uses alpine 3.15 and curl 7.80) does not have this issue.
• 8.0.8 (uses alpine 3.14 and curl 7.77) have this issue.

So probably just fiddle with the PHP + CURL versions...

Anyway it is testable using the above snippet and official docker images.

[mike-zukowski]

FYI openssl/openssl#17064

In our PHP environment we encountered a noticeable performance degradation between Alpine's 3.15 openssl 1.1.1 and Alpine's 3.17 openssl 3.0.x while using cURL.

[rlerdorf]

Yes, we need to find a workaround for this, although I am not sure we can at the PHP level for the curl extension. It has to do with EVP cipher initialization. Interesting read here on what they did in Java to address this a couple of weeks ago - ibmruntimes/openj9-openjdk-jdk#650

[rlerdorf]

The easy workaround here is to always link against libcurl-nss. That's the default on rpm-based distros, so I think those are fine. On .deb distros PHP needs to be linked against libcurl4-nss-dev and we should make sure the Windows build built against libcurl-nss as well.

[MichalPetryka]

The easy workaround here is to always link against libcurl-nss. That's the default on rpm-based distros, so I think those are fine. On .deb distros PHP needs to be linked against libcurl4-nss-dev and we should make sure the Windows build built against libcurl-nss as well.

CURL recently dropped support for the NSS backend.

[rlerdorf]

Yes, I understand that Curl 8.3 and later won't have NSS support, but we are in this weird situation where various distros are shipping PHP/Python/Ruby linked against libcurl/OpenSSL-3.0.x. But, as per curl/curl#11543 I don't think we can expect a fix in libcurl for this OpenSSL 3.0.x perf issue. So we are in a waiting pattern until a potential future OpenSSL 3.1. Ubuntu and other distros also provide libcurl linked against NSS (libcurl4-nss-dev) and that is not going to go away for the duration of the LTS versions so I think it is a safe holding pattern to suggest that distros link PHP against that instead for now.

[bukka]

I would not recommend NSS as its API sucks and it's very hard to do anything with it. It's specifically designed for Firefox and does not really care much about anything else. It's also less proven in terms of security than OpenSSL which should be always priority here.

It is entirely up to distribution what they do and we should not be suggesting anything. But if we really wanted to suggest anything it should be that they should update to 3.1 as there has been some perf fixes and it is already available.

I will close this as there is nothing we can do about this in php core code - even if we wanted to suggest anything to distros, there is nothing in the code to do so.

[bukka]

What I meant in terms of noting the NSS API is that there is a good reason why it was dropped from Curl and we should not asking people to use it. Maybe you could try gnutls but don't know much about it. From what I see it's not at least deprecated and still available.

[M1ke]

Example of this in a Linux production environment (issue is difference in openssl versions 1.1 & 3) #10753 (comment)

[digitalkaoz]

downgrading to bullseye solved it for us, so indeed its something curl related which needs to be fixed somehow

curl 7.74.0
PHP 8.4.5

using php:8.4-fpm-bullseye as docker image.

@bukka its neither completed nor irrelevant i guess, i dont know if there is anything php itself can do (so it is really to be closed) or its an underlying issue

[bukka]

This was due to OpenSSL issues that have been mostly fixed in later versions. OpenSSL 3.4 should be much better. Not sure if 3.5 has got also some extra fixes but it is the next LTS so it should get all resolved in distros with updates to the later OpenSSL versions.