Keep Alive on got ?

[Vance-ng-vn]

I am using got-scraping 4.0.2 (base on got 13.0.0). When I set 'http2: false' and check the value of got.defaults.options.agent.http/https, by default, the keepAlive values are set to false. I tried extend() it, but got requires a valid agent instead of extending some values! (and its doesnt work with 'new http/https.Agent()')

So, I used it like this. Is it correct?

let gotConfig = {
    headerGeneratorOptions:{
        browsers: [
            {
                name: 'firefox',
                minVersion: 102,
                maxVersion: 120
            }
        ],
        devices: ['desktop'],
        operatingSystems: ['linux']
    },
    timeout: {
        request: 30000
    },
    headers: {
        'Connection': 'keep-alive'
    },
    retry: {
        limit: 2,
        statusCodes: [400,401,403,404,408,409,413,429,500,502,503,504,521,522,524],
        calculateDelay: ({attemptCount, retryOptions, error}) => {
            if(attemptCount > retryOptions.limit) return 0;
            if(error.code == 'ECONNRESET') return 1000; //unknow error ...
            if(error.code == 'ETIMEDOUT') return 0; //dont retry when it's time out

            let response = err.response;
            if(response) {
                if([409,429].includes(response.statusCode)) return 1500; //too many request
                if([400,401,403,404,408,413,500,502,503,504,521,522,524].includes(response.statusCode)) return 0;
            }

            //default
            return 500;
        }
    },
    http2: false
}

let gotScraping;

const import_got = async () => (await import('got-scraping')).gotScraping.extend(gotConfig);

import_got().then(async got => {
    gotScraping = got
    //console.log(gotScraping)
    //console.log(await got.get('https://duckduckgo.com').then(res => res))

    let agents = got.defaults.options.agent;
    agents.https.agent.keepAlive = true;
    agents.https.agent.keepAliveMsecs = 60000;

    gotScraping.extend({agent: {
        http: agents.http,
        https: agents.https
    }})
});

const got = () => gotScraping;

module.exports = got;

[sindresorhus]

Got uses keep-alive by default! The underlying Node.js http/https agents have keepAlive: true enabled automatically.

If you need to customize the agent settings (like keepAliveMsecs or maxSockets), you can pass custom agents:

import http from 'http';
import https from 'https';
import got from 'got';

const httpAgent = new http.Agent({
	keepAlive: true,
	keepAliveMsecs: 30000,
	maxSockets: 50
});

const httpsAgent = new https.Agent({
	keepAlive: true,
	keepAliveMsecs: 30000,
	maxSockets: 50
});

await got('https://example.com', {
	agent: {
		http: httpAgent,
		https: httpsAgent
	}
});

For got-scraping, the same approach should work. The key is that you don't need to manually set keepAlive: true on the agents after creating them - you should pass it in the constructor options when creating the agent intance.