Skip to content

Child processes taking more time than parent for JSON processing #97

Open
@ishitach

Description

@ishitach

I am trying JSON stringify of very large datasets. I am creating child processes to perform the operations and doing equivalent operations in the parent process.

The child processes should execute the results faster as they are executing the process in parallel but I am observing opposite results in my test JSON.
For running this code please replace the url's with the actual JSON(unable to paste them due to character limit).
The code for main.js is as follows:

let workerFarm = require("worker-farm")
    , workers    = workerFarm(require.resolve('./child'))
    , ret        = 0
    ,final_out =0;

let start1 = new Date().getTime();
for (let i = 0; i < 100; i++) {
    workers('#' + i + ' FOO', function (err, outp) {
        if (++ret == 100){
            console.log('Doing it the fast (multi-process) way...');
            let end1 = new Date().getTime();
            console.log('fast took',  (end1 - start1), 'milliseconds')
            workerFarm.end(workers)
        }
    })
}

let start2 = new Date().getTime();

for (let i = 0; i < 100; i++) {

    let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));

        if (i == 99){
        
        console.log('Doing it the slow (single-process) way...')
        console.log('slow took', new Date().getTime() - start2, 'milliseconds')
    }
}

The code for child.js file is as follows:

'use strict'

module.exports = function (inp, callback) {
    let srt= new Date().getTime();
    let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));

    let end = new Date().getTime();
    return callback(null, end-srt);
}

Kindly help me identify the issue as the results of this iteration are as follows:

Doing it the slow (single-process) way...
slow took 967 milliseconds
Doing it the fast (multi-process) way...
fast took 1079 milliseconds

The child processes are taking more time. I have tried this for less as well as more iterations but the results are quite similar.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions