Description
I am trying JSON stringify of very large datasets. I am creating child processes to perform the operations and doing equivalent operations in the parent process.
The child processes should execute the results faster as they are executing the process in parallel but I am observing opposite results in my test JSON.
For running this code please replace the url's with the actual JSON(unable to paste them due to character limit).
The code for main.js is as follows:
let workerFarm = require("worker-farm")
, workers = workerFarm(require.resolve('./child'))
, ret = 0
,final_out =0;
let start1 = new Date().getTime();
for (let i = 0; i < 100; i++) {
workers('#' + i + ' FOO', function (err, outp) {
if (++ret == 100){
console.log('Doing it the fast (multi-process) way...');
let end1 = new Date().getTime();
console.log('fast took', (end1 - start1), 'milliseconds')
workerFarm.end(workers)
}
})
}
let start2 = new Date().getTime();
for (let i = 0; i < 100; i++) {
let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));
if (i == 99){
console.log('Doing it the slow (single-process) way...')
console.log('slow took', new Date().getTime() - start2, 'milliseconds')
}
}
The code for child.js file is as follows:
'use strict'
module.exports = function (inp, callback) {
let srt= new Date().getTime();
let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));
let end = new Date().getTime();
return callback(null, end-srt);
}
Kindly help me identify the issue as the results of this iteration are as follows:
Doing it the slow (single-process) way...
slow took 967 milliseconds
Doing it the fast (multi-process) way...
fast took 1079 milliseconds
The child processes are taking more time. I have tried this for less as well as more iterations but the results are quite similar.