Skip to content

Commit

Permalink
clean code base
Browse files Browse the repository at this point in the history
  • Loading branch information
Keyang committed Dec 24, 2016
1 parent d951b37 commit b4e35a6
Show file tree
Hide file tree
Showing 15 changed files with 24 additions and 1,216 deletions.
14 changes: 10 additions & 4 deletions docs/performance.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
# Extremely Fast

`csvtojson` takes care of performance and optimise for Node.js apps.

[This project](https://github.com/Keyang/csvbench) shows `csvtojson` is about **4 - 6 times** faster than other popular csv parsing libraries.

# Performance Optimisation

Version 1.1.0 is much faster than versions before.

Below test is parsing a 300K records csv
Below test is parsing a 300k records csv on a 4 core machine

Time | workNum = 1 | workNum = 2 (fork)* | workNum = 3 | workNum = 4
Time | workerNum = 1 | workerNum = 2 (fork)* | workerNum = 3 | workerNum = 4
--- | --- | --- | --- | ---
**1.0.3** | 11.806s | 15.945s | 8.611s | 8.314s
**1.1.0** | 9.707s | 10.065s | 5.955s | 4.563s

*when workNum=2, it only creates 1 extra worker to unblock main process. It is reasonalbe it is slightly slower than workerNum=1 (just use main process).
*when workerNum=2, it only creates 1 extra worker to unblock main process. It is reasonalbe it is slightly slower than workerNum=1 (just use main process).

The result shows V1.1.0 has about 30% - 50% performance boost.

# CPU usage leverage

Below is CPU core usages for v1.1.0 when running the test:

Core | workNum = 1 | workNum = 2| workNum = 3 | workNum = 4
Core | workerNum = 1 | workerNum = 2| workerNum = 3 | workerNum = 4
--- | --- | --- | --- | ---
Core 1 (Main) | 100% | 25% | 60% | 90%
Core 2 | N/A | 80% | 70% | 70%
Expand Down
100 changes: 0 additions & 100 deletions libs/core/CSVLine_bk.js

This file was deleted.

4 changes: 0 additions & 4 deletions libs/core/Converter.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
var util = require("util");
var Transform = require("stream").Transform;
var Readable = require("stream").Readable;
var Result = require("./Result");
var os = require("os");
var eol = os.EOL;
// var Processor = require("./Processor.js");
var utils = require("./utils.js");
var async = require("async");
var defParam=require("./defParam");
var csvline=require("./csvline");
var fileline=require("./fileline");
Expand Down
Loading

0 comments on commit b4e35a6

Please sign in to comment.