Skip to content

Latest commit

 

History

History
126 lines (96 loc) · 5.29 KB

README.md

File metadata and controls

126 lines (96 loc) · 5.29 KB
Raw

json2csv – Parse JSON files to CSV with data qualifier

Go version Go report Wiki License

The parser can read given folder with *.json files, filtering and qualifying input data with intent & stop words dictionaries and save results to CSV files by given chunk size.

Minimal dependency on other Go packages, but maximum performance even on large amounts of the input JSON data.

⚡️ Quick start

First, download and install Go. Version 1.20 (or higher) is required.

Installation is done by using the go install command:

go install github.com/koddr/json2csv@latest

💡 Note: See the repository's Release page, if you want to download a ready-made deb, rpm, apk or Arch Linux package.

GNU/Linux and macOS users available way to install via Homebrew:

# Tap a new formula:
brew tap koddr/tap

# Installation:
brew install koddr/tap/json2csv

Prepare folder with your data input in the *.json format, create JSON files with intents and filter.

💡 Note: See the repository's Wiki page to understand structures of JSON files and get general recommendations for preparing the input data.

Next, run json2csv parser with options:

json2csv \
  -json    /path/to/input/json/folder \
  -intents /path/to/intents-file.json \
  -filter  /path/to/filter-file.json \
  -output  /path/to/output/csv/folder \
  -content-field message \
  -min-word-len  5 \
  -chunk 1000

Done! 🎉 Output CSV file(s) has a default comma (,) separators between columns.

🐳 Docker-way to quick start

If you don't want to physically install json2csv to your system, you feel free to using our official Docker image and run it from isolated container:

docker run --rm -it -v ${PWD}:${PWD} -w ${PWD} koddr/json2csv:latest [OPTIONS]

🧩 Options

Option Description Is required? Default value Docs link
-json set a path to the folder with JSON source file(s) no ./json_files Wiki
-intents set a path to the file with intents no ./intents.json Wiki
-filter set a path to the file with a filter no ./filter.json Wiki
-output set a path to the output folder where the prepared CSV file(s) will be placed no ./output_files Wiki
-content-field set a name of the content field (attribute that contains string to qualify and filter) in JSON source file(s) no content
-min-word-len set a min word length count to filter input words (if a word is smaller than this option, it will be skipped) no 0
-chunk set a chunk size for one CSV file no 5000

✨ Solving case

Imagine that you have many JSON files of the same type in a folder on your computer. The structure of each such file is identical and can be described in a single Go structure.

But your task is not only to turn these files into a structured CSV, but also to make some qualification of incoming data based on dictionaries with intent.

The output should be CSV files (with a specified number of lines in each), where the first column contains the original phrases from JSON files, and the second column will contain the associated intents after qualification.

This is what this Go package solves! ✌️

⚡️ More details about the case I solved are here.

⚠️ License

json2csv is free and open-source software licensed under the Apache 2.0 License, created and supported with 🩵 for people and robots by Vic Shóstak.