Skip to content

Commit 7d639b8

Browse files
nao1215nao1215
authored
Merge pull request #56 from nao1215/add/benchmark
Add benchmark
2 parents 4f48ee1 + 2e7bf42 commit 7d639b8

File tree

8 files changed

+100188
-77
lines changed

8 files changed

+100188
-77
lines changed

‎.all-contributorsrc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"projectName": "sqly",
3+
"projectOwner": "nao1215",
4+
"repoType": "github",
5+
"repoHost": "https://github.com",
6+
"files": [
7+
"README.md"
8+
],
9+
"imageSize": 75,
10+
"commit": true,
11+
"commitConvention": "atom",
12+
"contributors": [
13+
{
14+
"login": "nao1215",
15+
"name": "CHIKAMATSU Naohiro",
16+
"avatar_url": "https://avatars.githubusercontent.com/u/22737008?v=4",
17+
"profile": "https://debimate.jp/",
18+
"contributions": [
19+
"code",
20+
"doc"
21+
]
22+
}
23+
],
24+
"contributorsPerLine": 7,
25+
"linkToUsage": true
26+
}

‎CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1-
## [](https://github.com/nao1215/sqly/compare/v0.7.0...) (2024-04-30)
1+
## [](https://github.com/nao1215/sqly/compare/v0.7.0...) (2024-05-01)
22

3+
* Add unit test for excel [#55](https://github.com/nao1215/sqly/pull/55) ([nao1215](https://github.com/nao1215))

‎Makefile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ test: ## Start test
2727
env GOOS=$(GOOS) $(GO_TEST) -cover $(GO_PKGROOT) -coverpkg=./... -coverprofile=cover.out
2828
$(GO_TOOL) cover -html=cover.out -o cover.html
2929

30+
bench: ## Start benchmark
31+
env GOOS=$(GOOS) go test -bench=BenchmarkImport100000Records -benchmem
32+
3033
coverage-tree: test ## Generate coverage tree
3134
go-cover-treemap -statements -coverprofile cover.out > doc/img/cover-tree.svg
3235

‎README.md

Lines changed: 115 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
2+
[![All Contributors](https://img.shields.io/badge/all_contributors-1-orange.svg?style=flat-square)](#contributors-)
3+
<!-- ALL-CONTRIBUTORS-BADGE:END -->
4+
15
![Coverage](https://raw.githubusercontent.com/nao1215/octocovs-central-repo/main/badges/nao1215/sqly/coverage.svg)
26
[![Build](https://github.com/nao1215/sqly/actions/workflows/build.yml/badge.svg)](https://github.com/nao1215/sqly/actions/workflows/build.yml)
37
[![LinuxUnitTest](https://github.com/nao1215/sqly/actions/workflows/linux_test.yml/badge.svg)](https://github.com/nao1215/sqly/actions/workflows/linux_test.yml)
@@ -8,37 +12,55 @@
812
![GitHub](https://img.shields.io/github/license/nao1215/sqly)
913
![demo](./doc/img/demo.gif)
1014

11-
**sqly** command imports CSV/TSV/LTSV/JSON and Microsoft Excel™ (XLAM / XLSM / XLSX / XLTM / XLTX) file(s) into an in-memory DB and executes SQL against them. sqly uses [SQLite3](https://www.sqlite.org/index.html) as its DB. So, sql syntax is same as SQLite3.
15+
**sqly** is a powerful command-line tool that can execute SQL against CSV, TSV, LTSV, JSON, and even Microsoft Excel™ files. The sqly import those files into [SQLite3](https://www.sqlite.org/index.html) in-memory database.
1216

13-
The sqly command has sqly-shell. You can interactively execute SQL with sql completion and command history. Of course, you can also execute SQL without running the sqly-shell.
17+
The sqly has **sqly-shell**. You can interactively execute SQL with sql completion and command history. Of course, you can also execute SQL without running the sqly-shell.
1418

15-
## Features
16-
✅ execute SQL against CSV / TSV / LTSV / JSON and Microsoft Excel™ (XLAM / XLSM / XLSX / XLTM / XLTX).
17-
✅ output SQL result to CSV / TSV / LTSV / JSON and Microsoft Excel™ (XLAM / XLSM / XLSX / XLTM / XLTX).
18-
✅ print SQL result in ASCII Table / CSV / TSV / LTSV / JSON file format.
19-
✅ interactive sqly shell with input completion, emacs-keybindings, input history.
19+
> [!WARNING]
20+
> The support for JSON is limited. There is a possibility of discontinuing JSON support in the future.
2021
2122
## How to install
2223
### Use "go install"
23-
If you does not have the golang development environment installed on your system, please install golang from the [golang official website](https://go.dev/doc/install).
24-
```
24+
```shell
2525
$ go install github.com/nao1215/sqly@latest
2626
```
2727
※ Main dependency is [github.com/mattn/go-sqlite3](https://github.com/mattn/go-sqlite3) and gcc.
2828

29+
## Supported OS
30+
- Windows
31+
- macOS
32+
- Linux
2933

3034
## How to use
31-
sqly command automatically imports the CSV/TSV/LTSV/JSON file into the DB when you pass a CSV/TSV/LTSV/JSON file as an argument. DB table name is the same as the file name (e.g., if you import user.csv, sqly command create the user table)
35+
The sqly automatically imports the CSV/TSV/LTSV/JSON/Excel file into the DB when you pass file path as an argument. DB table name is the same as the file name or sheet name (e.g., if you import user.csv, sqly command create the user table).
3236

33-
### Syntax
34-
```
37+
The sqly automatically determines the file format from the file extension.
38+
39+
### Syntax and Options
40+
```shell
41+
[Usage]
3542
sqly [OPTIONS] [FILE_PATH]
43+
44+
[OPTIONS]
45+
-c, --csv change output format to csv (default: table)
46+
-e, --excel change output format to excel (default: table)
47+
-j, --json change output format to json (default: table)
48+
-l, --ltsv change output format to ltsv (default: table)
49+
-m, --markdown change output format to markdown table (default: table)
50+
-t, --tsv change output format to tsv (default: table)
51+
-S, --sheet string excel sheet name you want to import
52+
-s, --sql string sql query you want to execute
53+
-o, --output string destination path for SQL results specified in --sql option
54+
-h, --help print help message
55+
-v, --version print sqly version
3656
```
57+
3758
※ The sqly option must be specified before the file to be imported.
3859
39-
### --sql option: execute sql in terminal
40-
--sql option takes an SQL statement as an optional argument. You pass file path(s) as arguments to the sqly command. sqly command import them. sqly command automatically determines the file format from the file extension.
41-
```
60+
### Execute sql in terminal: --sql option
61+
--sql option takes an SQL statement as an optional argument.
62+
63+
```shell
4264
$ sqly --sql "SELECT user_name, position FROM user INNER JOIN identifier ON user.identifier = identifier.id" testdata/user.csv testdata/identifier.csv
4365
+-----------+-----------+
4466
| user_name | position |
@@ -50,8 +72,14 @@ $ sqly --sql "SELECT user_name, position FROM user INNER JOIN identifier ON user
5072
```
5173
5274
### Change output format
53-
sqly command output sql results in ASCII table format, CSV format (--csv option), TSV format (--tsv option), LTSV format (--ltsv option) and JSON format (--json option). This means that conversion between csv and json is supported.
54-
```
75+
The sqly output sql query results in following formats:
76+
- ASCII table format (default)
77+
- CSV format (--csv option)
78+
- TSV format (--tsv option)
79+
- LTSV format (--ltsv option)
80+
- JSON format (--json option)
81+
82+
```shell
5583
$ sqly --sql "SELECT * FROM user LIMIT 2" --csv testdata/user.csv
5684
user_name,identifier,first_name,last_name
5785
booker12,1,Rachel,Booker
@@ -81,58 +109,29 @@ Rachel,1,Booker,booker12
81109
Mary,2,Jenkins,jenkins46
82110
```
83111
84-
### run sqly shell
85-
If the --sql option is not specified, the sqly shell is started. When you execute sqly command, it is optional whether or not to specify file(s). The sqly shell functions similarly to a common SQL client (e.g., sqlite3 command or mysql command). sqly shell has helper commands, SQL execution history management and input complement.
112+
### Run sqly shell
113+
![sqly-shell](./doc/img/sqly-shell.png)
86114
87-
#### sqly helper command
88-
The command beginning with a dot is the sqly helper command; I plan to add more features in the future to make the sqly shell run more comfortably.
89-
```
90-
$ sqly
91-
sqly v0.5.0 (work in progress)
92-
93-
enter "SQL query" or "sqly command that begins with a dot".
94-
.help print usage, .exit exit sqly.
95-
96-
sqly> .help
97-
.dump: dump db table to file in a format according to output mode (default: csv)
98-
.exit: exit sqly
99-
.header: print table header
100-
.help: print help message
101-
.import: import csv file(s)
102-
.mode: change output mode
103-
.tables: print tables
104-
```
105-
106-
![demo](./doc/img/shell-demo.png)
115+
The sqly-shell starts when you run the sqly command without the --sql option. When you execute sqly command with file path, the sqly-shell starts after importing the file into the SQLite3 in-memory database.
116+
117+
The sqly shell functions similarly to a common SQL client (e.g., `sqlite3` command or `mysql` command). The sqly-shell has helper commands that begin with a dot. The sqly-shell also supports command history, and input completion.
107118
108119
### Output sql result to file
109120
#### For linux user
110-
sqly command can save SQL execution results to a file using shell redirection. The --csv option outputs SQL execution results in CSV format instead of table format.
111-
```
121+
The sqly can save SQL execution results to the file using shell redirection. The --csv option outputs SQL execution results in CSV format instead of table format.
122+
```shell
112123
$ sqly --sql "SELECT * FROM user" --csv testdata/user.csv > test.csv
113124
```
125+
114126
#### For windows user
115-
```
116-
$ sqly --sql "SELECT * FROM user" --output=test.csv testdata/user.csv
117-
```
118127
119-
### All options
120-
```
121-
[OPTIONS]
122-
-c, --csv change output format to csv (default: table)
123-
-e, --excel change output format to excel (default: table)
124-
-j, --json change output format to json (default: table)
125-
-l, --ltsv change output format to ltsv (default: table)
126-
-m, --markdown change output format to markdown table (default: table)
127-
-t, --tsv change output format to tsv (default: table)
128-
-S, --sheet string excel sheet name you want to import
129-
-s, --sql string sql query you want to execute
130-
-o, --output string destination path for SQL results specified in --sql option
131-
-h, --help print help message
132-
-v, --version print sqly version
128+
The sqly can save SQL execution results to the file using the --output option. The --output option specifies the destination path for SQL results specified in the --sql option.
129+
130+
```shell
131+
$ sqly --sql "SELECT * FROM user" --output=test.csv testdata/user.csv
133132
```
134133
135-
### Key Binding
134+
### Key Binding for sqly-shell
136135
|Key Binding |Description|
137136
|:--|:--|
138137
|Ctrl + A |Go to the beginning of the line (Home)|
@@ -148,17 +147,27 @@ $ sqly --sql "SELECT * FROM user" --output=test.csv testdata/user.csv
148147
|Ctrl + U |Cut the line before the cursor to the clipboard|
149148
|Ctrl + L |Clear the screen|
150149
151-
## Features to be added
152-
- [ ] import swagger
153-
- [ ] import .gz file
154-
- [ ] ignore csv header option
155-
- [ ] history search (Ctrl-r)
156-
- [ ] Convert CSV character encoding to UTF-8 if necessary
157-
- [ ] Support MySQL driver
158-
- [ ] Support PostgreSQL driver
150+
## Benchmark
151+
CPU: AMD Ryzen 5 3400G with Radeon Vega Graphics
152+
Execute:
153+
```sql
154+
SELECT * FROM `table` WHERE `Index` BETWEEN 1000 AND 2000 ORDER BY `Index` DESC LIMIT 1000
155+
```
156+
157+
|Records | Columns | Time per Operation | Memory Allocated per Operation | Allocations per Operation |
158+
|---------|----|-------------------|--------------------------------|---------------------------|
159+
|100,000| 12| 1715818835 ns/op | 441387928 B/op |4967183 allocs/op |
160+
|1,000,000| 9| 11414332112 ns/op | 2767580080 B/op | 39131122 allocs/op |
161+
162+
163+
## Altenative Tools
164+
|Name| Description|
165+
|:--|:--|
166+
|[harelba/q](https://github.com/harelba/q)|Run SQL directly on delimited files and multi-file sqlite databases|
167+
|[dinedal/textql](https://github.com/dinedal/textql)|Execute SQL against structured text like CSV or TSV|
168+
|[noborus/trdsql](https://github.com/noborus/trdsql)|CLI tool that can execute SQL queries on CSV, LTSV, JSON, YAML and TBLN. Can output to various formats.|
169+
|[mithrandie/csvq](https://github.com/mithrandie/csvq)|SQL-like query language for csv|
159170
160-
## Unit Test Coverage Treemap
161-
![treemap](./doc/img/cover-tree.svg)
162171
163172
## Limitions (Not support)
164173
- DDL such as CREATE
@@ -170,11 +179,46 @@ First off, thanks for taking the time to contribute! ❤️ Contributions are no
170179
171180
[![Star History Chart](https://api.star-history.com/svg?repos=nao1215/sqly&type=Date)](https://star-history.com/#nao1215/sqly&Date)
172181
173-
174-
## Contact
182+
### Contact
175183
If you would like to send comments such as "find a bug" or "request for additional features" to the developer, please use one of the following contacts.
176184
177185
- [GitHub Issue](https://github.com/nao1215/sqly/issues)
178186
187+
### For Developers
188+
When adding new features or fixing bugs, please write unit tests.
189+
190+
![treemap](./doc/img/cover-tree.svg)
191+
179192
## LICENSE
180193
The sqly project is licensed under the terms of [MIT LICENSE](./LICENSE).
194+
195+
## Contributors ✨
196+
197+
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
198+
199+
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
200+
<!-- prettier-ignore-start -->
201+
<!-- markdownlint-disable -->
202+
<table>
203+
<tbody>
204+
<tr>
205+
<td align="center" valign="top" width="14.28%"><a href="https://debimate.jp/"><img src="https://avatars.githubusercontent.com/u/22737008?v=4?s=75" width="75px;" alt="CHIKAMATSU Naohiro"/><br /><sub><b>CHIKAMATSU Naohiro</b></sub></a><br /><a href="https://github.com/nao1215/sqly/commits?author=nao1215" title="Code">💻</a> <a href="https://github.com/nao1215/sqly/commits?author=nao1215" title="Documentation">📖</a></td>
206+
</tr>
207+
</tbody>
208+
<tfoot>
209+
<tr>
210+
<td align="center" size="13px" colspan="7">
211+
<img src="https://raw.githubusercontent.com/all-contributors/all-contributors-cli/1b8533af435da9854653492b1327a23a4dbd0a10/assets/logo-small.svg">
212+
<a href="https://all-contributors.js.org/docs/en/bot/usage">Add your contributions</a>
213+
</img>
214+
</td>
215+
</tr>
216+
</tfoot>
217+
</table>
218+
219+
<!-- markdownlint-restore -->
220+
<!-- prettier-ignore-end -->
221+
222+
<!-- ALL-CONTRIBUTORS-LIST:END -->
223+
224+
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

‎doc/img/sqly-shell.png

46 KB
Loading

‎infrastructure/sql.go

Lines changed: 28 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
package infrastructure
22

33
import (
4+
"fmt"
5+
"runtime"
46
"strconv"
57
"strings"
8+
"sync"
69

710
"github.com/nao1215/sqly/domain/model"
811
)
@@ -42,13 +45,33 @@ func SingleQuote(s string) string {
4245
// GenerateCreateTableStatement returns create table statement.
4346
// e.g. CREATE TABLE `table_name` (`column1` INTEGER, `column2` TEXT, ...);
4447
func GenerateCreateTableStatement(t *model.Table) string {
48+
indexTypeMap := make(map[int]string, len(t.Header))
49+
semaphore := make(chan int, runtime.NumCPU())
50+
wg := &sync.WaitGroup{}
51+
52+
var mu sync.RWMutex
53+
for i := range t.Header {
54+
wg.Add(1)
55+
go func(i int) {
56+
defer wg.Done()
57+
semaphore <- 1
58+
defer func() { <-semaphore }()
59+
if isNumeric(t, i) {
60+
mu.Lock()
61+
indexTypeMap[i] = "INTEGER"
62+
mu.Unlock()
63+
} else {
64+
mu.Lock()
65+
indexTypeMap[i] = "TEXT"
66+
mu.Unlock()
67+
}
68+
}(i)
69+
}
70+
wg.Wait()
71+
4572
ddl := "CREATE TABLE " + Quote(t.Name) + "("
4673
for i, v := range t.Header {
47-
if isNumeric(t, i) {
48-
ddl += Quote(v) + " INTEGER"
49-
} else {
50-
ddl += Quote(v) + " TEXT"
51-
}
74+
ddl += fmt.Sprintf("%s %s", Quote(v), indexTypeMap[i])
5275
if i != len(t.Header)-1 {
5376
ddl += ", "
5477
} else {

‎main_test.go

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,19 @@ func Test_runErrPatern(t *testing.T) {
152152
})
153153
}
154154

155+
func BenchmarkImport100000Records(b *testing.B) {
156+
b.ResetTimer()
157+
158+
for i := 0; i < b.N; i++ {
159+
run([]string{
160+
"sqly",
161+
"--sql",
162+
"SELECT * FROM `customers100000` WHERE `Index` BETWEEN 1000 AND 2000 ORDER BY `Index` DESC LIMIT 1000",
163+
"testdata/benchmark/customers100000.csv",
164+
})
165+
}
166+
}
167+
155168
func getStdoutForRunFunc(t *testing.T, f func([]string) int, list []string) []byte {
156169
t.Helper()
157170
backupColorStdout := config.Stdout

0 commit comments

Comments
 (0)