Skip to content

Commit fd2082d

Browse files
tolbertamtolbertam
committed
Update README.md for 3.0.0-alpha3.
1 parent 76d831a commit fd2082d

File tree

1 file changed

+88
-190
lines changed

1 file changed

+88
-190
lines changed

‎README.md

Lines changed: 88 additions & 190 deletions
Original file line numberDiff line numberDiff line change
@@ -2,51 +2,60 @@
22

33
[![Build Status](https://travis-ci.org/tolbertam/sstable-tools.svg?branch=master)](https://travis-ci.org/tolbertam/sstable-tools)[ ![Download](https://api.bintray.com/packages/tolbertam/sstable-tools/sstable-tools.jar/images/download.svg) ](https://bintray.com/tolbertam/sstable-tools/sstable-tools.jar/_latestVersion)
44

5-
A toolkit for parsing, creating and doing other fun stuff with Cassandra 3.x SSTables. This project is under heavy development and not yet stable.
5+
A toolkit for parsing, creating and doing other fun stuff with Cassandra 3.x SSTables. This project is under development and not yet stable.
66

77
Pre compiled binary available from bintray:
88

9-
* [sstable-tools-3.0.0-alpha2.jar](https://bintray.com/artifact/download/tolbertam/sstable-tools/sstable-tools-3.0.0-alpha2.jar) - Currently tested with 3.0, 3.1, 3.1.1, 3.2.1, 3.3.
9+
* [sstable-tools-3.0.0-alpha3.jar](https://bintray.com/artifact/download/tolbertam/sstable-tools/sstable-tools-3.0.0-alpha3.jar) - Currently tested with 3.0, 3.1, 3.1.1, 3.2.1, 3.3.
1010

1111
Example usage:
1212

13-
java -jar sstable-tools-3.0.0-alpha2.jar toJson ma-2-big-Data.db
14-
java -jar sstable-tools-3.0.0-alpha2.jar describe ma-2-big-Data.db
15-
java -jar sstable-tools-3.0.0-alpha3.jar SELECT count(*) FROM ma-2-big-Data.db WHERE age > 30 (TODO: Escape this as it doesnt work as is.)
16-
13+
java -jar sstable-tools.jar cqlsh
14+
java -jar sstable-tools.jar toJson ma-2-big-Data.db
15+
java -jar sstable-tools.jar describe ma-2-big-Data.db
16+
1717
Example shell usage:
1818

19-
java -jar sstable-tools-3.0.0-alpha2.jar cqlsh
20-
21-
## Select one or more (space delimited, or choose directory to include all)
19+
java -jar sstable-tools.jar cqlsh
20+
21+
## Select one or more sstables (space delimited, or choose directory to include all)
2222
cqlsh> use ma-2-big-Data.db;
23-
24-
## Use create table statement to set the schema (can view with 'describe schema'). This is optional but without
25-
## it the partition key and clustering index names are unknown.
23+
Using: /home/user/sstable-tools/ma-2-big-Data.db
24+
25+
## Use predefined schema file.
26+
## Can view with 'schema'.
27+
## This is optional but without it the partition
28+
## key and clustering index names are unknown.
29+
cqlsh> schema schema.cql
30+
Successfully imported schema from '/home/user/sstable-tools/schema.cql'.
31+
32+
## Alternatively, use 'CREATE TABLE' statement to enter a schema.
2633
cqlsh> CREATE TABLE users (
2734
... user_name varchar PRIMARY KEY,
2835
... password varchar,
2936
... gender varchar,
3037
... state varchar,
3138
... birth_year bigint
3239
... );
33-
40+
3441
## Discover the data in sstable(s) using CQL queries
3542
cqlsh> SELECT * FROM sstable WHERE age > 1 LIMIT 1
36-
43+
3744
┌────────────┬─────────────┬─────────┬───────────┬────────┐
3845
│user_name │birth_year │gender │password │state │
3946
╞════════════╪═════════════╪═════════╪═══════════╪════════╡
4047
│frodo │1985 │male │pass@ │CA │
4148
└────────────┴─────────────┴─────────┴───────────┴────────┘
42-
43-
## Display raw sstable data (useful to see tombstones and expired ttls) with optional where clause
49+
50+
## Display raw sstable data (useful to see tombstones and expired ttls)
51+
## with optional where clause
4452
cqlsh> DUMP WHERE age > 1 LIMIT 1
53+
4554
[frodo] Row[info=[ts=1455937221199050] ]: | [birth_year=1985 ts=1455937221199050], [gender=male ts=1455937221199050], [password=pass@ ts=1455937221199050], [state=CA ts=1455937221199050]
46-
55+
4756
## Describe the sstable data and metadata
4857
cqlsh> describe sstables;
49-
58+
5059
/Users/clohfink/git/sstable-tools/./src/test/resources/ma-2-big-Data.db
5160
=======================================================================
5261
Partitions: 1
@@ -60,7 +69,7 @@ Example shell usage:
6069
Tombstone Leaders:
6170
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
6271
Bloom Filter FP chance: 0.010000
63-
Size: 50 (50 B)
72+
Size: 50 (50 B)
6473
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
6574
Compression ratio: -1.0
6675
Minimum timestamp: 1455937221199050 (02/19/2016 21:00:21)
@@ -69,19 +78,39 @@ Example shell usage:
6978
SSTable max local deletion time: 2147483647 (01/18/2038 21:14:07)
7079
TTL min: 0 (0 milliseconds)
7180
...[snip]...
72-
73-
>>>>>>> update readme a little
74-
**Note:** No environment configuration is necessary for this tool to work if all components of the sstable are available but the cql create statement allows for more details.
75-
76-
**Note:** CQL statements require bash escaping when using the "select" command via command line ```java -jar sstable-tools.jar select \* from \"path\"```
7781

82+
## Paging is enabled by default and can be manipulated by using 'PAGING'
83+
cqlsh> PAGING 20;
84+
Now Query paging is enabled
85+
Page size: 20
86+
cqlsh> PAGING OFF;
87+
Disabled Query paging.
88+
cqlsh> PAGING ON;
89+
Now Query paging is enabled
90+
Page size: 20
91+
92+
## Used sstables, schema, and paging settings and persisted for future use.
93+
## Use the 'PERSIST' command to view preferences and to enable/disable
94+
## persistence.
95+
cqlsh> PERSIST;
96+
Preferences are currently enabled:
97+
pagingEnabled=true
98+
pagingSize=20
99+
preferencesEnabled=true
100+
schema=""
101+
sstables=[
102+
"/home/user/sstable-tools/ma--big-Data.db"
103+
]
104+
cqlsh> PERSIST OFF;
105+
Disabled Preferences.
78106

79-
see more below
107+
**Note:** No environment configuration is necessary for this tool to work if all components of the sstable are available but the cql create statement allows for more details.
80108

81109
**Features:**
82110

111+
* [cqlsh](#cqlsh) - Drop into an interactive shell to make queries against SSTables.
112+
* [describe](#describe) - Describe SSTable data and metadata.
83113
* [sstable2json](#sstable2json) - Utility for exporting C\* 3.X SSTables into JSON.
84-
* [select](#select) - Make CQL queries against SSTables
85114

86115
## Building
87116

@@ -95,36 +124,58 @@ mvn package
95124
The executable jar will be present in the target directory.
96125

97126
## cqlsh
98-
cql shell similiar and modeled after the C* cqlsh tool. This will allow issuing cql queries against raw sstables and
99-
providing additional diagnostic tools over them. Provides history (reverse searchable with ctrl-r) for ease of use.
127+
cql shell similiar and modeled after the C* cqlsh tool. Enables issuing cql queries against raw sstables and
128+
provides additional diagnostic tools over them. Provides history (reverse searchable with ctrl-r) for ease of use.
100129

101130
```text
102-
EXIT - leaves the shell (also ctrl-d on prompt, ctrl-c to break back to blank prompt)
103-
CREATE TABLE ... - A CREATE TABLE cql statement to use as metadata when reading sstables (HIGHLY RECOMMENDED!)
104-
DESCRIBE SCHEMA - Show currently used schema (or serialized cfmetadata if generated)
105-
DESCRIBE SSTABLES - Provide details and statistics on current sstable(s)
106-
USE - update the sstable[s] used by default with select, dump, describe commands
131+
Commands:
132+
133+
HELP - prints this message
134+
EXIT - leaves the shell
135+
CREATE TABLE ... - A CREATE TABLE cql statement to use as metadata when reading sstables (HIGHLY RECOMMENDED!)
136+
DESCRIBE SCHEMA - Show currently used schema (or serialized cfmetadata if generated)
137+
DESCRIBE SSTABLES - Provide details and statistics on current sstable(s)
138+
PAGING [(ON|OFF)] - Enables, disables, or shows current status of query paging.
139+
PAGING <SIZE> - Enables paging and sets paging size.
140+
PERSIST [(ON|OFF)] - Enables, disables, or shows current status of persistence of settings state.
141+
SCHEMA [<FILE>] - Imports a cql file as the active table schema or shows active user-defined schema.
142+
USE - update the sstable[s] used by default with select, dump, describe commands
107143
USE /var/lib/cassandra/data/system/peers/ma-1-big-Data.db
108144
or with multiple sstables separated with spaces. This can also be a directory which will add all sstables in it.
109145
USE ma-1-big-Data.db ma-2-big-Data.db "/home/path with space/db/data/sstables"
110146
111-
SELECT - run a cql query against the current sstable (unless other specified)
147+
SELECT - run a cql query against the current sstable (unless other specified)
112148
SELECT * FROM sstables WHERE key > 1 LIMIT 10
113149
the keyword sstables will use the current sstable set with the USE command or set when running cqlsh. You can also
114150
specify an sstable here
115151
SELECT avg(someColumn) FROM /var/lib/cassandra/data/system/peers/ma-1-big-Data.db WHERE key > 1
116152
117-
DUMP - dump the raw unfiltered partitions/rows. Useful for debuging TTLed/tombstoned data.
153+
DUMP - dump the raw unfiltered partitions/rows. Useful for debuging TTLed/tombstoned data.
118154
DUMP;
119155
Can also specify a where clause for filtering the results.
120156
DUMP WHERE partitionKey = 'OpsCenter';
121157
```
122158

159+
## describe
160+
161+
Provides information about an sstable's data and its metadata.
162+
163+
Example Output:
164+
165+
```
166+
```
167+
168+
### Usage
169+
170+
```
171+
java -jar sstable-tools.jar describe /path/to/file.db
172+
```
173+
123174
## sstable2json
124175

125176
sstable2json is a utility in the spirit of the [original sstable2json](https://docs.datastax.com/en/cassandra/1.2/cassandra/tools/toolsSstable2JsonUtilsTOC.html)
126-
which has since been deprecated ([CASSANDRA-9618](https://issues.apache.org/jira/browse/CASSANDRA-9618))
127-
and has since been entirely removed with plans to add a replacement ([CASSANDRA-7464](https://issues.apache.org/jira/browse/CASSANDRA-7464)).
177+
which was previously deprecated ([CASSANDRA-9618](https://issues.apache.org/jira/browse/CASSANDRA-9618)).
178+
This functionality was merged into cassandra by ([CASSANDRA-7464](https://issues.apache.org/jira/browse/CASSANDRA-7464)) and to be released in Cassandra 3.0.4 and 3.4. This will likely be removed from sstable-tools in a future release.
128179

129180
A key differentiator between the storage format between older verisons of
130181
Cassandra and Cassandra 3.0 is that an SSTable was previously a representation
@@ -217,159 +268,6 @@ which represent deletes. In Cassandra 3.0, users can now delete ranges of
217268
rows ([CASSANDRA-6237](https://issues.apache.org/jira/browse/CASSANDRA-6237))
218269
which creates range tombstones.
219270

220-
221271
```json
222272
TODO: update with loss of cql create option
223273
```
224-
225-
## select
226-
227-
Use the CQL parser to query the sstables directly without Cassandra or any configuration. Does not currently support ORDER_BY and DISTINCT but all other features should work. It will search the classpath for a ```schema.cql``` (override with ```-Dsstabletools.schema=...```) that contains the CQL ```CREATE TABLE``` statement for the schema. If it cannot find one it will fall back to a best guess from the sstable metadata. The data is dumped as a result set in a data table, but to see the raw data set the ```-Dquery.toJson``` for the alternative output.
228-
229-
**WARNING:** without the schema the queries become difficult for any partition/clustering columns as their names are not included in meta data yet (CASSANDRA-9587)
230-
231-
### Usage
232-
233-
```
234-
java -jar sstable-tools.jar SELECT ...
235-
236-
usage: SELECT <column selection> FROM <sstable> WHERE <where clause>
237-
```
238-
239-
240-
### Examples
241-
242-
With a schema.cql file:
243-
```CQL
244-
CREATE TABLE mykeyspace.users (
245-
user_name varchar PRIMARY KEY,
246-
password varchar,
247-
gender varchar,
248-
state varchar,
249-
birth_year bigint
250-
);
251-
```
252-
253-
An example that selects users by birth year and state
254-
255-
```json
256-
java -jar sstable-tools.jar SELECT * FROM ma-2-big-Data.db WHERE birth_year >= 1985 AND state = 'CA'
257-
258-
┌────────────┬─────────────┬─────────┬───────────┬────────┐
259-
│user_name │birth_year │gender │password │state │
260-
╞════════════╪═════════════╪═════════╪═══════════╪════════╡
261-
│frodo │1985 │male │pass@ │CA │
262-
└────────────┴─────────────┴─────────┴───────────┴────────┘
263-
```
264-
265-
Another example is given the table
266-
```cql
267-
CREATE TABLE IF NOT EXISTS test.wide ( key text, key2 text, val text, PRIMARY KEY (key, key2));
268-
```
269-
With four partitions, each with 9 rows, key2 1-9 all with the val = "X"
270-
```
271-
java -jar sstable-tools.jar SELECT * FROM ma-3-big-Data.db
272-
┌──────┬───────┬──────┐
273-
│key │key2 │val │
274-
╞══════╪═══════╪══════╡
275-
│4 │1 │X │
276-
├──────┼───────┼──────┤
277-
│4 │2 │X │
278-
├──────┼───────┼──────┤
279-
│4 │3 │X │
280-
├──────┼───────┼──────┤
281-
│4 │4 │X │
282-
├──────┼───────┼──────┤
283-
...
284-
285-
│1 │7 │X │
286-
├──────┼───────┼──────┤
287-
│1 │8 │X │
288-
├──────┼───────┼──────┤
289-
│1 │9 │X │
290-
└──────┴───────┴──────┘
291-
```
292-
You can perform aggregates
293-
294-
```
295-
java -jar sstable-tools.jar SELECT count(*) FROM ma-3-big-Data.db
296-
┌────────┐
297-
│count │
298-
╞════════╡
299-
│36 │
300-
└────────┘
301-
302-
java -jar sstable-tools.jar SELECT min(key2), max(key2) FROM ma-3-big-Data.db
303-
┌───────────────────┬───────────────────┐
304-
│system.min(key2) │system.max(key2) │
305-
╞═══════════════════╪═══════════════════╡
306-
│1 │9 │
307-
└───────────────────┴───────────────────┘
308-
```
309-
310-
To see unfiltered data (useful for tombstone debugging) use the raw json format
311-
```
312-
java -jar -Dquery.toJson=true sstable-tools.jar SELECT * FROM ma-3-big-Data.db WHERE key2 = '1'
313-
[
314-
{
315-
"partition" : {
316-
"key" : [ "4" ]
317-
},
318-
"rows" : [
319-
{
320-
"type" : "row",
321-
"clustering" : [ "1" ],
322-
"liveness_info" : { "tstamp" : 1456111364877667 },
323-
"cells" : [
324-
{ "name" : "val", "value" : "X" }
325-
]
326-
}
327-
]
328-
},
329-
{
330-
"partition" : {
331-
"key" : [ "3" ]
332-
},
333-
"rows" : [
334-
{
335-
"type" : "row",
336-
"clustering" : [ "1" ],
337-
"liveness_info" : { "tstamp" : 1456111364856446 },
338-
"cells" : [
339-
{ "name" : "val", "value" : "X" }
340-
]
341-
}
342-
]
343-
},
344-
{
345-
"partition" : {
346-
"key" : [ "2" ]
347-
},
348-
"rows" : [
349-
{
350-
"type" : "row",
351-
"clustering" : [ "1" ],
352-
"liveness_info" : { "tstamp" : 1456111364834000 },
353-
"cells" : [
354-
{ "name" : "val", "value" : "X" }
355-
]
356-
}
357-
]
358-
},
359-
{
360-
"partition" : {
361-
"key" : [ "1" ]
362-
},
363-
"rows" : [
364-
{
365-
"type" : "row",
366-
"clustering" : [ "1" ],
367-
"liveness_info" : { "tstamp" : 1456111364803946 },
368-
"cells" : [
369-
{ "name" : "val", "value" : "X" }
370-
]
371-
}
372-
]
373-
}
374-
]
375-
```

0 commit comments

Comments
 (0)