Skip to content

Commit 29c0de3

Browse files
PreetamPreetam
authored
Implement 0.1.0 (#65)
* Clean up 0.1.0 branch * 0.1.0 (#64)
1 parent 766c768 commit 29c0de3

File tree

2,447 files changed

+1529405
-2288
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,447 files changed

+1529405
-2288
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,5 @@ bin
44
pkg
55
cistern
66
!vendor/*
7+
cistern.json
8+
data/

‎.gitmodules

Lines changed: 0 additions & 21 deletions
This file was deleted.

‎Gopkg.lock

Lines changed: 39 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Gopkg.toml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
2+
# Gopkg.toml example
3+
#
4+
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
5+
# for detailed Gopkg.toml documentation.
6+
#
7+
# required = ["github.com/user/thing/cmd/thing"]
8+
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
9+
#
10+
# [[constraint]]
11+
# name = "github.com/user/project"
12+
# version = "1.0.0"
13+
#
14+
# [[constraint]]
15+
# name = "github.com/user/project2"
16+
# branch = "dev"
17+
# source = "github.com/myfork/project2"
18+
#
19+
# [[override]]
20+
# name = "github.com/x/y"
21+
# version = "2.4.0"
22+
23+
[[constraint]]
24+
name = "github.com/aws/aws-sdk-go"
25+
version = "=1.10.14"

‎LICENSE

Lines changed: 16 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,19 @@
1-
Copyright (c) 2014 Preetam Jinka
2-
All rights reserved.
1+
Copyright (c) 2017 Cistern Authors
32

4-
Redistribution and use in source and binary forms, with or without modification,
5-
are permitted provided that the following conditions are met:
3+
Permission is hereby granted, free of charge, to any person obtaining a copy
4+
of this software and associated documentation files (the "Software"), to deal
5+
in the Software without restriction, including without limitation the rights
6+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
7+
copies of the Software, and to permit persons to whom the Software is
8+
furnished to do so, subject to the following conditions:
69

7-
1. Redistributions of source code must retain the above copyright notice, this
8-
list of conditions and the following disclaimer.
10+
The above copyright notice and this permission notice shall be included in all
11+
copies or substantial portions of the Software.
912

10-
2. Redistributions in binary form must reproduce the above copyright notice,
11-
this list of conditions and the following disclaimer in the documentation and/or
12-
other materials provided with the distribution.
13-
14-
3. Neither the name of the copyright holder nor the names of its contributors
15-
may be used to endorse or promote products derived from this software without
16-
specific prior written permission.
17-
18-
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19-
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20-
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21-
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
22-
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
23-
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
24-
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
25-
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26-
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
27-
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
SOFTWARE.

‎README.md

Lines changed: 186 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,190 @@
1-
<img src='https://cloud.githubusercontent.com/assets/379404/12572133/3a54ee4e-c3b6-11e5-9756-3e13617b5970.png' width='150px'/>
2-
3-
[![Circle CI](https://circleci.com/gh/Cistern/cistern.svg?style=svg&circle-token=d06bfb21101f58f7492cc3b2c118415c6f3e9b0d)](https://circleci.com/gh/Cistern/cistern) [![Join the chat at https://gitter.im/Preetam/cistern](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Preetam/cistern?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
1+
<p align="center">
2+
<img src='https://user-images.githubusercontent.com/379404/28300304-b38a05b2-6b4c-11e7-9e80-19bc3c41fa0b.png' width=400/>
3+
</p>
44

5+
<p align="center">
56
Cistern is a network flow collector.
7+
</p>
8+
9+
![experimental](https://img.shields.io/badge/status-experimental-orange.svg)
10+
[![Join the chat at https://gitter.im/Preetam/cistern](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Preetam/cistern?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
611

7-
License
812
---
9-
BSD (see LICENSE)
13+
14+
###### Supported sources
15+
16+
* CloudWatch Logs
17+
* VPC Flow Logs
18+
19+
Coming soon:
20+
* JSON CloudWatch Logs events
21+
* sFlow v5
22+
23+
## Getting started
24+
25+
Cistern has no external dependencies.
26+
27+
```
28+
Usage of ./cistern:
29+
-api-addr string
30+
API listen address (default "localhost:2020")
31+
-config string
32+
Path to config file (default "./cistern.json")
33+
-data-dir string
34+
Data directory (default "./data/")
35+
```
36+
37+
38+
39+
#### Config file
40+
41+
The config file has two main options:
42+
43+
* cloudwatch_logs: A list of CloudWatch Logs log groups to consume. See [VPC Flow Logs](#vpc-flow-logs) for more details.
44+
* retention: The retention of events in days.
45+
46+
```json
47+
{
48+
"cloudwatch_logs": [],
49+
"retention": 3
50+
}
51+
```
52+
53+
### VPC Flow Logs
54+
55+
You can specify the flow log groups to consume in the config file.
56+
In the `cloudwatch_logs` section, add an object for each log group
57+
with the name and `flowlog: true`.
58+
59+
**Example**
60+
61+
```json
62+
{
63+
"cloudwatch_logs": [
64+
{
65+
"name": "flowlogs",
66+
"flowlog": true
67+
}
68+
],
69+
"retention": 3
70+
}
71+
```
72+
73+
#### Credentials
74+
75+
Cistern will try to use AWS credentials from the following locations:
76+
77+
* The environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
78+
* The Shared Credentials file (~/.aws/credentials)
79+
* EC2 Instance Role Credentials
80+
81+
To specify the region, set the AWS_REGION environment variable.
82+
83+
## Querying
84+
85+
You can query Cistern using the API or the CLI, which uses the API.
86+
Most of the logic is built into the API, so it's not a bad idea to
87+
learn how the API works in order to make sense of the CLI.
88+
89+
### API
90+
91+
**POST /collections/:collection/compact**
92+
93+
This endpoint enforces the retention policy for the collection.
94+
95+
**POST /collections/:collection/query**
96+
97+
This endpoint queries events within a collection.
98+
99+
The query endpoint accepts a **Query** object, which has the following
100+
syntax:
101+
102+
```js
103+
Query: {
104+
/* All of the following are optional. */
105+
"time_range": TimeRange
106+
"columns": []Column
107+
"group_by": []string
108+
"filters": []Filter
109+
"point_size": integer
110+
"order_by": []string
111+
"limit": integer
112+
"descending": bool
113+
}
114+
115+
TimeRange : {
116+
"start": string
117+
"end": string
118+
}
119+
120+
Column: {
121+
"name": string
122+
"aggregate": string
123+
}
124+
125+
Filter: {
126+
"column": string
127+
"condition": string
128+
"value": *
129+
}
130+
```
131+
132+
#### Filters
133+
134+
Filters are applied as the first stage of query execution. A filter requires
135+
a column name, a condition, and a value for the condition. The supported
136+
conditions are:
137+
138+
* **eq**: equal
139+
* **neq**: not equal
140+
141+
If multiple filters are specified, they are applied in an "AND" condition.
142+
143+
#### Generating time series
144+
145+
Time series can be generated by providing a nonzero `point_size`. This will
146+
automatically group events by time ranges determined by the provided point size.
147+
148+
### CLI
149+
150+
```
151+
Usage of ./cistern-cli:
152+
-address string
153+
Cistern node address (default "http://localhost:2020")
154+
-collection string
155+
Collection to query
156+
-columns string
157+
Comma-separated list of columns to aggregate.
158+
Example: 'sum(bytes), sum(packets)'
159+
-descending
160+
Sort in descending order.
161+
-end int
162+
End Unix timestamp
163+
-filters string
164+
Comma-separated list of filters.
165+
Filters have the format '<column> <condition> <value>'.
166+
Possible conditions are [eq,neq].
167+
Values have to be valid JSON values.
168+
Example: 'dest_address neq "172.31.31.192" , packets eq 3'
169+
-group string
170+
Comma-separated list of fields to group by.
171+
Example: 'source_address, dest_address'
172+
-limit int
173+
Maximum number of events to return.
174+
-order-by string
175+
Comma-separated list of columns to order by.
176+
Providing multiple columns means the results are ordered
177+
by the first column, then the next, etc.
178+
-point-size duration
179+
Point size of time series. 0 means series will not be generated.
180+
-start int
181+
Start Unix timestamp
182+
-version
183+
Show version and exit.
184+
```
185+
186+
The CLI prints the output of the API response in JSON format.
187+
188+
## License
189+
190+
MIT (see LICENSE)

‎_test/portscan_flowlog.txt.gz

1.26 KB
Binary file not shown.

0 commit comments

Comments
 (0)