Skip to content

Commit 42eda59

Browse files
josephlewis42josephlewis42
authored
Initial Google Implementation (#5)
Initial implementation with Java.
1 parent 8b20ce0 commit 42eda59

25 files changed

+1311
-2
lines changed

‎.gitignore

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
## Prevent someone from accidentally including a GCP JSON key file
2+
*.json
3+
## Ignore active vim swap files
4+
*.swp
5+
*.iml
6+
.idea/
7+
8+
## Default gitignore from logstash
9+
*.gem
10+
*.rbc
11+
/.config
12+
/coverage/
13+
/InstalledFiles
14+
/pkg/
15+
/spec/reports/
16+
/spec/examples.txt
17+
/test/tmp/
18+
/test/version_tmp/
19+
/tmp/
20+
21+
## Specific to RubyMotion:
22+
.dat*
23+
.repl_history
24+
build/
25+
26+
## Documentation cache and generated files:
27+
/.yardoc/
28+
/_yardoc/
29+
/doc/
30+
/rdoc/
31+
*.log
32+
.gradle
33+
vendor/
34+
35+
## Environment normalization:
36+
/.bundle/
37+
/vendor/bundle
38+
/lib/bundler/man/
39+
40+
# for a library or gem, you might want to ignore these files since the code is
41+
# intended to run in multiple environments; otherwise, check them in:
42+
Gemfile.lock
43+
.ruby-version
44+
.ruby-gemset
45+
46+
# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
47+
.rvmrc

‎.travis.yml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
sudo: false
3+
language: ruby
4+
cache: bundler
5+
matrix:
6+
include:
7+
- rvm: jruby-9.1.13.0
8+
env: LOGSTASH_BRANCH=master
9+
- rvm: jruby-9.1.13.0
10+
env: LOGSTASH_BRANCH=6.x
11+
- rvm: jruby-9.1.13.0
12+
env: LOGSTASH_BRANCH=6.2
13+
- rvm: jruby-1.7.27
14+
env: LOGSTASH_BRANCH=5.6
15+
fast_finish: true
16+
install: true
17+
script: ci/build.sh
18+
jdk: oraclejdk8

‎CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
## 1.0.0
2+
- Initial implementation

‎CONTRIBUTING.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Contributing
2+
3+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
4+
5+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
6+
7+
It is more important to the community that you are able to contribute.
8+
9+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

‎CONTRIBUTORS

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
The following is a list of people who have contributed ideas, code, bug
2+
reports, or in general have helped logstash along its way.
3+
4+
Contributors:
5+
6+
* Google LLC
7+
* Eric Johnson - [email protected]
8+
* Joseph Lewis III - [email protected]
9+
10+
Note: If you've sent us patches, bug reports, or otherwise contributed to
11+
Logstash, and you aren't on the list above and want to be, please let us know
12+
and we'll make sure you're here. Contributions from folks like you are what make
13+
open source awesome.

‎Gemfile

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
source 'https://rubygems.org'
2+
gemspec
3+
4+
logstash_path = ENV['LOGSTASH_PATH'] || '../../logstash'
5+
use_logstash_source = ENV['LOGSTASH_SOURCE'] && ENV['LOGSTASH_SOURCE'].to_s == '1'
6+
7+
if Dir.exist?(logstash_path) && use_logstash_source
8+
gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
9+
gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
10+
end

‎LICENSE

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.

‎README.md

Lines changed: 96 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,96 @@
1-
# logstash-output-google_pubsub
2-
Logstash output for sending events to the Google Pub/Sub service
1+
# Logstash Output Google Pubsub
2+
3+
A Logstash plugin to upload log events to https://cloud.google.com/pubsub/[Google Cloud Pubsub].
4+
Events are batched and uploaded in the background for the sake of efficiency.
5+
Message payloads are serialized JSON representations of the events.
6+
7+
Example use-cases:
8+
9+
* Stream events to Dataproc via Pub/Sub for real-time analysis.
10+
* Forward events from an on-prem datacenter to the Logstash in the cloud.
11+
* Use Pub/Sub as an scalable buffer to even out event flow between processing steps.
12+
13+
Note: While this project is partially maintained by Google, this is not an official Google product.
14+
15+
## Documentation
16+
17+
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
18+
19+
- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
20+
- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
21+
22+
## Need Help?
23+
24+
Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
25+
26+
## Developing
27+
28+
### 1. Plugin Developement and Testing
29+
30+
#### Code
31+
- To get started, you'll need JRuby with the Bundler gem installed.
32+
33+
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
34+
35+
- Install dependencies
36+
```sh
37+
bundle install
38+
bundle exec rake vendor
39+
```
40+
41+
#### Test
42+
43+
- Update your dependencies
44+
45+
```sh
46+
bundle install
47+
bundle exec rake vendor
48+
```
49+
50+
- Run tests
51+
52+
```sh
53+
bundle exec rspec
54+
```
55+
56+
### 2. Running your unpublished Plugin in Logstash
57+
58+
#### 2.1 Run in a local Logstash clone
59+
60+
- Edit Logstash `Gemfile` and add the local plugin path, for example:
61+
```ruby
62+
gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
63+
```
64+
- Install plugin
65+
```sh
66+
bin/logstash-plugin install --no-verify
67+
```
68+
- Run Logstash with your plugin
69+
```sh
70+
bin/logstash -e 'filter {awesome {}}'
71+
```
72+
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
73+
74+
#### 2.2 Run in an installed Logstash
75+
76+
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
77+
78+
- Build your plugin gem
79+
```sh
80+
gem build logstash-filter-awesome.gemspec
81+
```
82+
- Install the plugin from the Logstash home
83+
```sh
84+
bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
85+
```
86+
- Start Logstash and proceed to test the plugin
87+
88+
## Contributing
89+
90+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
91+
92+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
93+
94+
It is more important to the community that you are able to contribute.
95+
96+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

‎Rakefile

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# encoding: utf-8
2+
require "jars/installer"
3+
require "fileutils"
4+
5+
task :default do
6+
system('rake -vT')
7+
end
8+
9+
task :vendor do
10+
exit(1) unless system './gradlew vendor'
11+
end
12+
13+
task :clean do
14+
["vendor/jar-dependencies", "Gemfile.lock"].each do |p|
15+
FileUtils.rm_rf(p)
16+
end
17+
end
18+

‎build.gradle

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
import java.nio.file.Files
2+
import static java.nio.file.StandardCopyOption.REPLACE_EXISTING
3+
4+
apply plugin: 'java'
5+
apply plugin: 'idea'
6+
apply plugin: 'maven-publish'
7+
8+
def CLOUD_PUBSUB_VERSION = '1.37.1'
9+
10+
repositories {
11+
mavenCentral()
12+
}
13+
14+
dependencies {
15+
compile "com.google.cloud:google-cloud-pubsub:${CLOUD_PUBSUB_VERSION}"
16+
17+
compile "com.fasterxml.jackson.core:jackson-core:2.1.3"
18+
compile "com.google.api:api-common:1.6.0"
19+
compile "com.google.api:gax:1.29.0"
20+
compile "com.google.api:gax-grpc:1.29.0"
21+
compile "com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.19.1"
22+
compile "com.google.api.grpc:proto-google-cloud-pubsub-v1:1.19.1"
23+
compile "com.google.api.grpc:proto-google-common-protos:1.12.0"
24+
compile "com.google.api.grpc:proto-google-iam-v1:0.12.0"
25+
compile "com.google.auth:google-auth-library-credentials:0.9.1"
26+
compile "com.google.auth:google-auth-library-oauth2-http:0.9.1"
27+
compile "com.google.auto.value:auto-value:1.4"
28+
compile "com.google.cloud:google-cloud-core:1.37.1"
29+
compile "com.google.cloud:google-cloud-core-grpc:1.37.1"
30+
compile "com.google.code.findbugs:jsr305:3.0.2"
31+
compile "com.google.code.gson:gson:2.7"
32+
compile "com.google.errorprone:error_prone_annotations:2.2.0"
33+
compile "com.google.guava:guava:20.0"
34+
compile "com.google.http-client:google-http-client:1.23.0"
35+
compile "com.google.http-client:google-http-client-jackson2:1.19.0"
36+
compile "com.google.protobuf:protobuf-java:3.6.0"
37+
compile "com.google.protobuf:protobuf-java-util:3.6.0"
38+
compile "com.google.protobuf:protobuf-lite:3.0.1"
39+
compile "commons-codec:commons-codec:1.3"
40+
compile "commons-logging:commons-logging:1.1.1"
41+
compile "io.grpc:grpc-auth:1.13.1"
42+
compile "io.grpc:grpc-context:1.13.1"
43+
compile "io.grpc:grpc-core:1.13.1"
44+
compile "io.grpc:grpc-netty-shaded:1.13.1"
45+
compile "io.grpc:grpc-protobuf:1.13.1"
46+
compile "io.grpc:grpc-protobuf-lite:1.13.1"
47+
compile "io.grpc:grpc-stub:1.13.1"
48+
compile "io.opencensus:opencensus-api:0.12.3"
49+
compile "io.opencensus:opencensus-contrib-grpc-metrics:0.12.3"
50+
compile "joda-time:joda-time:2.9.2"
51+
compile "org.apache.httpcomponents:httpclient:4.0.1"
52+
compile "org.apache.httpcomponents:httpcore:4.0.1"
53+
compile "org.threeten:threetenbp:1.3.3"
54+
}
55+
56+
task generateGemJarRequiresFile {
57+
doLast {
58+
File jars_file = file('lib/logstash-output-google_pubsub_jars.rb')
59+
jars_file.newWriter().withWriter { w ->
60+
w << "# AUTOGENERATED BY THE GRADLE SCRIPT. DO NOT EDIT.\n\n"
61+
w << "require \'jar_dependencies\'\n"
62+
configurations.runtime.allDependencies.each {
63+
w << "require_jar(\'${it.group}\', \'${it.name}\', \'${it.version}\')\n"
64+
}
65+
}
66+
}
67+
}
68+
69+
task vendor {
70+
doLast {
71+
String vendorPathPrefix = "vendor/jar-dependencies"
72+
configurations.runtime.allDependencies.each { dep ->
73+
File f = configurations.runtime.filter { it.absolutePath.contains("${dep.group}/${dep.name}/${dep.version}") }.singleFile
74+
String groupPath = dep.group.replaceAll('\\.', '/')
75+
File newJarFile = file("${vendorPathPrefix}/${groupPath}/${dep.name}/${dep.version}/${dep.name}-${dep.version}.jar")
76+
newJarFile.mkdirs()
77+
Files.copy(f.toPath(), newJarFile.toPath(), REPLACE_EXISTING)
78+
}
79+
}
80+
}
81+
82+
vendor.dependsOn(generateGemJarRequiresFile)

‎ci/build.sh

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
# version: 1
3+
########################################################
4+
#
5+
# AUTOMATICALLY GENERATED! DO NOT EDIT
6+
#
7+
########################################################
8+
set -e
9+
10+
echo "Starting build process in: `pwd`"
11+
source ./ci/setup.sh
12+
13+
if [[ -f "ci/run.sh" ]]; then
14+
echo "Running custom build script in: `pwd`/ci/run.sh"
15+
source ./ci/run.sh
16+
else
17+
echo "Running default build scripts in: `pwd`/ci/build.sh"
18+
bundle install
19+
bundle exec rake vendor
20+
bundle exec rspec spec
21+
fi

‎ci/setup.sh

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
# version: 1
3+
########################################################
4+
#
5+
# AUTOMATICALLY GENERATED! DO NOT EDIT
6+
#
7+
########################################################
8+
set -e
9+
if [ "$LOGSTASH_BRANCH" ]; then
10+
echo "Building plugin using Logstash source"
11+
BASE_DIR=`pwd`
12+
echo "Checking out branch: $LOGSTASH_BRANCH"
13+
git clone -b $LOGSTASH_BRANCH https://github.com/elastic/logstash.git ../../logstash --depth 1
14+
printf "Checked out Logstash revision: %s\n" "$(git -C ../../logstash rev-parse HEAD)"
15+
cd ../../logstash
16+
echo "Building plugins with Logstash version:"
17+
cat versions.yml
18+
echo "---"
19+
# We need to build the jars for that specific version
20+
echo "Running gradle assemble in: `pwd`"
21+
./gradlew assemble
22+
cd $BASE_DIR
23+
export LOGSTASH_SOURCE=1
24+
else
25+
echo "Building plugin using released gems on rubygems"
26+
fi

0 commit comments

Comments
 (0)