Merge pull request #9 from snowplow-incubator/remote-tunnel

Add ability to support ngrok tunnels

Author: miike

.gitignore

@@ -1,4 +1,5 @@
 .DS_Store
 .env
 volume/
-grafana/*
+grafana/*
+tunnel/ngrok.yml

README.md

@@ -15,6 +15,7 @@ Snowplow Local is designed to provide a fast, easy to use local development envi
 - Test new or existing versions of the Snowplow pipeline
 - Write enriched data to remote destinations (including S3, GCS etc)
 - Test and validate Snowbridge configurations
+- Send events remotely from another machine to your local pipeline (via `--profile tunnel`)
 
 ## Licensing
 
@@ -46,6 +47,8 @@ A user interface listens to events that have been enriched (good, bad and incomp
 
 A basic website that can be used to fire test events can be accessed at [http://localhost:8082](http://localhost:8082). This is useful for testing your pipeline and ensuring that events are being collected correctly.
 
+Although test events can be fired locally you can also send events from another remote machine using the `--tunnel` profile. See the 'Tunneling' section below.
+
 ## Differences to a production pipeline
 
 This software is not designed to be run in production or at high volume and is instead designed for a local experience.
@@ -128,6 +131,10 @@ Logs for components that write them (e.g., collector, enrich) are written to Clo
 
 The storage for Grafana is also mounted as a volume so you can persist any dashboards and data across restarts, rather than losing them on reboot!
 
+## Tunneling
+
+By using the `--profile tunnel` flag this will start a ngrok tunnel locally that enables traffic forwarding from an arbitrary URL to your local collector. You can configure this forwarding URL and other settings in `tunnel/ngrok.yml` by copying over the example file - `ngrok.yml.example`. In addition you can configure security settings (e.g., only allowing traffic from certain IP ranges / CIDRs) by customising the `ngrok/policy.yml` file. For more configuration options see the ngrok agent config documentation [here](https://ngrok.com/docs/agent/config/v3/).
+
 ## Gotchas
 
 ### Snowflake loader

docker-compose.yml

@@ -289,4 +289,20 @@ services:
     container_name: "local-failed-ui"
     image: "snowplow/failed-events-ui:0.0.1"
     ports:
-      - "3001:3000"
+      - "3001:3000"
+
+  ngrok:
+    container_name: ngrok-tunnel
+    image: ngrok/ngrok:latest
+    profiles: [tunnel]
+    restart: unless-stopped
+    command:
+      - "start"
+      - "--all"
+      - "--config"
+      - "/etc/ngrok.yml"
+    volumes:
+      - "./tunnel/ngrok.yml:/etc/ngrok.yml"
+      - "./tunnel/policy.yml:/etc/policy.yml"
+    ports:
+      - "4040:4040"

tunnel/ngrok.yml.example

@@ -0,0 +1,12 @@
+version: 3
+
+agent:
+  authtoken: ""
+
+endpoints:
+  - name: collector
+    url: https://snowplow.ngrok.app
+    upstream:
+      url: http://localhost:8080
+    traffic_policy_file: /etc/policy.yml
+

tunnel/policy.yml

@@ -0,0 +1,7 @@
+on_tcp_connect:
+  - actions:
+      - type: restrict-ips
+        config:
+          enforce: false
+          allow:
+            - 1.1.1.1/32