GitHub - datawrangling/gardenhose-microslurp: bootstrap scripts for getting a micro ec2 instance piping gardenhose to s3

datawrangling / gardenhose-microslurp Public

forked from matpalm/gardenhose-microslurp

Notifications You must be signed in to change notification settings
Fork 1
Star 5

bootstrap scripts for getting a micro ec2 instance piping gardenhose to s3

matpalm.com

5 stars 5 forks Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
README		README
bootstrap.sh		bootstrap.sh
process_yesterday.sh		process_yesterday.sh
start.sh		start.sh

Repository files navigation

simple set of scripts for slurping twitter gardenhose and uploading to s3 daily

restarts curl from gardenhose hourly to ensure dropped connections are reestablished from time to time

requires a s3 bucket to be created beforehand (should really roll into the script)

start a micro instance and ssh on and run...
$ yum install git
$ git clone git://github.com/matpalm/gardenhose-microslurp.git
$ cp gardenhose-microslurp/* .
$ ./bootstrap.sh

bootstrap sets some env vars (in ~/env_vars) for twitter uid, 
s3 creds and sets up some cron jobs