Azure Blob Storage file output plugin for Embulk

Embulk file output plugin stores files on Microsoft Azure Blob Storage

Overview

• Plugin type: file output
• Resume supported: no
• Cleanup supported: no

Configuration

First, create Azure Storage Account.

• account_name: storage account name (string, required)
• account_key: primary access key (string, required)
• container: container name (string, required)
• path_prefix: prefix of target keys (string, required) (string, required)
• file_ext: e.g. "csv.gz, json.gz" (string, required)

Auto create container

container will create automatically when container doesn't exists.

When a container was deleted, a container with same name cannot be created for at least 30 seconds. It's a service specification of Azure Blob Storage.

Example

out:
  type: azure_blob_storage
  account_name: myaccount
  account_key: myaccount_key
  container: my-container
  path_prefix: logs/csv-
  file_ext: csv.gz
  formatter:
    type: csv
    header_line: false
  encoders:
  - {type: gzip}

Build

$ ./gradlew gem  # -t to watch change of files and rebuild continuously

Test

$ ./gradlew test  # -t to watch change of files and rebuild continuously

To run unit tests, we need to configure the following environment variables.

When environment variables are not set, skip some test cases.

AZURE_ACCOUNT_NAME
AZURE_ACCOUNT_KEY
AZURE_CONTAINER
AZURE_CONTAINER_DIRECTORY (optional, if needed)

If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.

$ vi ~/Library/LaunchAgents/environment.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>my.startup</string>
  <key>ProgramArguments</key>
  <array>
    <string>sh</string>
    <string>-c</string>
    <string>
      launchctl setenv AZURE_ACCOUNT_NAME my-account-name
      launchctl setenv AZURE_ACCOUNT_KEY my-account-key
      launchctl setenv AZURE_CONTAINER my-container
      launchctl setenv AZURE_CONTAINER_DIRECTORY unittests
    </string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

$ launchctl load ~/Library/LaunchAgents/environment.plist
$ launchctl getenv AZURE_ACCOUNT_NAME //try to get value.

Then start your applications.