tap-stackexchange

Singer tap for the StackExchange API.

Built with the Meltano SDK for Singer Taps and Targets.

Capabilities

• sync
• catalog
• state
• discover

Settings

Setting	Required	Default	Description
key	False	None	Pass this to receive a higher request quota
site	False	stackoverflow.com	StackExchange site
tags	False	None	Question tags
start_date	False	None	The earliest record date to sync

A full list of supported settings and capabilities is available by running: tap-stackexchange --about

Custom filter

The StackExchange API supports a number of custom filters that can be used to include or exclude certain fields from the response objects. This application has a baked-in filter with the following parameters:

• include

  Parameter	Description
  question.comment_count	The number of comments on the question
  tag.last_activity_date	The date of the last activity on the tag

• unsafe=false

Update the baked-in filter

To update the baked-in filter, edit the FILTER_ID constant in tap_stackexchange/tap.py.

To generate a new filter, use the Try It button on the StackExchange API documentation, using the baked-in filter as the base parameter.

Use a custom filter

To use a custom filter, set the filter setting to the filter ID. Note that if you use a custom filter, you will need to use a custom catalog that includes the fields you want to sync. That is you will need to

1. Write the default catalog to a file: tap-stackexchange --discover > catalog.json
2. Edit the catalog file to include the fields not included by the default API filter
3. Run the tap with the custom catalog: tap-stackexchange --config config.json --catalog catalog.json

Installation

pipx install git+https://github.com/edgarrmondragon/tap-stackexchange.git

Source Authentication and Authorization

Register a new application on Stack Apps and copy the generated key.

Usage

You can easily run tap-stackexchange by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-stackexchange --version
tap-stackexchange --help
tap-stackexchange --config CONFIG --discover > ./catalog.json

Initialize your Development Environment

pipx install poetry
poetry install

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Your project comes with a custom meltano.yml project file already created. Open the meltano.yml and follow any "TODO" items listed in the file.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-stackexchange
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-stackexchange --version

# OR run a test `elt` pipeline:
meltano elt tap-stackexchange target-sqlite --job_id=stackexchange-sqlite

# Runtime configuration
TAP_STACKEXCHANGE__LOAD_SCHEMA=dragon_ball_gt \
TAP_STACKEXCHANGE_SITE=anime \
TAP_STACKEXCHANGE_TAGS='["dragon-ball-gt"]' \
meltano elt tap-stackexchange target-sqlite

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.