Consider running tap output into a target in CI

[Ry-DS]

In #93 I decided to run the tap thru a simple target target-sqlite to see if the new streams I added worked.
By doing this, I instantly found mistyped fields in the Events stream.
Example error:

CRITICAL '20730626430' is not of type 'integer', 'null'
CRITICAL 
CRITICAL Failed validating 'type' in schema['properties']['id']:
CRITICAL     {'type': ['integer', 'null']}
CRITICAL
CRITICAL On instance['id']:
CRITICAL     '20730626430'

I think this would be a great testing method to ensure our schemas are correctly typed.

Possibly a new step that runs:
poetry run tap-github --config c.json --catalog ca.json | poetry run target-sqlite -c sq.json and crashes if the exit code is non-zero.

Thoughts on implementing this?

[laurentS]

Possibly related https://gitlab.com/meltano/sdk/-/issues/259

[aaronsteers]

Hi, @laurentS and @Ry-DS. I agree running a test to include landing in target-sqlite or even target-jsonl is probably helpful for catching JSON Schema typing issues. This is fairly common for tap developers in general, I think, and I've heard of others passing data through target-jsonl exactly for this purpose.

There are SDK-based solutions as well, such as including an option for JSON Schema type validations against reach record when testing in pytest.