Skip to content

Conversation

rvantunes
Copy link

Context

We need to read JSONL files with complex structs, including many data types like array, object, datetime, string, etc...
The current implementation offers jsonl_type_coercion_strategy to decide how the tap is going to infer the schema. The problem is that none of the coercion strategies works for our use case. Besides that, the jsonl_sampling_strategy only supports first which does not work for our use case.

This PR implements a possible solution, by adding a new parameter named schema. This parameter would be parsed by Meltano SDK, see here.

If the user decides to specify a fixed schema, the tap does not need to create the schema dynamically, ignoring the coercion strategy.

Changes

  • Adding a new parameter to specify schema
  • Updating documentation
  • Fixing typos
  • Adding IDE files in .gitignore

@rvantunes rvantunes changed the title Adding support for schema feat(tap): Adding support for schema Sep 28, 2023
@leonardoeloy
Copy link

leonardoeloy commented Sep 28, 2023

@visch this approach seemed to be the best solution when we have full control of the schema

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants