Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: content/metadata file splitter #3

Open
Uinelj opened this issue Dec 8, 2021 · 1 comment · Fixed by #6
Open

Feature: content/metadata file splitter #3

Uinelj opened this issue Dec 8, 2021 · 1 comment · Fixed by #6

Comments

@Uinelj
Copy link
Member

Uinelj commented Dec 8, 2021

Since the new pipeline outputs a unique .jsonl file containing both textual content and metadata, working exclusively with textual data may be less comfortable:

  1. Forced download of metadata that will be discarded
  2. Forced usage of a jsonl-compatible parser in order to extract content

For now, we should provide a tool that splits up a given OSCAR Schema v2 file into a OSCAR Schema v1.2-compatible file.
@Uinelj Uinelj transferred this issue from oscar-project/ungoliant Jan 26, 2022
@Uinelj
Copy link
Member Author

Uinelj commented Feb 8, 2022

As a first step, extract a v1.1-compatible file.

@Uinelj Uinelj linked a pull request Feb 9, 2022 that will close this issue
Text extraction #6
Merged
@Uinelj Uinelj moved this from Selected to In Progress in OSCAR Feb 10, 2022
@Uinelj Uinelj moved this from In Progress to Selected in OSCAR Feb 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
OSCAR
Status: Selected
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Text extraction oscar-project/oscar-tools
1 participant
@Uinelj