Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentence splitter uses unbounded memory in -k mode #30

Open
kpu opened this issue Jun 12, 2021 · 0 comments · May be fixed by #31
Open

Sentence splitter uses unbounded memory in -k mode #30

kpu opened this issue Jun 12, 2021 · 0 comments · May be fixed by #31

Comments

@kpu
Copy link
Owner

kpu commented Jun 12, 2021

When used in -k mode, one would expect the sentence splitter to use a small amount of RAM, just enough to store a single line. However, it actually stores the entire split file before printing.

preprocess/moses/ems/support/split-sentences.perl

Line 119 in 344208c

$out .= &split_block($_,"");

@jelmervdl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change split_single_document to work on STDIN & STDOUT jelmervdl/preprocess
1 participant
@kpu