1651 add clean extracted text method #1653
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grossir
reviewed
Nov 5, 2025
Contributor
grossir
left a comment
grossir
left a comment
There was a problem hiding this comment.
this is almost ready; some small style changes needed
| :param content: The scraped text | ||
| :return: Cleaned text | ||
| """ | ||
| content = content.replace("Page Proof Pending Publication\n", "") |
Contributor
There was a problem hiding this comment.
I picked up an existing opinion's text
and this will work as expected
text = "ost joined a suit brought against the Board under\nPage Proof Pending Publication\n the Administrative Procedure Act (APA). The complaint c"
text.replace("Page Proof Pending Publication\n", "")
'ost joined a suit brought against the Board under\n the Administrative Procedure Act (APA). The complaint c'
juriscraper/opinions/united_states/federal_appellate/scotus_slip.py
Outdated
Show resolved
Hide resolved
juriscraper/opinions/united_states/federal_appellate/scotus_slip.py
Outdated
Show resolved
Hide resolved
grossir
reviewed
Nov 7, 2025
Comment on lines
+150
to
+152
| cleaned_extracted_text = site.cleanup_extracted_text(extracted_content) | ||
|
|
||
| metadata_dict = site.extract_from_text(cleaned_extracted_text) |
Contributor
There was a problem hiding this comment.
I think site.extract_from_text(cleaned_extracted_text) should go before, since we may use some of the content we want to clean as reference to find the content we want to extract. site.cleanup_extracted_text will reduce the amount of text in the file, while site.extract_from_text may take advantage of that extra text to find stuff
Contributor
|
@Luis-manzur this needs a test - probably since it's a new feature a new test. Try to keep it minimal - in the same vein as extract from text - tests. please. |
…thod' into 1651-add-clean_extracted_text-method
for more information, see https://pre-commit.ci
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new post-extraction text cleanup mechanism to the codebase. The main addition is a
cleanup_extracted_textmethod, which allows for sanitizing plain text after it has been extracted from source documents. This enables removal of extraction artifacts and unwanted content, improving the quality of processed text. The method is implemented as a no-op in the abstract base class and is overridden with custom logic in a subclass for SCOTUS slip opinions. Additionally, the sample caller is updated to use the cleaned text for further processing.This PR addresses - #1651