Improve NY Trial Court Scraping Logic to Handle Backdated Opinions

[flooie]

NY Trial courts often post their opinions well after the official publish date. This creates a challenge in our current scraping logic, which stops fetching once it encounters a set number of already-seen opinions.

What is happening

Opinions on the page are sorted chronologically by publish date.
Our scraper stops once it sees ~5–10 opinions it has already indexed (to avoid redundant requests and be a respectful scraper).
When a newly posted opinion is backdated (e.g., posted in April but dated February), it can fall far enough down the list that it’s missed entirely.

What can we do?

Add logic to detect and continue fetching beyond the seen-opinion threshold if newer HTML content is being served (e.g., based on publication date vs scrape timestamp).
Perhaps we should add a flag here in juriscraper to run a full crawl all the time? Or we should limit how much is identified. If we pulled less back perhaps it wouldnt be an issue

@grossir thoughts?

[grossir]

Currently the back scrape iterable creation is not dynamic, but we could easily fix that

juriscraper/juriscraper/opinions/united_states/state/nytrial.py

Lines 34 to 37 in 3703d9f

	date_keys = rrule(
	MONTHLY, dtstart=self.first_opinion_date, until=date(2023, 12, 30)
	)
	self.back_scrape_iterable = [i.date() for i in date_keys]

And then set a cron job on courtlistener's side to backscrape the past X months; maybe at the end of each week?