[Bug] Votes scraper not pulling in most recent vote, until I cleared cache

[sseshan7]

Hi!

I had an up-to-date cache of 116th congress 2020 session votes as of Sunday June 7. 2020.
ran
./run votes --congress=116 --session=2020
this morning (Tuesday June 9) and it was missing a senate vote on June 8th.
The vote was present in the xml file when I called the script.

Once I cleared the votes cache, it pulled correctly.
If this is a bug, I'll go ahead and try to fix this.
Thanks!

[JoshData]

I haven't had a problem. It could be a bug, but it might have just been some sort of fluke.

[jerrywithaz]

I just experienced the same issue for the current congress and year (117, 2022)

[tmountjr]

I've seen the same thing in the current session (119th, 2025, session 1). The cache seems locked at 315 votes (around April 10th or 11th, 102 for the House and 213 for the Senate) but when I hit the XML files manually today there's 387 listed (129 for the House and 258 for the Senate), with the most recent ones being yesterday. I'll do some debugging and see if I can figure out why this is because I don't really want to have to keep clearing out my cache every time I need to pull new votes.

EDIT: I'm stepping through votes.py now, it successfully found https://clerk.house.gov/evs/2025/ROLL_100.asp but on line 112:

votelinks = doc.xpath(
    "//a[re:match(@href, '%s')]" % link_pattern,
    namespaces={"re": "http://exslt.org/regular-expressions"})

it looks like it should pick up all the the links on the page but it only seems to grab the bottom three (100, 101, and 102). It gets 102 for example but stops and doesn't pick up 103:

<a href="http://clerk.house.gov/cgi-bin/vote.asp?year=2025&rollnumber=102">102</a>
<a href="http://clerk.house.gov/cgi-bin/vote.asp?year=2025&rollnumber=103">103</a>

After a little digging, though, it looks like the page that was downloaded by utils.download() only had those three in there. Doing even more digging showed that i have a file at cache/119/votes/2025/pages/house_100.html which only has those three entries. I would think the cache should have a TTL, right? If someone (like me) is doing a regular update of votes, wouldn't it be better not to cache the roll call list page, and instead compare the votes listed on a fresh version of that page to the votes that have already been downloaded?

As a workaround, I can probably use ./run votes --congress=119 --force --fast since there's a line in should_process() that indicates that --fast will "only download new votes or votes taken in the last three days."