[wallhaven.cc] add imageByFragment that picks up default saved filename

[spaceyuck]

Generated by an automatic template. Can be removed if not applicable.

Scraper type(s)

• imageByFragment
• imageByURL

Examples to test

SFW because NSFW needs account and API token

https://wallhaven.cc/w/9dqojx
https://wallhaven.cc/w/yxd8jk

Short description

When right-click saving a wallpaper from wallhaven, the default filename has a predictable structure ("wallhaven-."). This change adds image fragment scraping to allow automatically picking up IDs that are a clear and separable postfix to the filename.

[feederbox826]

This might be the weirdest regex I've seen yet, did you mean to do a-z0-9? and also escape the . for extension?

Also some of their older posts follow a seperate format, before they migrated. A strict regex might be possible + better

[feederbox826]

Will also put in draft/ potentially close as they have IQDB search for actual ByFragment. Will close if/when IQDB search implemented

https://wallhaven.cc/forums/thread/1169

[spaceyuck]

This might be the weirdest regex I've seen yet, did you mean to do a-z0-9? and also escape the . for extension?

\d and 0-9 are equivalent, so it's the same as a-z0-9. Point taken about the ., one of those I was stupid but it worked anyway kind of things, fixed now.

Also some of their older posts follow a seperate format, before they migrated. A strict regex might be possible + better

The oldest thing they have still uses an ID in the scheme of [a-z0-9]{6}, and I've checked some of my oldest from around 2014, they use the same format too. I can't find any info or example off another ID format still being in use, they might have migrated everything over to the current scheme.

I made it very restrictive explicitly to avoid false positives, it really should only match file names that have 6 characters / digits at the end, before the extension, with some kind of separator before.

IQDB search

Huh, I've never actually noticed that feature. I've looked into it a bit, and it's not mentioned in the API docs. Playing around with it a bit, it's just a POST to their search endpoint , maybe it just works for API search too. Otherwise this will be a whole new thing, I already see XSRF tokens and Cloudflare cookies, plus the login requirement for NSFW would probably need credentials or a session cookie in the scraper.

I'll look into it more later, but this really might be a feature of it's own.

[feederbox826]

would have to be in python but imo would be leagues above trying to match filename

[spaceyuck]

Just an update after looking into it a bit:

• definitely not supported by API (405 on POST to /search)
• after playing with the site in Firefox a bit, Cloudflare might be present but optional, it does still seem to work without the magic Cloudflare cookie and Cloudflares magic Javascript blocked - but I may have missed something
• right now running into status 419 (session expired) errors even with session cookies set and sent
• in the worst case, this might need CDP - is CDP even supported for Python? Might also give cloudscraoer a try, IAFD scrapers seems to use it

[spaceyuck]

Current broken state sequestered into its own subbranch wallhaven-imageByFragtment-iqdb, can't get it to work right now.

[feederbox826]

Just an update after looking into it a bit:

• definitely not supported by API (405 on POST to /search)
• after playing with the site in Firefox a bit, Cloudflare might be present but optional, it does still seem to work without the magic Cloudflare cookie and Cloudflares magic Javascript blocked - but I may have missed something
• right now running into status 419 (session expired) errors even with session cookies set and sent
• in the worst case, this might need CDP - is CDP even supported for Python? Might also give cloudscraoer a try, IAFD scrapers seems to use it

405 is method not allowed, maybe PUT instead of POST? but hm

cdp is supported on python but it's a lot weirder

[feederbox826]

Thanks for your hard work, I don't think it's quite worth struggling through cloudflare just for IQDB, there is a standalone IQDB.org which is what it's probably based on. I'll merge it since it's better than nothing and we'll just have to wait for IQDB api to eventually be opened up