-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Madewithlau #1068
Update Madewithlau #1068
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This generally looks great to me, thank you @mlduff. Only one change required:
Either:
- Add some test coverage for
_get_url_slug
or - Relocate
_get_url_slug
to themadewithlau.py
scraper module (removing the re-use fromgoustojson.py
)
@jayaddison Test should be added - let me know if you want more tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mlduff - no need for the additional test coverage at the moment; it's mainly intended to help by providing a known place where we can add test coverage if the expectations of different scrapers diverge.
@mlduff this has been included and released in |
Ah, a correction, sorry - this is included in v14.56.0 but not included in v15.0.0-rc3 (since the v15 branch does not currently include scrapers that require more than the initial HTTP network request to retrieve HTML). |
Looks like madewithlau.com underwent some changes in their website. The new website uses a tRPC API to load data after the initial page load. There is some fairly complex cross-referencing of returned data, hence the complexity of this PR.
I also added a utility to get the URL slug.
Resolves #1015