Fix URL import returning empty results and improve image download compatibility #4357
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…patibility URL Import Fix: - Changed from requests.get() + scrape_html() to scrape_html(online=True) - This lets recipe_scrapers handle HTTP requests with proper User-Agent, cookies, and site-specific logic - Fixes imports from sites that inject JSON-LD schema via JavaScript, which was causing empty results while the bookmarklet worked fine Image Download Fix: - Updated User-Agent from outdated Firefox 86 (2021) to modern Chrome - Added proper Accept headers for image content types - Added Referer header for sites that validate referrers - Improves compatibility with CDNs and image hosting services Co-Authored-By: Claude <[email protected]>
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes two related issues with recipe importing:
1. URL Import Fix (line ~2394)
Problem: "Import from URL" on
/recipe/importpage returns empty results (200 with no recipe data), while the bookmarklet import works fine for the same URLs.Root Cause: The existing code used
requests.get()to fetch page HTML, then passed it toscrape_html(). Many modern recipe sites (Food Network, AllRecipes, etc.) inject their JSON-LD schema data via JavaScript. Sincerequests.get()only fetches raw HTML without executing JavaScript, the schema data was missing andscrape_html()couldn't find any recipe data.The bookmarklet worked because it captures
document.documentElement.outerHTMLfrom the browser - the fully-rendered DOM after JavaScript execution.Fix: Use
scrape_html(html=None, org_url=url, online=True, supported_only=False)which lets recipe_scrapers handle the HTTP request internally with:2. Image Download Fix (line ~1657)
Problem: Recipe images failing to download from some CDNs and image hosting services.
Root Cause: The User-Agent was set to Firefox 86 (released February 2021), which some servers reject or serve different content to.
Fix: Updated to modern Chrome User-Agent with proper headers:
Acceptheader with image MIME typesAccept-LanguageheaderRefererheader for sites that validate referrersTest Plan
Changes
cookbook/views/api.py: 2 locations modified (10 insertions, 7 deletions)🤖 Generated with Claude Code