Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement content sniffing for HTML parsing #808

Merged
merged 2 commits into from
Mar 27, 2024
Merged

Implement content sniffing for HTML parsing #808

merged 2 commits into from
Mar 27, 2024

Commits on Mar 27, 2024

  1. Improve Content-Type parsing 

    Instead of looking for "html" substring, actually parse the MIME type
string. Don't use mime.ParseMediaType though as it doesn't handle
invalid duplicate parameters (e.g. "text/html; charset=UTF-8; charset=utf-8")
that occur in the wild.
    @WGH-
    WGH- committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    31f0876 View commit details
    Browse the repository at this point in the history

  2. Implement content sniffing for HTML parsing 

    Web pages can be served without Content-Type set, in which case
browsers employ content sniffing. Do the same here, in Colly.
    @WGH-
    WGH- committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    bad50ff View commit details
    Browse the repository at this point in the history