[Feature Request]: Remove GDPR Banners

[nikolakretschmer]

What needs to be done?

Create a CrawlerRunConfig option/feature to reliably remove GDPR / Cookie-Consent popups and banners from crawled sites.

What problem does this solve?

With these banners/popups present the resulting markdown will often contain content of the cookie popup and not the actual site content.

Target users/beneficiaries

EU users

Current alternatives/workarounds

I've tried every trick under the sun to remove these obnoxious popups. Injecting JS-Code, loading a chrome/firefox extension, remove_overlay_elements ... nothing has worked so far. Until this fix is implemented, I might have to switch to playwright and use some other parser for the markdown.

Proposed approach

I don't know. You guys are the wizards.

[hansvangent]

Yes this please... those damn cookie banners are really preventing 99% of websites to be crawled and it is really unusable like this.

[an0mal1a]

I hope they release a update with this function.

[stack-one-tech]

Up!

[iamtommydev]

up!

[s-ernst]

this! adds a lot of clutter to the markdown.

[ntohidi]

Hi everyone! Have you tried using the remove_overlay_elements? This JS script, located in the crawl4ai/js_snippet directory, is designed to enhance web scraping by improving content visibility and data extraction accuracy by removing these pop-ups?

To enable this functionality, you can set the remove_overlay_elements parameter to True in your CrawlerRunConfig:

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig

async def crawl_with_overlay_removal(url: str):
    config = CrawlerRunConfig(
        remove_overlay_elements=True,
        # Other configurations...
    )
    browser_config = BrowserConfig(headless=True)

    async with AsyncWebCrawler(browser_config=browser_config) as crawler:
        result = await crawler.crawl(url, config=config)
        return result.html

You also can set the magic parameter, which attempts automatic handling of overlays/popups, and simulate_user to simulate user interactions for anti-bot measures, both of which can be enabled in CrawlerRunConfig