[Bug]: playwright._impl._errors.TargetClosedError: BrowserType.launch: Target page, context or browser has been closed

[kivous]

crawl4ai version

0.5.0.post8

Expected Behavior

I am trying to execute AsyncWebCrawler().arun(with configs) with headless=True for multiple urls executed one after another, and it should return the markdown content of the url for all the urls given.

Current Behavior

I am trying to execute AsyncWebCrawler().arun(with configs) with headless=True for multiple urls executed one after another, and its failing with the mentioned error after the first run. In 0.5.0.post6 it works fine.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

Ubuntu

Python version

3.10.16

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[stevenh]

I have a fix for that which needs to broken out from #891.

The issue is the processing doesn't complete before returning due to missing task management.

[fireyanci]

TU ,i back 0.6

[fireyanci]

I think if LLM is used as an information extraction tool and API calls are made to LLM, it can increase concurrency because a single task takes too long

[aravindkarnam]

@kivous I believe this is same bug as #842. Therefore closing this as duplicate.

[zzmbulldog]

@kivous I believe this is same bug as #842. Therefore closing this as duplicate.

@aravindkarnam When i close browser artificially，the error happened. This is bug? version 0.5.0.post8