[Bug]: tests have diverged from code base

[stevenh]

crawl4ai version

v0.5.0.post1

Expected Behavior

Developers should be able to run the tests to confirm that any changes made don't introduce regressions.

I've fixed all of the tests and the bugs associated #891 so raising this to track the issue and breakdown of that draft into more consumable PRs.

Current Behavior

The tests have diverged from the code base and so don't fail even without any changes. Also some of the tests aren't correctly discovered by pytest so don't attempt to run.

Is this reproducible?

Yes

Inputs Causing the Bug

Just trying to run the tests, not specific to inputs.

Steps to Reproduce

Try to run the tests in VS Code.

Code snippets

N/A

OS

macOS

Python version

3.12

Browser

Chrome

Browser version

134.0.6998.166

Error logs & Screenshots (if applicable)

No response

[stevenh]

I'd like to use sub issues to break down and track the split of all the fixes, but that requires triage permission as detailed here.

I used the precursor task list for similar work on testcontainers/testcontainers-go#2685 which was very successful.

In the meantime I'll use a checklist to start the breakdown now I have the full test suite working as expected in this draft #891

[stevenh]

• Make parameters defaulted to None Optional
• Fix serialization typing and support None input
• Add missing abstract methods to AsyncCrawlerStrategy
• Correct Logger type to AsyncPlaywrightCrawlerStrategy
• Fix task management in _crawl_web which was racy
• Fix error message handling in log_consol
• Fix default logger in AsyncPlaywrightCrawlerStrategy
• Fix download_path calculation in _handle_download
• Fix robust_execute_user_script handling of scripts with var or const declarations
• Fix markdown storage in cache which was missing fields breaking cached results
• Fix processing of crawl returns which had inconsistent handling of return types
• Fix BrowserConfig and CrawlerRunConfig loading from kwargs
• Fix inalid screenshot arg to aprocess_html
• Fix content extraction when given NoExtractionStrategy
• Fix incorrect call in aclear_cache
• Fix legacy import paths
• Fix BM25Okapi implementation causing no matches due to zero scores
• Fix inconsistent handling of HEAD tags during extraction
• Fix split at 150 chars not handling exact match
• Fix exception when no body found
• Fix missing initialisation of RelevantContentFilter
• Fix missing markdown parameter to ScrapingResult constructor
• Fix ascrap by calling scrap instead of _scrap
• Fix image desc use and clean text so that Web and LLM scrapers match
• Fix failures due invalid attributes on unexpected types during HTML parsing
• Fix failure when parsing image height and width when not present
• Add missing json import to amazon crawler
• Fix incompatible amazon crawler run method declaration
• Use transport to allow direct test of service without docker, improving developer experience when debugging
• Add crawl_async to improve type safety
• Fix crawl stream handling on non 200 OK responses
• Add missing imports for CosineStrategy
• Fix exception when no llm_config in generate_schema
• Fix missing meta in CrawlerHub
• Remove unused parameter from take_screenshot
• Add non legacy fields to legacy DB table creation so that test run order doesn't matter
• Fix return type for get_cached_url
• Fix doc manager saving
• Fix extraction_strategy access when undefined in WebCrawler.run
• Fix invalid defaults for screenshot_data and extracted_content in WebCrawler.run
• Raise a ValueError when warmup requested but not ready so user knows the crawl failed
• Fix screenshot parameter type for WebCrawler
• Fix media, links and metadata output in WebCrawler
• Fix parameter defaults for MarkdownGenerationResult
• Export markdown_result from StringCompatibleMarkdown needed for caching
• Fix MediaItem constructor handling for width
• Fix Media, Links and ScrapingResult use of shared list and dict instead of Field.
• Add missing markdown field to ScrapingResult
• Fix return type of get_next_proxy and add_proxies
• Fix parser name passed to BeautifulSoup in format_html
• Correctly handle CrawlRequest decode and checks in server
• Fix parameter order for handle_markdown_request to allow correct config type
• Add validation of stream parameter to server API endpoints
• Use status constants for API errors.
• Update docker-compose comment to english
• Use cache_mode instead of bypass_cache to avoid deprecation failure
• Update examples to use cache_mode instead of bypass_cache
• Update setuptools to handle new license format without conflicting with pyproject.toml
• Remove deprecated License classifier which was conflicting with the license, Apache vs MIT
• Pass user_data_dir as string not Path to match function type hint
• Fix invalid entry in potential_sources due to missing comma
• Return Self from aenter so typing hinting validates correctly
• Fix raw protocol content
• Fix DB request retry when table has been removed by aflush_db
• Add missing packages
• Fix get_base_domain to handle ports
• Fix task clean up in filter chains