[Feature Request]: Add progress_callback support to AdaptiveCrawler.digest()

[blghtr]

What needs to be done?

Update the AdaptiveCrawler.digest() method to accept an optional progress_callback argument. This callback should be invoked periodically during the crawling loop (e.g., after each batch of pages is processed or when the confidence score updates).
Proposed signature:

async def digest(
    self, 
    start_url: str, 
    query: str, 
    progress_callback: Optional[Callable[[CrawlState], Awaitable[None]]] = None
) -> CrawlState:

What problem does this solve?

The adaptive crawling process can be long-running (crawling multiple pages, generating embeddings, analyzing relevance). Currently, the caller has zero visibility into the progress until the entire digest() calls returns.

This makes it difficult to:

• Show progress bars to end-users (e.g., "Crawled 5/20 pages").
• Display the current "AI Confidence" score in real-time.
• Detect if the crawler has stalled or is stuck on a specific subdomain.

Target users/beneficiaries

• Developers building interactive UI/CLI tools on top of crawl4ai.
• MCP servers that need to stream progress updates to LLM agents.

Current alternatives/workarounds

• Polling: Not possible because digest() is an async blocking call that holds control until finished.
• Monkey-patching: We considered injecting a callback into the loop dynamically, but it's brittle and relies on internal implementation details.

Proposed approach

Inject the callback invocation inside the main while loop in digest():

            # adaptive expansion
            depth = 0
            while depth < self.config.max_depth:
                # ... existing logic ...

                # Calculate confidence
                confidence = await self.strategy.calculate_confidence(self.state)
                
                # ... existing logic ...
                
                # >>> NEW: Report progress
                if progress_callback:
                    await progress_callback(self.state)
                
                # ... continue loop ...

This would likely require minimal changes but add significant value for integration.