[Feature Request]: Make generate_schema resilient across multi-page HTML samples

[SohamKukreti]

crawl4ai version

v0.7.8

Expected Behavior

generate_schema should propose stable selectors that work across multiple pages with varying DOM positions (e.g., table rows shifting). Given several representative HTML samples, it should prefer attribute/text-anchored selectors over fragile nth-child positions.

Current Behavior

generate_schema inspects a single HTML sample and may emit brittle selectors like table.pdp-sku-info tbody tr:nth-child(6) td:nth-child(2) a. On other product pages, the same field appears in nth-child(5) or nth-child(7), so extraction fails until a human finds a more stable selector (e.g., a[href*="/product?Manufacturer="]).

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

Any

Python version

3.10+

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Root Cause Analysis

Primary Root Cause: Single-Sample Design Without Cross-Page Validation

Location: crawl4ai/extraction_strategy.py:1280-1386

The generate_schema method signature accepts only a single HTML string:

@staticmethod
def generate_schema(
    html: str,  # ← Only ONE HTML sample
    schema_type: str = "CSS",
    query: str = None,
    target_json_example: str = None,
    ...
) -> dict:

Problem: The method has no mechanism to:

1. Accept multiple HTML samples from different pages
2. Test generated selectors against multiple pages
3. Detect when a selector is fragile (e.g., nth-child that varies)
4. Prefer stable selectors (attribute-based, text-anchored) over positional ones

Secondary Root Cause: LLM Prompt Instructions Are Insufficient

Location: crawl4ai/extraction_strategy.py:1362-1366

The prompt includes instructions to avoid fragile selectors:

user_message["content"] += """IMPORTANT: 
0/ Ensure your schema remains reliable by avoiding selectors that appear to generate dynamically...
2/ Every selector must refer to only one unique element... not fragile, meaning if we reload the page now or in the future, the selector should remain reliable.

Problem:

1. Vague guidance: "Not fragile" doesn't explicitly prohibit nth-child selectors
2. No examples of bad selectors: Prompt doesn't show what NOT to do (e.g., tr:nth-child(6))
3. No preference hierarchy: Doesn't instruct LLM to prefer attribute/text selectors over positional ones
4. Single-page context: LLM only sees one page, so it can't detect cross-page fragility

Evidence: Despite these instructions, LLMs still generate nth-child selectors because:

• They work on the provided sample
• LLM has no way to know they'll fail on other pages
• Instructions are advisory, not enforced

---

Contributing Factor: No Selector Robustness Scoring

The codebase has no mechanism to:

• Score selector stability: Attribute selectors > text-anchored > positional
• Detect common anti-patterns: nth-child, nth-of-type without stable anchors
• Suggest improvements: Auto-rewrite tr:nth-child(6) to tr:has(td:contains("Brand")) or attribute-based

---

Contributing Factor : Architecture Assumes Single-Page Schema Generation

Evidence from codebase:

• All examples generate schema from one page, then apply to multiple pages
• No API design for "generate schema that works across these 5 sample pages"
• Documentation doesn't mention multi-page considerations

Example from test_extract_pipeline.py:258-271:

# Use the first instance as sample
sample_html = lxml_html.tostring(parent_elements[0], encoding='unicode')
schema = JsonCssExtractionStrategy.generate_schema(html=sample_html, ...)
# Then apply to all URLs - but schema wasn't validated against them!

---

Corrective Actions (Immediate)

1. Enhance generate_schema to Accept Multiple HTML Samples

Proposed API:

@staticmethod
def generate_schema(
    html: Union[str, List[str]],  # ← Accept single or multiple
    schema_type: str = "CSS",
    query: str = None,
    target_json_example: str = None,
    validate_cross_page: bool = True,  # ← New parameter
    ...
) -> dict:

Implementation:

• If html is a list, pass all samples to LLM with instructions to find common selectors
• If validate_cross_page=True, test generated selectors against all samples
• Reject or rewrite selectors that don't work on all samples

2. Strengthen LLM Prompt with Explicit Anti-Patterns

Add to prompt:

CRITICAL SELECTOR RULES:
- NEVER use nth-child(N) or nth-of-type(N) unless absolutely necessary
- PREFER attribute selectors: a[href*="pattern"], [data-field="value"]
- PREFER text-anchored: td:contains("Brand"), th:contains("Price")
- If you must use positional, explain why in a comment
- If multiple HTML samples provided, selectors MUST work on ALL samples

Add Selector Robustness Scoring

New function:

def score_selector_robustness(selector: str) -> float:
    """
    Score selector from 0.0 (fragile) to 1.0 (robust).
    - Attribute selectors: 1.0
    - Text-anchored: 0.9
    - Class-based: 0.8
    - nth-child without anchor: 0.2
    """