Scrapix V2

[qdequele]

The idea is to create a new version of Scrapix on steroids.

PR: #101

This is the list of features and fixes it should include.

Features

P1

• Create custom scraper #18
• Create Puppeteer crawling backend option #108
• Create Cheerio crawling backend option #109
• Create Markdown scraping strategy #110
• Keep the previous settings #103
• Index the PDF files #104

P2

• Provide option to slow or rate limit requests #99
• Handle pages not found #48
• Load the sitemap as starter point for crawling.  #102
• Improve errors logs #34
• Handle pages not found #48

P3

• Retrieve page titles from meta tags #53
• Create Playwright crawling backend option #107
• Add possibility to exclude selectors #93
• Detect/handle localized pages #105
• Update Schema scraping strategy to handle microdata #106

Breaking

• Remove the useless headless option #112
• By default use cheerio instead of Puppeteer #113

Fix

• Throw error when redis server is not answering #56