Prepare scrapix for production use

[bidoubiwa]

Context

In the current state of this project, the scraping works and does its job as intended. Nonetheless, some essential parts are missing for this repository to become a tool that we advertise and are comfortable using in production.

First step: MVP

Code maintenance

• Use typescript Transform repository into typescript #15
• Rename some wrongly named variables (headless) (WIP)
• Add a linter (eslint, yamllint) Add linting #19
• Add a playground Add a playground #17
• Add basic tests Add basic tests #16
• Fix the trailing slash bug (WIP)
• Remove unnecessary features (WIP)
• Remove unfinished custom scrapper Remove the unfinished custom scraper #27
• create issue on custom scrapper as possible future enhancement Create custom scraper #18

Features

• User-agent definition "Meilisearch JS (v.X.X.X); Scrapix (vX.X.X)" Add use-agent on Meilisearch calls. #20
• Signal State (Webhook, Websocket, Logs, progress route... whatever) Being able to follow the crawling process #5

Repository

• Change the name (think about it)
• More intuitive explanation on usage in the README.md Improvement the README #21
• Simple explanation of how to run the project and launch its tests in the contributing guide Create CONTRIBUTING.md #22
• Adding the required files and tools we usually have in our repos:
  • CI (tests, releases,...) Create testing and release CI #23
  • LICENSE file Create the LICENSE file #24
  • dependabot Adding a dependabot file #25
• docker + docker compose

Second step

• Finish README and CONTRIBUTING to our standards

Futur possibilities

• run multiple parsers at the same time
• Use browserless
• custom scraper
• Create strategy for a product focussed scraping
• Push a docker image to run the scrapper
• Create a web page to use the scraper without having to call the routes directly Create a console for starting the crawling #4
• Scrap code-samples