Xandria Distributed Scraper

The system covered in this repositry is with regards to the scraper component of Xandria.

Installation

Install requirements:

pip3 install -r requirements.txt

Run:

celery -A tasks worker

uvicorn main:app --reload

Name		Name	Last commit message	Last commit date
Latest commit History 81 Commits
.github/workflows		.github/workflows
assets		assets
collectors		collectors
parsers		parsers
proxies		proxies
xandria		xandria
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
crawler.py		crawler.py
headers.py		headers.py
main.py		main.py
repo.py		repo.py
requirements.txt		requirements.txt
tasks.py		tasks.py