Robots Exclusion Standard/Protocol Parser

for Web Crawling/Scraping

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

You can find the documentation at crwlr.software.

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).

Name	Name	Last commit message	Last commit date
Latest commit otsch Also use query and fragment when matching URIs Jan 27, 2025 420aea3 · Jan 27, 2025 History 28 Commits
.github/workflows	.github/workflows	Run tests in CI also with PHP 8.4	Nov 6, 2024
bin	bin	Initial commit	Oct 24, 2021
git-hooks	git-hooks	More tests, phpstan and improve pre-commit hook	Nov 7, 2021
src	src	Also use query and fragment when matching URIs	Jan 27, 2025
tests	tests	Also use query and fragment when matching URIs	Jan 27, 2025
.editorconfig	.editorconfig	Initial commit	Oct 24, 2021
.gitattributes	.gitattributes	Initial commit	Oct 24, 2021
.gitignore	.gitignore	Initial commit	Oct 24, 2021
.php-cs-fixer.dist.php	.php-cs-fixer.dist.php	Remove forgotten variable	Nov 6, 2024
CHANGELOG.md	CHANGELOG.md	Also use query and fragment when matching URIs	Jan 27, 2025
CONTRIBUTING.md	CONTRIBUTING.md	v1.0.0	Sep 22, 2022
LICENSE	LICENSE	Also use query and fragment when matching URIs	Jan 27, 2025
README.md	README.md	v1.0.0	Sep 22, 2022
composer.json	composer.json	Run tests in CI also with PHP 8.4	Nov 6, 2024
phpstan.neon	phpstan.neon	More tests, phpstan and improve pre-commit hook	Nov 7, 2021
phpunit.xml	phpunit.xml	Initial commit	Oct 24, 2021