v0.3.0-alpha Customizable User-Agent
Pre-release
Pre-release
andreaskoch
released this
02 May 12:01
·
15 commits
to develop
since this release
You can now customize the user-agent that is used by the crawler:
gargantua crawl \
--url https://www.sitemaps.org/sitemap.xml \
--workers 5 \
--user-agent "gargantua bot / iPhone"