Replies: 1 comment 1 reply
-
|
Should I post this as an 'Issue'? I'm not sure if it is, or if I'm just missing something! |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Maybe post a note sometwhere if this place isn't monitored |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
Just testing the crawler and it seems to work fine on some sites and just gets one page on others.
Example - www.jfmc.co.uk
It only crawls the first page. Even pointing it at the sitemap results in just that page being crawled.
The site pages are all at the root of directories without a .html etc file being specified, eg https://www.jfmc.co.uk/Sitemap , but that doesn't seem to be a problem on other sites I tested.
Nothing in the robots.txt should break it.
Beta Was this translation helpful? Give feedback.
All reactions