Post processing of URLs during Deep Crawling. #1673
Unanswered
Assylbek-the-qt
asked this question in
Forums - Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello, Guys! I have a question that I haven't found an answer to in docs. So I have a problem I need to deep crawl a website and of course it contains some internal links to other pages of that website. But the problem is for the website I must scrape the URL checking is very strange (at least for me idk). So https://example.com/pages and https://example.com/pages/ are 2 different URLS and it doesn't forward to the URL with slash at the end(but it only contains slashless URLs). So my question is: Is there a native way to append the '/' to the end of the URL? It would really be helpful for me
Beta Was this translation helpful? Give feedback.
All reactions