Root relative path infinite loop

[phughesion]

I am making a separate issue for this because while it is related to this issue, it is for the currently implemented root-relative only traversal mechanism.

Python web server for testing:
malweb.py:

from flask import Flask

app = Flask(__name__)

@app.route("/", methods=["GET"])
def root():
    return f"""
        <a href="/catch_all_root_relative/1/">Go deeper</a>
    """

@app.route("/catch_all_root_relative/<path:text>", methods=["GET"])
def catch_all_root_relative(text=None):
    count = sum([int(x) for x in text if x == "1"]) + 1
    text = "/catch_all_root_relative" + ("/1" * count) + "/"
    return f"""
        <a href="{text}">Go deeper</a>
    """

python3 -m flask --app malweb run

Spider:

extern crate spider;

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("http://127.0.0.1:5000/")
    .build()
    .unwrap();
    let mut rx2 = website.subscribe(16).unwrap();

    tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            println!("{:?}", res.get_url());
        }
    });

    website.crawl_smart().await;

    println!("Links found {:?}", website.get_size().await);
}

The infinite recursion problem is a problem for both root-relative and base-relative URLs. Spider should handle this accordingly by keeping track of the link depth and stop crawling once the link depth is reached. There should be detection of this behavior rather than the current depth process that only goes based on the number of path segments.

[phughesion]

Here is another example that bypasses the depth - it goes infinitely without increasing the path segment depth. (ie. if you had a depth limit of 5):

from flask import Flask
import uuid

app = Flask(__name__)

@app.route("/", methods=["GET"])
def root():
    guid = uuid.uuid1()
    hrefs = [
        #f'<a href="/linkloop1/{guid}">Get stuck in a link loop</a>',
        #f'<a href="/catch_all_br/1/">Go deeper</a>',
        #f'<a href="/1/">Loop 1</a>',
        #f'<a href="/catch_all_root_relative/1/">Go deeper</a>',
        f'<a href="/infinite_root_relative_shallow/{guid}">Go deeper</a>',
    ]
    list_items = "\n".join([f"<li>{x}</li>" for x in hrefs])
    return f"""
        <ul>
            {list_items}
        </ul>
    """

@app.route("/infinite_root_relative_shallow/<uuid:guid>", methods=["GET"])
def infinite_root_relative_shallow(guid=None):
    link = uuid.uuid1()
    return f"""
        <a href="/{link}">Go deeper</a>
    """

@app.route("/<uuid:guid>", methods=["GET"])
def infinite_root_relative_shallow_guid(guid=None):
    link = uuid.uuid1()
    link2 = uuid.uuid1()
    return f"""
        <a href="/{link}/{link2}">Go deeper</a>
    """

@app.route("/<uuid:guid1>/<uuid:guid2>", methods=["GET"])
def infinite_root_relative_shallow_guid2(guid1=None, guid2=None):
    link = uuid.uuid1()
    return f"""
        <a href="/infinite_root_relative_shallow/{link}">Go deeper</a>
    """

[j-mendez]

Hi @phughesion , this is how we calculate the depth. We do not want to do anything with the link depth in this situation. Please convert this into a discussion. Spider keeps track of the crawling by using the URL. This is not a bug and how we want to handle the web crawling.

.