[FEATURE REQUEST] Adding `sitemap.xml` for link extraction

[n3rada]

Hi there! 👋

Thanks for all your work on feroxbuster. I was wondering, maybe I've seen it wrong but it seems to me that link extraction takes robots.txt into account but not sitemap.xml. 🤔

That would be "simple" and interesting to add, wouldn't it?

[epi052]

hey there, sorry, i forgot to reply to this one. parsing the sitemap would absolutely be an interesting feature to add. I see you submitted a PR 🎉 , ill be able to take a look soon. Thank you !

[epi052]

Ok, adding some thoughts here on how to go about parsing sitemaps.

• there are various sitemap formats:
  • traditional xml
  • rss feed (still xml, different tags)
  • plaintext
• sitemaps can be located inside any directory on the webserver. when located in a non-root directory, they only describe locations within that directory
• urls in a sitemap are entity escaped (i.e. " becomes ")
• sitemaps may be gzipped if they're huge, but have a max uncompressed size of 50MB/50,000 URLs
• a sitemap may be a sitemap index. a sitemap index points to multiple sitemaps

It may seem like a lot, but if we're going to add sitemap parsing, I'd like to get the majority of the cases above handled.

[n3rada]

I clearly underestimated the sitemap.xml then. I guess I'm not going to be very useful in this job after all. But I think it would be amazing to have this feature as well! 😊

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[L1-0]

This would be a great feature!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[n3rada]

Hi @epi052,

If I'd like to try developing this feature myself, could you please indicate which files I should modify and provide any relevant build instructions? Additionally, could you share your website where I can test out all your features?

Thanks!

[epi052]

hi @n3rada!

Sorry for the late reply. Yes, I can put something together over the weekend most likely.

There isn't a personal website for testing. I use either local servers/tests or bugbounty / ctf targets for testing, usually

[epi052]

@n3rada the parsing calls can go in the same files you updated in your last WIP PR.

For now though, I'd focus on simply getting the parser working / fleshed out.

Let's start with a sub-module under extractor, since there's quite a bit of sitemap logic to cover. if it turns out to be overkill we can reduce it to a single file later.

./src
├── extractor
│   ├── sitemap
│       ├── mod.rs
│       ├── tests.rs
│       ├── ... etc ...

there's at least four types for the basic elements you'll run into that i can think of offhand

struct SitemapUrl {}
struct SitemapIndexEntry {}
enum SitemapType {}
enum SitemapFormat {}

you'll need a parser implementation for

• standard sitemaps
• rss feed sitemaps
• plaintext sitemaps
• sitemap indexes

After that, you'll need to be able to detect which sitemap type you've encountered

fn detect_sitemap_type(content: &str) -> SitemapFormat {
    if ... some detection rule ... {
        SitemapFormat::Standard
    }
    ... etc ...
}

with parsers and detection, you can detect and then call the appropriate parser

let format = detect_sitemap_type(...);
match format {
    SitemapFormat::Standard => parse_standard_sitemap(...)
}

with a parsed sitemap (that isn't an index), you'll need to be able to request the urls in the sitemap. Once you get to this step, we can leverage existing feroxbuster code and begin integrating into the larger project. everything above can be simply written and stand on its own. Knowing that, let' s start with what's above, and then once you've gotten to this point, lmk and we'll discuss next steps.