[BUG] Burp option not working with HTTPS

[thibon]

Describe the bug
When using '--burp' option or '--proxy http://localhost:8080' with an url in HTTPS, I get the following error:

WRN      2.224 feroxbuster::utils Error while making request: error sending request for url (https://target/)                                                                                           WRN      2.224 feroxbuster::utils err: error sending request for url (https://target/)                                                                                                                  Could not connect to https://www.dma.gp, skipping...
  => error sending request for url (https://target/)

However, looks like it's working with '--replay-prox' option. Note that I'm not sure to understand the difference between the two based on the documentation

I'm using feroxbuster 2.10.4

To Reproduce
Steps to reproduce the behavior:

1. Just try to connect to an https target

Expected behavior
Normal request going through the proxy

Traceback / Error Output

~ ❯ feroxbuster --url 'https://target' -w 'SecLists/Discovery/Web-Content/raft-large-words-lowercase.txt' --burp -vvv                                27/06/24 11:28:23 AST
DBG      0.001 feroxbuster::utils set open file descriptor limit to 8192                                                                                                                                                                                                                                                                                                                                                DBG      0.013 feroxbuster::event_handlers::filters filters: FiltersHandler { data: FeroxFilters { filters: RwLock { data: [], poisoned: false, .. } }, receiver: UnboundedReceiver { chan: Rx { inner: Chan { tx: Tx { block_tail: 0x14e80a400, tail_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 1, rx_fields: "..." } } } }
 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://target
 🚀  Threads               │ 50
 📖  Wordlist              │ SecLists/Discovery/Web-Content/raft-large-words-lowercase.txt
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💎  Proxy                 │ http://127.0.0.1:8080
 🔎  Extract Links         │ true
 🏁  HTTP methods          │ [GET]
 🔓  Insecure              │ true
 🔊  Verbosity             │ 3
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
WRN      2.176 feroxbuster::utils Error while making request: error sending request for url (https://target/)                                                                                           WRN      2.176 feroxbuster::utils err: error sending request for url (https://target/)                                                                                                                  Could not connect to https://target, skipping...
  => error sending request for url (https://target/)                                                                                                                                                    WRN      2.177 feroxbuster::heuristics error sending request for url (https://target/)                                                                                                                  INF      2.177 feroxbuster All scans complete!                                                                                                                                                              INF      2.178 feroxbuster::event_handlers::statistics Stats {
    kind: "statistics",
    timeouts: 0,
    requests: 2,
    expected_per_scan: 0,
    total_expected: 0,
    errors: 1,
    successes: 1,
    redirects: 0,
    client_errors: 0,
    server_errors: 0,
    total_scans: 0,
    initial_targets: 0,
    links_extracted: 0,
    extensions_collected: 0,
    status_200s: 1,
    status_301s: 0,
    status_302s: 0,
    status_401s: 0,
    status_403s: 0,
    status_429s: 0,
    status_500s: 0,
    status_503s: 0,
    status_504s: 0,
    status_508s: 0,
    wildcards_filtered: 0,
    responses_filtered: 0,
    resources_discovered: 0,
    url_format_errors: 0,
    redirection_errors: 0,
    connection_errors: 0,
    request_errors: 1,
    directory_scan_times: Mutex {
        data: [],
        poisoned: false,
        ..
    },
    total_runtime: Mutex {
        data: [
            0.0,
        ],
        poisoned: false,
        ..
    },
    json: false,
    targets: Mutex {
        data: [
            "https://target",
        ],
        poisoned: false,
        ..
    },
}                                                                                                                                                                                                           ERROR: Could not connect to any target provided

Environment (please complete the following information):

• feroxbuster version: 2.10.4
• OS: MacOS Ventura 13.4

Additional context
No additional context, it was working well in the past, did many test without any problem but can't find the cause here

[godylockz]

@thibon Did you add burp's cert to your systems cert store?
update-ca-certificates

HTTPS doesn't allow MITM interceptions - you have to enable trust.

I would first try to check if this works:

export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
curl https://target

[thibon]

Hi, thanks for your answer but this test works, and burp cert is in my cert store.
What is weird is that when I configure feroxbuster with --burp option or with --proxy, the first request is OK, I can see it in burp but after that the second fail and I get the error

[epi052]

@thibon can you elaborate on the first request? is it the request to github.com that checks for updates? or is it a request to the actual target? If so, can you post the server response to that first request?

additionally, a pcap of the conversation might shed some light on things as well.

[epi052]

Also, the --replay-proxy option waits until you get a response from the server that wasn't filtered out, and then passes it to burp. It's just a convenience option to only send interesting things to burp for further analysis, instead of sending all the 404s et al and cluttering up the history

[epi052]

just for the record, i'm able to run -u https://bitdiscovery.com --burp with no issues.

i looked through the replay proxy code vs the proxy code and i don't see any obvious differences that would lead to one working adn the other not.

[thibon]

Hi,

Let's forget about --replay-proxy option, it's not related to the problem and I didn't fully understood this option at first

@thibon can you elaborate on the first request? is it the request to github.com that checks for updates? or is it a request to the actual target? If so, can you post the server response to that first request?

additionally, a pcap of the conversation might shed some light on things as well.

Traffic is HTTPS so it's a bit hard to provide an deciphered pcap

However, digging in with your example, it works pretty well with some targets like https://bitdiscovery.com or https://google.com, but not with others like https://dma.gp or https://www.groupe-paralliance.fr

It looks like it is because the two other targets don't support HTTP/2 and when feroxbuster sends the second request, it's using HTTP/2.0 regarding of the response of the server to the first request (if the server respond with HTTP/1.1 for example). I was able to have it work by forcing burp to use HTTP/1 in the proxy options because I couldn't find an option to tell feroxbuster to use HTTP/1.1, and it didn't follow server instructions

[epi052]

thanks for the targets that exhibit the behavior!

regarding the pcap, i actually was interested in the tls conversation, since we can see the http stuff in burp.

for https://dma.gp, i got a bad certificate error.

so, i then added -k and the scan worked fine. (--dont-filter --dont-extract-links simply limit the number of requests sent to the target, they can be removed and the scan will still work).

feroxbuster -u https://dma.gp -k  --dont-filter --dont-extract-links

 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://dma.gp
 🚀  Threads               │ 50
 📖  Wordlist              │ twl.smol
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💉  Config File           │ /home/epi/PycharmProjects/feroxbuster/ferox-config.toml
 🔎  Extract Links         │ true
 🏁  HTTP methods          │ [GET]
 🔓  Insecure              │ true
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
403      GET        9l       28w      272c Auto-filtering found 404-like response and created new filter; toggle off with --dont-filter
301      GET        0l        0w        0c Auto-filtering found 404-like response and created new filter; toggle off with --dont-filter
[####################] - 2s         2/2       0s      found:0       errors:0      
[####################] - 2s         2/2       1/s     https://dma.gp/                                                                        

I get the same results for the other site provided

feroxbuster -u https://www.groupe-paralliance.fr -w small.txt --dont-filter --dont-extract-links

 ___  ___  __   __     __      __         __   ___
|__  |__  |__) |__) | /  `    /  \ \_/ | |  \ |__
|    |___ |  \ |  \ | \__,    \__/ / \ | |__/ |___
by Ben "epi" Risher 🤓                 ver: 2.10.4
───────────────────────────┬──────────────────────
 🎯  Target Url            │ https://www.groupe-paralliance.fr
 🚀  Threads               │ 50
 📖  Wordlist              │ twl.smol
 👌  Status Codes          │ All Status Codes!
 💥  Timeout (secs)        │ 7
 🦡  User-Agent            │ feroxbuster/2.10.4
 💉  Config File           │ /home/epi/PycharmProjects/feroxbuster/ferox-config.toml
 🏁  HTTP methods          │ [GET]
 🤪  Filter Wildcards      │ false
 🔃  Recursion Depth       │ 4
───────────────────────────┴──────────────────────
 🏁  Press [ENTER] to use the Scan Management Menu™
──────────────────────────────────────────────────
404      GET      651l     4403w    99427c https://www.groupe-paralliance.fr/test
200      GET     1553l     8152w   227424c https://www.groupe-paralliance.fr/
[####################] - 2s         2/2       0s      found:1       errors:0      
[####################] - 2s         2/2       1/s     https://www.groupe-paralliance.fr/                                                     

So, at this point, we can say for certain that feroxbuster can handle talking to the sites directly. When we add --proxy http://127.0.0.1:8080, then we see the problematic behavior.

I don't have the time right this minute to setup certs to see how burp handles its side of the conversation on the wire, but it appears to be a burp problem, or at worst a ferox problem realted to how it interacts with burp in certain configurations.

I'm also curious if we use zap or another intercepting proxy, what the behavior looks like.

[epi052]

small update: using ZAP as the intercepting proxy also works fine. it seems to be a burp issue

[thibon]

I can confirm, did the test too.
I don't understand at which point the combination of feroxbuster and Burp decide to send the other request in HTTP/2, I will try to dive into it in a few days. For now I will just force downgrade in Burp

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[epi052]

small update, here's some more verbose logging of the error

WRN      0.568 feroxbuster::utils exit: make_request -> error sending request for url (https://www.groupe-paralliance.fr/)
WRN      0.569 feroxbuster::utils exit: make_request -> client error (SendRequest)
WRN      0.569 feroxbuster::utils exit: make_request -> http2 error
WRN      0.569 feroxbuster::utils exit: make_request -> stream error detected: unspecific protocol error detected