Feat/1117 auto content type

[zar3bski]

Landing a Pull Request (PR)

Long form explanations of most of the items below can be found in the CONTRIBUTING guide.

Branching checklist

• There is an issue associated with your PR (bug, feature, etc.. if not, create one)
• Your PR description references the associated issue (i.e. fixes #123456)
• Code is in its own branch
• Branch name is related to the PR contents
• PR targets main

Static analysis checks

• All rust files are formatted using cargo fmt
• All clippy checks pass when running cargo clippy --all-targets --all-features -- -D warnings -A clippy::mutex-atomic
• All existing tests pass

I do not understand the clipy erros yet, just noticing it while submitting this PR

Documentation

• New code is documented using doc comments
• Documentation about your PR is included in the docs, as needed. The docs live in a separate repository. Update the appropriate pages at the links below.
  • update example config file section
  • update help output section
  • add an example

Not yet

Additional Tests

• New code is unit tested
• New code is integration tested, as needed
• New tests pass

Not

[zar3bski]

I removed the post-form composite arg, added Unit tests, squashed the branch and I think that we are good to go. While syncing my fork, Github integrated origin/main in a weird fashion (contrib + Fish)

I handled x-www-form-urlencoded encoding without -Q, for it is a body to be displayed in Body banner section.

One test fails nlp::model::tests::model_generates_expected_tf_idf_scores but seems unrelated to this feature. I do not understand the problem pointed by the clippy tests.

Ready for review, @epi052 . No rush :-)

[zar3bski]

I'm puzzled by how tests behave differently locally and on the CI

Locally

Whether I run cargo test from main or from 289389d, I end up with this unrelated error:

failures:
    nlp::model::tests::model_generates_expected_tf_idf_scores

Same goes when I run cargo nextest run --all-features --all-targets --retries 4 --no-fail-fast

     Summary [   8.731s] 440 tests run: 439 passed, 1 failed, 1 skipped
  TRY 5 FAIL [   0.016s] feroxbuster nlp::model::tests::model_generates_expected_tf_idf_scores

Env:

• arch linux
• rustc 1.84.0 (9fc6b4312 2025-01-07)
• cargo 1.84.0 (66221abde 2024-11-19)

CI

cargo nextest run --all-features --all-targets --retries 4 --no-fail-fast
ends up with an error on

feroxbuster::test_main main_parallel_creates_output_directory

no sign of nlp::model::tests::model_generates_expected_tf_idf_scores.

This difference of behavior together with the fact that I'm quite new to rust and do not understand in details the architecture of feroxbuster makes tests debugging pretty hard for me. Are those false positives?

[epi052]

@zar3bski don't sweat those two tests. usually when i see a difference between ci/local tests that pass/fail differently, it's because my local toolchain is behind what runs in CI. If you do a rustup update, the test fails/passes may match afterward.

As for the tests themselves, it's not code that's related to your changes. I'll take a look before we merge your branch and get things sorted. Same goes for clippy (which are just new lints in the latest toolchain version).

[epi052]

ok, i did some thinking about this and looked at how some other cli tools handle this sort of thing (if they do at all).

Below is what I'm thinking at this point, please let me know what you think!

• --data
  • implies -m POST
  • if the user provides a -m that should override the default
• --post-data should become --data-urlencoded
  • implies -m POST
  • if the user provides a -m that should override the default
  • implies -H Content-Type: application/x-www-form-urlencoded
  • values from each key/value pair within the body are url
  • should support using @ symbol to use a file path (same as --data)
• --post-json should become --data-json
  • implies -m POST
  • if the user provides a -m that should override the default
  • implies -H Content-Type: application/json
  • should support using @ symbol to use a file path (same as --data)
• if the user provides both --data-* and -m POST, we can emit an informational log line stating that the combination is redundant

[zar3bski]

@epi052 ready for your review. Two comments on it.

1. I initiated an Enum in src/config/utils.rs instead of handling the literal headers values in src/config/container.rs. The idea behind it was to provide at one point a single source for headers parsing and generation; the type of enum I miss in reqwest. If you think this is a good idea, I'll docstring the Enum, if not I remove it
2. about this logging if the user provides both --data- and -m POST, we can emit an informational log line stating that the combination is redundant, my logs do not appear in STDOUT or log files. Should I use the banner for this feature?

[zar3bski]

@epi052 You're probably busy so no rush but I would like arbitrate point 1. and finish this PR in the next following weeks. Tell me what you prefer.

[epi052]

@zar3bski apologies, i've been preoccupied with life stuff recently. for #1, the enum is fine, go ahead and add comments etc.

for #2, you need to add -vv to see info level logs. verbosity controls log level and emission (v == warn, vv == info, etc)

[zar3bski]

Don't worry about the delay, @epi052 , no problem at all (I'm not the quick type).

I finally understood why I did not get any logs from feroxbuster::config, no matter the number of -v: logger initialization occurs after config instantiation (which is logical, for it depends on it) https://github.com/epi052/feroxbuster/blob/main/src/main.rs#L641. I identified two ways out of this circularity:

1. init logger before config instantiation

logger::initialize(Arc::new(Configuration::default()))?
let config = Arc::new(Configuration::new().with_context(|| "Could not create Configuration")?);
// setup logging based on the number of -v's used
if matches!(
    config.output_level,
    OutputLevel::Default | OutputLevel::Quiet
) {
    // don't log on --silent
    logger::initialize(config.clone())?; // PB: what to do here?
}

Calling ::initialize caused an exception because of multiple calls to Builder.init(). I thought about tuning feroxbuster::logger::initialize and add a setter to set the various config params afterward (logLevel, STDOUT VS file) but, without knowing all the config fields related to logging, I would have break things.

2. Set a minimal yet Ferox friendly logger for feroxbuster::config. You'll find it in e7fa414cfe935033f45352adf9cb75e2cb74f36b. I do not think that it would appear on file logs but at least, the two log::warn of this module can now appear

Let me know what you think when you have time (life first!).
Cheers