Suppressor, wordlist, rules: bug fix and enhancements/cleanup

[solardiz]

Previously, we only supported a repeated initialization call after cracking mode transition, but as magnum found we may also call more than once from within wordlist mode.

Fixes #5714

[magnumripper]

Looks good to me. I can test it tomorrow with the files from the original problem.

[solardiz]

I sneaked a couple of other suppressor enhancements in here (separate commits).

[solardiz]

Got yet another subtly related change in here:

Wordlist, rules: Drop the dupe check against the previous word

This check dates back to 1990s when our primary target was descrypt with
its length truncation, its computation was much slower, and we didn't have
the generic dupe suppressor.  In the suppressor, a word matching the
previous one would be an L1 cache hit, so we don't gain much by
pre-checking for that special case.  And if the suppressor is disabled,
that's probably because we don't want any overhead like that.

Besides, this check didn't behave correctly along with an external filter()
possibly modifying the word (a bug that also dates back to 1990s).

OTOH, we do not currently support the suppressor on top of stacked rules
(we do before stacked rules), so we do lose a bit of functionality with
this change.

This is something I had thought of doing years ago, and especially after I added the suppressor in 2022.

Testing with our default wordlist and rules, but without suppressor, this changes the output only slightly. In the first 100M candidates produced (including dupes), here are the numbers of unique, before:

Total lines read: 100000000, unique lines written: 90070771 (90%), no slow passes

After:

Total lines read: 100000000, unique lines written: 90070624 (90%), no slow passes

meaning that ~150 more dupes got into the first 100M. I think that difference is so tiny it wasn't worth checking for those dupes, even without suppressor, and now the suppressor would take care of all of them when enabled.

[solardiz]

~150 more dupes got into the first 100M. I think that difference is so tiny it wasn't worth checking for those dupes, even without suppressor, and now the suppressor would take care of all of them when enabled.

Confirming with -dup=256, before:

Total lines read: 100000000, unique lines written: 98577332 (98%), no slow passes

After:

Total lines read: 100000000, unique lines written: 98577332 (98%), no slow passes

[magnumripper]

Testing with our default wordlist and rules, but without suppressor, this changes the output only slightly

Is there any measurable performance improvement? Perhaps it's insignificant.

[solardiz]

Testing with our default wordlist and rules, but without suppressor, this changes the output only slightly

Is there any measurable performance improvement? Perhaps it's insignificant.

On the system I'm testing on, there's no obvious performance change when running with rules (there's fluctuation of a few percent), but there's a ~10% speedup when running without rules. Perhaps that ARCH_WORD sized pre-check in rules.c was very efficient, and the strcpy / strcmp in wordlist.c not so efficient.

[solardiz]

Looks good to me. I can test it tomorrow with the files from the original problem.

Did you? OK to merge?

[magnumripper]

Looks good to me. I can test it tomorrow with the files from the original problem.

Did you? OK to merge?

I was delayed by the LM test vector problem. I expect it to be fine now but I will verify it when possible.