[hub] introduce cscli hub fix command #3264

LaurenceJJones · 2024-09-26T11:08:53Z

There has been an outstanding issue for a long time if a user "accidentally" install the debian packages and then upgrades to our repository version all symlinks point towards deleted files.

An idea could be to have cscli hub fix command that goes through the current symlinks and tries to rectify the broken symlinks for example after installing the debian package and upgrading to our this is the output of cscli parsers list

root@bookworm:~# cscli parsers list
WARN link target does not exist: /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml -> /var/lib/crowdsec/hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/nginx-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/dateparse-enrich.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/http-logs.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/http-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/whitelists.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-26134.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-26134.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-35914.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-35914.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-37042.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-37042.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-40684.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-40684.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41082.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41082.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41697.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41697.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-42889.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-42889.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-44877.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-44877.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-46169.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-46169.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/apache_log4j2_cve-2021-44228.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/f5-big-ip-cve-2020-5902.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/fortinet-cve-2018-13379.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/grafana-cve-2021-43798.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-backdoors-attempts.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-backdoors-attempts.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-bad-user-agent.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-bad-user-agent.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-crawl-non_statics.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-41773.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-41773.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-42013.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-42013.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-generic-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-generic-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-open-proxy.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-open-proxy.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-path-traversal-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-path-traversal-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sensitive-files.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sensitive-files.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sqli-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sqli-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-w00tw00t.yaml -> /var/lib/crowdsec/hub/scenarios/ltsich/http-w00tw00t.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-xss-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-xss-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/jira_cve-2021-26086.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/jira_cve-2021-26086.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/nginx-req-limit-exceeded.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/nginx-req-limit-exceeded.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/pulse-secure-sslvpn-cve-2019-11510.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/spring4shell_cve-2022-22965.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-slow-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-slow-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/thinkphp-cve-2018-20062.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-cve-2022-22954.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-vcenter-vmsa-2021-0027.yaml
WARN link target does not exist: /etc/crowdsec/collections/apache2.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/apache2.yaml
WARN link target does not exist: /etc/crowdsec/collections/base-http-scenarios.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/base-http-scenarios.yaml
WARN link target does not exist: /etc/crowdsec/collections/http-cve.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/http-cve.yaml
WARN link target does not exist: /etc/crowdsec/collections/linux.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/linux.yaml
WARN link target does not exist: /etc/crowdsec/collections/nginx.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/nginx.yaml
WARN link target does not exist: /etc/crowdsec/collections/sshd.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/sshd.yaml

PARSERS
──────────────────────────────────────
Name  📦 Status  Version  Local Path
──────────────────────────────────────
──────────────────────────────────────

I did used to have a script https://gist.github.com/LaurenceJJones/6960107296145e8e365009973b9d7f6d that would fix this, however, with recent changes to the hub no items are displayed and it cannot be fixed like this anymore.

Edit: This will improve the user experience then having to completely remove the package (potentially remove there own custom configuration) and having to restart all over again if they happen to notice this late in the process.

The text was updated successfully, but these errors were encountered:

github-actions · 2024-09-26T11:09:05Z

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

github-actions · 2024-09-26T11:09:06Z

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

/kind feature
/kind enhancement
/kind refactoring
/kind bug
/kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

bakrhaso · 2025-01-18T01:23:29Z

In this situation right now, having a command to fix it would be nice!

OfficialMuffin · 2025-01-21T11:37:11Z

Reinstalled crowdsec after a while, getting the same list of warnings.

UPDATE:

Uninstalling crowdsec and bouncers via APT, installing locate, running updatedb, then running locate to find all the residual crowdsec files. Then going through and manually deleting the residual files. After that, reinstall crowdsec and the bouncer. This fixes this issue for me and is most likely the current workaround.

mmetc · 2025-01-21T14:42:23Z

@LaurenceJJones I tried to go down the route of "cscli hub fix", noting that

the command runs when the hub content from the previous package (1.4.6) is already removed. we have the links but no source or index, no consistent state to copy or reinstall
running it in pre-remove has its own issues, like there is an old hub but not the new one, or index
adding that in cscli would make the code unnecessary more complex when it can be done outside of the process, in bash

So I wrote a script that can be run right after installing 1.6.5 (not tested with 1.6.4), here

https://raw.githubusercontent.com/crowdsecurity/crowdsec/refs/heads/migrate-hub/debian/migrate-hub.sh

#!/usr/bin/env sh

set -eu

# Download everything on the new hub but don't install anything yet

echo "Downloading Hub content..."

for itemtype in $(cscli hub types -o raw); do
    ALL_ITEMS=$(cscli "$itemtype" list -a -o raw | tail +2 | cut -d, -f1)
    if [ -n "${ALL_ITEMS}" ]; then
        # shellcheck disable=SC2086
        cscli "$itemtype" install \
            $ALL_ITEMS \
            --download-only -y
    fi
done

# Fix links

BASEDIR=/etc/crowdsec/
OLD_PATH=/var/lib/crowdsec/hub/
NEW_PATH=/etc/crowdsec/hub/

find "$BASEDIR" -type l 2>/dev/null | while IFS= read -r link
do
    target="$(readlink "$link")" || continue

    case "$target" in
        "$OLD_PATH"*)
            suffix="${target#"$OLD_PATH"}"
            new_target="${NEW_PATH}${suffix}"

            if [ -e "$target" ]; then
                continue
            fi

            if [ ! -e "$new_target" ]; then
                continue
            fi

            echo "Update symlink: $link"
            ln -sf "$new_target" "$link"
            ;;
        *)
            ;;
    esac
done

# upgrade tainted collections

cscli hub upgrade --force

with the following caveats

it takes some time and downloads all the hub and data files
it's very verbose and doesn't hide the warnings until it's finished
tainted stuff will be replaced.

Otherwise it should do the job. I'm not sure if it's a good idea to install and run it automatically in its current state.

github-actions bot added the needs/triage label Sep 26, 2024

github-actions bot added the needs/kind label Sep 26, 2024

buixor added this to the 1.6.4 milestone Sep 27, 2024

buixor assigned mmetc Sep 27, 2024

buixor added kind/enhancement New feature or request area/cscli value/low Doing this kinda improves some areas and removed needs/triage needs/kind labels Oct 15, 2024

github-actions bot added the needs/triage label Oct 15, 2024

buixor modified the milestones: 1.6.4, 1.6.5 Nov 15, 2024

LaurenceJJones linked a pull request Jan 21, 2025 that will close this issue

Migration script from debian/ubuntu package 1.4.6 #3420

Open

buixor modified the milestones: 1.6.5, 1.6.6 Jan 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hub] introduce cscli hub fix command #3264

[hub] introduce cscli hub fix command #3264

LaurenceJJones commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

bakrhaso commented Jan 18, 2025

OfficialMuffin commented Jan 21, 2025 •

edited

Loading

mmetc commented Jan 21, 2025 •

edited

Loading

[hub] introduce cscli hub fix command #3264

[hub] introduce cscli hub fix command #3264

Comments

LaurenceJJones commented Sep 26, 2024 • edited Loading

github-actions bot commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

bakrhaso commented Jan 18, 2025

OfficialMuffin commented Jan 21, 2025 • edited Loading

mmetc commented Jan 21, 2025 • edited Loading

LaurenceJJones commented Sep 26, 2024 •

edited

Loading

OfficialMuffin commented Jan 21, 2025 •

edited

Loading

mmetc commented Jan 21, 2025 •

edited

Loading