Help: I can’t load Threat Intelligence via misp.yaml in Malcolm

[mmguero]

cross-posted from Reddit:

Hi! I’m trying to load TI into a fresh Malcolm Zeek container via misp.yaml, but nothing gets ingested. Plain .intel/CSV files load fine.

• ./zeek/intel/MISP/misp.yaml

- type: misp
  url: https://www.circl.lu/doc/misp/feed-osint/manifest.json

What I see:

• Cron runs /opt/zeek/bin/zeek_intel_setup.sh true (job “succeeded”).
• No new intel files under ./zeek/intel/, and .threat_autogen.zeek doesn’t update.

Question
Why isn’t misp.yaml being picked up? Are manifest.json endpoints supported in misp.yaml, or do I need /events//attributes? Any minimal working example would be appreciated.

Thanks!

[mmguero]

I took the feed URLs from the MISP Default Feeds list with "feed format: misp" and ended up with this list:

- type: misp
  url: https://www.circl.lu/doc/misp/feed-osint/manifest.json
- type: misp
  url: https://raw.githubusercontent.com/infobloxopen/threat-intelligence/refs/heads/main/indicators/misp/manifest.json
- type: misp
  url: https://bazaar.abuse.ch/downloads/misp/manifest.json
- type: misp
  url: https://www.botvrij.eu/data/feed-osint/manifest.json
- type: misp
  url: https://threatfox.abuse.ch/downloads/misp/manifest.json
- type: misp
  url: https://urlhaus.abuse.ch/downloads/misp/manifest.json

(The DigitalSide Threat-Intel OSINT Feed was also listed, but I removed it from my list because it no longer appears to be up.)

Like all feeds, however, your "since" value (ZEEK_INTEL_FEED_SINCE in zeek.env) will determine how many indicators you actually get from these. That "since" value tells Malcolm to only process threat indicators that have been created or modified since that time (i.e., only take indicators "fresher" than that value). Ideally you set that "since" value to some relative period (e.g., "1 week ago", "2 weeks ago", "1 month ago", etc.), but what I just learned is not every threat feed in that list has indicators from the last week; I had to go further back on some of them.

Just for fun, I made up this script to run each of these feeds individually with "since" times of 1 to 8 weeks, 3 months, 6 months, and 1 year:

#!/usr/bin/env bash

URLS=(
    'https://bazaar.abuse.ch/downloads/misp/manifest.json'
    'https://raw.githubusercontent.com/infobloxopen/threat-intelligence/refs/heads/main/indicators/misp/manifest.json'
    'https://threatfox.abuse.ch/downloads/misp/manifest.json'
    'https://urlhaus.abuse.ch/downloads/misp/manifest.json'
    'https://www.botvrij.eu/data/feed-osint/manifest.json'
    'https://www.circl.lu/doc/misp/feed-osint/manifest.json'
)

for URL in ${URLS[*]}; do
    for WEEK in $(seq 1 8) 13 26 52; do
        FEED_HOST="$(echo "${URL}" | cut -d/ -f3)"
        LINES=$(docker run --rm --entrypoint=/bin/bash ghcr.io/idaholab/malcolm/zeek:25.08.1 -c "zeek_intel_from_threat_feed.py --since '${WEEK} weeks ago' -i 'misp|${URL}' -o /tmp/indicators.zeek >/dev/null 2>&1; wc -l /tmp/indicators.zeek" | awk '{print $1}')
        echo "${FEED_HOST} (${WEEK} weeks): ${LINES} lines"
    done
done

Here's what I learned:

Source	1w	2w	3w	4w	5w	6w	7w	8w	13w	26w	52w
bazaar.abuse.ch	0	0	0	0	0	0	0	0	0	0	0
raw.githubusercontent.com	0	0	2,098	2,098	2,098	2,098	2,098	2,098	104,120	105,125	105,464
threatfox.abuse.ch	1,975	3,441	4,442	5,311	6,123	7,087	7,767	9,501	18,444	101,548	159,344
urlhaus.abuse.ch	4,941	10,571	16,331	22,319	27,277	33,349	40,818	46,160	67,474	207,349	626,393
www.botvrij.eu	0	0	0	0	0	0	0	134	134	141	760
www.circl.lu	0	0	4	1,327	1,340	1,340	1,340	3,465	3,496	3,868	6,087

Naturally I'm a bit curious about the one that doesn't return any so I'll dig into it a little more and see if there's something different about the way it's representing timestamps or something.

Hopefully this is enough to get you going.

[mmguero]

So after looking at it closer, I found that different MISP feeds are returning their lists of attributes (indicators) in the MISP event objects differently.

I've created a bug and am testing a fix now.