Zeek Intel Framework - TAXII autogen failing

[bjamesschmitt]

We've tested OpenCTI presenting both authenticated and non-authenticate feeds to Malcolm using both methods outlined for TAXII feeds in the Malcolm documentation. In this case. we know Malcolm picks up STIX-formatted JSON files just fine and we observed Malcolm reaching out to our OpenCTI host and consuming a collection just fine. The odd part is no matter how we configure the TAXII feed (taxii.yaml file or the stix_input.txt method), while it pulls the JSON down from OpenCTI...nothing else happens and the .threat_autogen.zeek file is not updated. Where can we "debug" this further to understand how to get our internal OpenCTI host to provide Malcom STIX files?

I've pasted an example of the YAML file - we know that Malcolm obtains the JSON from that collection as we have the pcap...just doesn't do anything after that. The JSON file it pulls down passes JSON linting and has simple comparison IOCs.

- type: taxii version: 2.1 url: "http://1.2.3.4:8080/taxii2/root/collections/a81befa5-3c1b-4f20-8305-c51bb2634710/objects" collection: "a81befa5-3c1b-4f20-8305-c51bb2634710"

here is an example indicator from the above collection:
{ "id": "indicator--ddc5190a-6701-5de2-8537-f4e0e04a9611", "spec_version": "2.1", "type": "indicator", "extensions": { "extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba": { "extension_type": "property-extension", "id": "93df2e00-2f0e-4282-b82b-7da717e6ff4a", "type": "Indicator", "created_at": "2025-06-26T19:06:04.610Z", "updated_at": "2025-06-26T19:08:58.420Z", "is_inferred": false, "creator_ids": [ "88ec0c6a-13ce-5e39-b486-354fe4a7084f" ], "created_by_ref_id": "31225020-4a0d-4de0-84a9-040fb5832c5b", "detection": false, "score": 20, "main_observable_type": "Domain-Name", "observable_values": [ { "type": "Domain-Name", "value": "pouractionconforme.pro" } ] }, "extension-definition--322b8f77-262a-4cb8-a915-1e441e00329b": { "extension_type": "property-extension" } }, "created": "2025-06-26T19:05:05.933Z", "modified": "2025-06-26T19:08:58.420Z", "revoked": true, "confidence": 20, "lang": "en", "object_marking_refs": [ "marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da" ], "created_by_ref": "identity--85bcd2a1-e457-5ca1-a3b9-7659dfd0d91b", "name": "pouractionconforme.pro", "pattern": "[domain-name:value = 'pouractionconforme.pro']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2024-11-01T00:19:22.000Z", "valid_until": "2025-03-29T00:14:47.129Z" }

[mmguero]

Here's how I'd debug.

1. Get Malcolm up and running
2. get a shell in the Zeek container

$ docker compose exec zeek bash

4. manually run the script that pulls from your TAXII feed, with debugging output enabled:

$ cd /tmp
$ zeek_intel_from_threat_feed.py -vvv --since "1 week ago" --input-file /opt/zeek/share/zeek/site/intel/STIX/taxii.yaml -o /tmp/debugintel.zeek

It will spit out a bunch of stuff like this:

$ zeek_intel_from_threat_feed.py -vvv --since "1 week ago" --input-file /opt/zeek/share/zeek/site/intel/STIX/taxii.yaml -o /tmp/debugintel.zeek
2025-07-16 17:57:08 INFO: /opt/zeek/bin/zeek_intel_from_threat_feed.py
2025-07-16 17:57:08 INFO: Arguments: ['-vvv', '--since', '1 week ago', '--input-file', '/opt/zeek/share/zeek/site/intel/STIX/taxii.yaml', '-o', '/tmp/debugintel.zeek']
2025-07-16 17:57:08 INFO: Arguments: Namespace(verbose=10, notice=True, cif=True, extended=True, sslVerify=False, input=None, inputFile=['/opt/zeek/share/zeek/site/intel/STIX/taxii.yaml'], output='/tmp/debugintel.zeek', since='1 week ago', threads=2)
2025-07-16 17:57:08 DEBUG: /etc/timezone found, contents:
 Etc/UTC

2025-07-16 17:57:08 DEBUG: /etc/localtime found
2025-07-16 17:57:08 DEBUG: 2 found:
 {'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
2025-07-16 17:57:08 DEBUG: Inputs: [{'type': 'taxii', 'version': 2.1, 'url': 'https://example.org/taxii', 'collection': '*'}]
2025-07-16 17:57:08 DEBUG: [1]: started
2025-07-16 17:57:08 DEBUG: Starting new HTTPS connection (1): example.org:443
2025-07-16 17:57:08 DEBUG: [2]: started
2025-07-16 17:57:08 DEBUG: [2]: finished
2025-07-16 17:57:09 DEBUG: https://example.org:443 "GET /taxii/ HTTP/1.1" 200 247
2025-07-16 17:57:09 DEBUG: Starting new HTTPS connection (1): example.org:443
2025-07-16 17:57:09 DEBUG: https://example.org:443 "GET /taxii/root/collections/ HTTP/1.1" 200 277
2025-07-16 17:57:09 DEBUG: Starting new HTTPS connection (1): example.org:443
2025-07-16 17:57:09 DEBUG: https://example.org:443 "GET /taxii/root/collections/7f9137a7-59ff-4fc6-957a-a90cc66c91b5/ HTTP/1.1" 200 258
...
2025-07-16 17:57:09 DEBUG: [1]: finished

Hopefully we could use some of that to get things figured out. You can tweak your taxii.yaml file (just cp /opt/zeek/share/zeek/site/intel/STIX/taxii.yaml /tmp/taxii.yaml and then use vi to make changes to it, then run the above command with --input-file /tmp/taxii.yaml) and see how it affects the output.

I've found different TAXII clients/servers are not all the same when it comes to following the standard. Another thing I'd maybe try is fiddling with the URL in the taxii.yaml again. You've tried:

• http://1.2.3.4:8080/taxii2/root/collections/

Maybe also try these permutations and see if it behaves differently. You're already specifying the collection: in the YAML.

• http://1.2.3.4:8080/taxii2
• http://1.2.3.4:8080/taxii2/root

Best of luck, let me know what you see.

[mmguero]

Yeah, I mean that part is expected. When you restart Malcolm, it automatically pulls and generates those new files. So you would expect to see those being created every time you start Malcolm.

So the .threat_autogen.zeek file and .old file in that screenshot... they're not 0 bytes, where did those ones come from?

[ericvo28]

That was from testing a previous set up where we would manually take .json files from CISA and put them in the STIX directory; which worked and created the .threat_autogen correctly

[mmguero]

Ah, I see. That should still actually happen, after it tries to do the pull and process the threat intel from the feeds, it'll take those json files from the STIX directory and merge them all into the new autogen file at the end.

I'm not 100% sure what the issue is at this point. The next step for me would be I'd need to try to set up something locally with OpenCTI to reproduce the issue and debug through what that script is seeing. I don't have cycles to do that right now, but hopefully sometime in the next few weeks I can dedicate some time to trying to repro and debug.

[ericvo28]

No problem, I will keep working on debugging and update you as I find more

[ericvo28]

After running zeek_intel_from_threat_feed.py -vvv --since "1 week ago" --input-file /opt/zeek/share/zeek/site/intel/STIX/taxii.yaml -o /tmp/debugintel.zeek
it seems to run in an infinite loop, continuously grabbing from the taxii collection even though we have "--since '1 week ago'". I also have it set to "7 days ago" in zeek.env so i'm not sure if the taxii collection is just massive to where it would take days to finish pulling or if Malcolm is pulling the same information over and over.

Edit: The script finished running eventually. Nothing was outputted to the debug file. Still nothing being written to the .threat_autogen.zeek file