pfSense to Azure Sentinel via Logstash

[noodlemctwoodle]

@a3ilson - I have finally reached my end goal of getting my pfSense logs to Azure Sentinel and I just wanted to show my appreciation and say thank you, because without your project it would have taken me considerably longer. I've attempted many ways to get my pfSence logs to Azure.

As you aware I really wanted to use a docker container on UnRAID, however it was causing a my server to run out of resources and at the end of the day I didnt really want to use Elastic Search or Kibana for my project as I am already using Azure Sentinel as my SIEM solution.

For anyone else that is intersted in my Azure Sentinel project you can view it on my GitHub.

Once again without the work from @a3ilson this would not be possible so thank you very much for your hard work... :D

[a3ilson]

@noodlemctwoodle - Thanks and greatly appreciated. This should certainly helps others with a similar setup! I also know that others are making gains on the UNRaid variant (#188).

[noodlemctwoodle]

@a3ilson sent a small gesture of appreciation on PayPal.

Happy Christmas 🎅

[a3ilson]

Much appreciated - Thank you!

[noodlemctwoodle]

@a3ilson I have a question about the 35-rules-desc.conf I have populated the rule-names.csv however nothing is being populated in the logs for this field.

rule-names.csv - Example - My actual CSV has over 150 lines.

"Rule","Label"
"0","null"
"3","pass IPv6 loopback"
"4","pass IPv6 loopback"
"5","Block all IPv6"
"6","Block all IPv6"
"7","Block IPv4 link-local"
"8","Block IPv4 link-local"

35-rules-desc.conf

filter {
  if [observer][type] == "firewall" {
    if [rule][id] {
      translate {
        field => "[rule][id]"
        destination => "[rule][alias]"
        dictionary_path => "/etc/logstash/conf.d/databases/rule-names.csv"
        refresh_interval => 60
        refresh_behaviour => replace
        fallback => "%{[rule][id]}"
      }
      mutate {
        add_field => { "[rule][description]" => "%{[interface][alias]}: %{[rule][alias]}" }
      }
    }
  }
}

Am I right in thinking that rule alias and rule description are missing?

[a3ilson]

Can you provide one of your original messages (e.g. <134>Dec 21 09:57:12 filterlog: 5,,,1000000103,igb0,match,block,in,4,0x0,,238,55851,0,none,6,tcp,40,92.63.196.125,xxx.xxx.xxx.xxx,58329,14112,0,S,3426744830,,1024,,)? I suspect the message isn't parsing out the rule.id field needed for the 35-rules-desc.conf file but rather it may be parsed out as another field (e.g. rule.ruleset, rule.uuid etc...)

I'm using OPNsense which message is slightly different and rule numbers that don't parse well compared to pfSense

[a3ilson]

Or take the look at the one you provide and let me know what the rule number should be?

Your example has, rule_ruleset_s = 64; rule_uuid_s = 10000008920....I suspect your rule number is 64 and the 35-rules-desc.conf will need to be amended.

[a3ilson]

found it! The GROK pattern was adjusted to align with pfSense documentation and ECS but the 35-rules-desc.conf was not.

I'll submitted a pull request...let me know if the update works.

[noodlemctwoodle]

Thanks, I've added the change. I'll give it 10 mins to update

[a3ilson]

You'll need to restart Logstash for it to take, due to the field name change, but thereafter, amending the rules database file should be sufficient as it refreshes the database every 60-seconds.

[a3ilson]

Are those all the fields? If so it doesn't`t appear to be parsing the message in it's entirety? Can you provide the log_original text?

[noodlemctwoodle]

No all the fields dont fit on one screen :)

[noodlemctwoodle]

Where can I see the parsed Logstash message on the commandline? Is there a file I can tail?

[a3ilson]

Can you confirm the value within the rule_ruleset field matches your firewall rule within pfSense?

[noodlemctwoodle]

[a3ilson]

I added two wiki pages but the pull request didn't`t capture... I'll attach include here for inclusion to your repo.

[noodlemctwoodle]

Please could you link me to the wiki pages you created for MaxMind as it's resulted in a dead link in my README?

[a3ilson]

You can copy these Wiki pages and amend to your repo.

MaxMind

• Wiki Page: https://github.com/pfelk/pfelk/wiki/How-To:-MaxMind-via-GeoIP-with-pfELK

Rule Script Generator

• Wiki Page: https://github.com/pfelk/pfelk/wiki/References:-Rule-Description-Script-Generator

[noodlemctwoodle]

Perfect, I've already linked them there :)

[noodlemctwoodle]

@a3ilson I've fixed it :)

if [rule][ruleset] {

[a3ilson]

dang... sorry I missed/overlooked one

[noodlemctwoodle]

Thanks for your help 👍

[a3ilson]

no worries...glad it worked out.

[noodlemctwoodle]

I'll amend my guide to account for OPNsense later this afternoon.

I'll add the pfSense-35-rules-desc.conf and an OPNsense-35-rules-desc.conf to my config.

[a3ilson]

You should need to add those files? The current one suffices for both.

[noodlemctwoodle]

Now I have to go Christmas food shopping before all the shops run out of food as France have shut the borders to the UK due to covid and there is mass panic that the shops will run short of stock in the run-up to Christmas...... What a chore!! :@

[a3ilson]

Good luck and stay safe...hopefully all this madness ends soon.

[a3ilson]

Any though on amending the repository from pfsense-azure-sentinel to pf-azure-sentinel? The setup will work for both pfSense and OPNsense?

Also I would recommend added tags to your repository aiding those searching for this capability.

Example:

[noodlemctwoodle]

@a3ilson - FYI I've renamed the repo to pf-azure-sentinel

[noodlemctwoodle]

@a3ilson Hope you are well :)

Is there a way to drop all GROK parsing failures using 45-cleanup.conf so they are not sent to 50-outputs.conf

[a3ilson]

@noodlemctwoodle

You could drop grok failures by something like this:

    filter {
      if "_grokfailure" in [tags] {
        drop { }
      }
    }

or you could amend 45-cleanup.conf as:

# 45-cleanup.conf
################################################################################
# Version: 21.02                                                               #
# Required: No - Optional                                                      #
# Description: Removed unwanted logs based on the process.pid field and        #
# additional fields. Additionally, pf.tcp.options is split (multiple values)   #
################################################################################
#
# Update as needed to remove unwanted logs based on the process.pid field
filter {
  if "_grokfailure" in [tags] {
	drop { }
  }
  mutate {
    remove_field => ["pfelk_message"]
    remove_field => ["filter_message"]
    remove_field => ["type"]
    split => { "[pf][tcp][options]" => ";" }
    rename => { "message" => "[event][original]" }
    remove_field => [ "host" ]
  }
}

[noodlemctwoodle]

@a3ilson Legend, thank you!

I need to drop the failures as they are costing me money 💰💰💰

Another question, is there a way of viewing the processed data before it gets to 50-outputs.conf?

[a3ilson]

@noodlemctwoodle - Depends...I've commented the 45-cleanup.conf in the past to get all logs indexed...this would allow you to view and modify/refine the dropped events.