This projects is being designed such that it could by copy-pasted to a new environment and everything should 'just work' only changes would be to update environment variables to point at a different Production and Staging account in AWS.
- AWS_ACCESS_KEY_ID
-
- scope Production
- AWS_ACCESS_KEY_ID
-
- scope Staging
- AWS_SECRET_ACCESS_KEY_ID
-
- scope Production
- AWS_SECRET_ACCESS_KEY_ID
-
- scope Staging
Note S3_Key needs to match s3_key in the user.ini file
/AWSLogs/${OrgID}/${AccountID}/${S3_Key}/${AWS_REGION}/${YEAR}/${MONTH}/${DAY}/${AccountID}_${S3_Key}_${AWS_REGION}_${TIMESTAMP}_${HASH}.json.gz
${S3_Key}/${AWS_REGION}/${AccountID}/${Resource_Name}[/${Log_Type}]?
Note that Log_Type is optional
Examples:
sns/us-east-1/${AccountID}/aes-siem-loader-us-east-1
sns/us-east-1/${AccountID}/aes-siem-loader-us-east-1/Failure
/AWSLogs/${OrgID}/${AccountID}/${S3_Key}/${AWS_REGION}/${Resource_Name}[/${Log_Type}]?
Note that Log_Type is optional
Examples:
/AWSLogs/${OrgID}/${AccountID}/vpc-flow-logs/us-east-1/vpc-XXXXXXXXXXXXXX
/ThirdParty/${OrgID}/${AccountID OR Global}/${Vendor}/${S3_Key}/${AWS_Region or Global}/${Resource_Name/ID}/log_file.extension
SIEM on Amazon OpenSearch Service is a solution for collecting multiple types of logs from multiple AWS accounts, correlating and visualizing the logs to help investigate security incidents. Deployment is easily done with the help of AWS CloudFormation or AWS Cloud Development Kit (AWS CDK), taking only about 30 minutes to complete. As soon as AWS services logs are put into a specified Amazon Simple Storage Service (Amazon S3) bucket, a purpose-built AWS Lambda function automatically loads those logs into SIEM on OpenSearch Service, enabling you to view visualized logs in the dashboard and correlate multiple logs to investigate security incidents.
SIEM on OpenSearch Service can load and correlate the following log types.
| AWS Service | Log | |
|---|---|---|
| Security, Identity, & Compliance | AWS CloudHSM | HSM audit logs |
| Security, Identity, & Compliance | Amazon GuardDuty | GuardDuty findings |
| Security, Identity, & Compliance | Amazon Inspector | Inspector findings |
| Security, Identity, & Compliance | AWS Directory Service | Microsoft AD |
| Security, Identity, & Compliance | AWS WAF | AWS WAF Web ACL traffic information AWS WAF Classic Web ACL traffic information |
| Security, Identity, & Compliance | AWS Security Hub | Security Hub findings GuardDuty findings Amazon Macie findings Amazon Inspector findings AWS IAM Access Analyzer findings |
| Security, Identity, & Compliance | AWS Network Firewall | Flow logs Alert logs |
| Management & Governance | AWS CloudTrail | CloudTrail Log Event CloudTrail Insight Event |
| Management & Governance | AWS Config | Configuration History Configuration Snapshot Config Rules |
| Management & Governance | AWS Trusted Advisor | Trusted Advisor Check Result |
| Networking & Content Delivery | Amazon CloudFront | Standard access log Real-time log |
| Networking & Content Delivery | Amazon Route 53 Resolver | VPC DNS query log |
| Networking & Content Delivery | Amazon Virtual Private Cloud (Amazon VPC) | VPC Flow Logs (Version5) |
| Networking & Content Delivery | Elastic Load Balancing | Application Load Balancer access logs Network Load Balancer access logs Classic Load Balancer access logs |
| Networking & Content Delivery | AWS Client VPN | connection log |
| Storage | Amazon FSx for Windows File Server | audit log |
| Storage | Amazon Simple Storage Service (Amazon S3) | access log |
| Database | Amazon Relational Database Service (Amazon RDS) (Experimental Support) |
Amazon Aurora(MySQL) Amazon Aurora(PostgreSQL) Amazon RDS for MariaDB Amazon RDS for MySQL Amazon RDS for PostgreSQL |
| Database | Amazon ElastiCache | ElastiCache for Redis SLOWLOG |
| Analytics | Amazon OpenSearch Service | Audit logs |
| Analytics | Amazon Managed Streaming for Apache Kafka (Amazon MSK) | Broker log |
| Compute | Linux OS via CloudWatch Logs |
/var/log/messages /var/log/secure |
| Compute | Windows Server 2012/2016/2019 via CloudWatch Logs |
System event log Security event log |
| Containers | Amazon Elastic Container Service (Amazon ECS) via FireLens |
Framework only |
| End User Computing | Amazon WorkSpaces | Event log Inventory |
Experimental Support: We may change field type, normalization and something in the future.
Supported logs are normalized in accordance with the Elastic Common Schema. Click here to see the correspondence table of the original and normalized field names for the logs.
You can add country information as well as latitude/longitude location information to each IP address. To get location information, SIEM on OpenSearch Service downloads and uses GeoLite2 Free by MaxMind. If you want to add location information, get your free license from MaxMind.
Threat information can be enriched based on IP addresses and domain names (EXPERIMANTAL). Threat information sources include your own IoCs (Indicators of compromise) in TXT and STIX 2.x formats, Tor Project, Abuse.ch Feodo Tracker, AlienVault OTX. If there are many IoCs, the processing time of Lambda will increase, so please select IoCs carefully. If you want to use the IoC on AlienVault OTX, please get your API key at AlienVault OTX. See Threat Information Enrichment by IoC for more details.
Note: The CloudFormation template will deploy OpenSearch Service with a t3.medium.search instance. It's not the AWS Free Tier. Change it to an instance type that can deliver higher performance than t3 when using SIEM in the production environment as it requires higher processing power when aggregating many logs. Use the AWS Management Console to change the instance type, extend the volume, or use UltraWarm. This is because the CloudFormation template for SIEM on OpenSearch Service is designed for the initial deployment purpose only, and cannot be used for management purposes like changing/deleting nodes.
TODO: Update this later.
It will take about 30 mins for the deployment of SIEM on OpenSearch Service to complete. You can then continue to configure OpenSearch Dashboards.
- Navigate to the AWS CloudFormation console, choose the stack that you've just created, and then choose "Outputs" from the tab menu at the top right. You can find your username, password, and URL for OpenSearch Dashboards. Log into OpenSearch Dashboards using the credentials.
- When you login for the first time, [Select your tenant] is displayed. Select [Global]. You can use the prepared dashboard etc.
- You can also select [Private] instead of [Global] in [Select your tenant] and customize configuration and dashboard etc. for each user. The following is the procedure for each user. If you select Global, you do not need to set it.
- To import OpenSearch Dashboards' configuration files such as dashboard, download saved_objects.zip. Then unzip the file.
- Navigate to the OpenSearch Dashboards console. Click on "Stack Management" in the left pane, then choose "Saved Objects" --> "Import" --> "Import". Choose dashboard.ndjson which is contained in the unzipped folder. Then log out and log in again so that the imported configurations take effect.
All you need to do to load logs into SIEM on OpenSearch Service is PUT logs to the S3 Bucket named aes-siem-<YOUR_AWS_ACCOUNT>-log. Then the logs will be automatically loaded into SIEM on OpenSearch Service. See this for detailed instructions on how to output AWS services logs to the S3 bucket.
We have published the workshop, SIEM on Amazon OpenSearch Service Workshop. In this workshop, we will build the SIEM, ingest AWS resource logs, learn OpenSearch Dashboards, investigate security incident, create dashboard, configure alerts and ingest logs of Apache HTTPD server.
If you want to update "SIEM on OpenSearch Service/SIEM on Amazon ES" to the latest version, upgrade the OpenSearch / Elasticsearch domain and then update it in the same way as you did for the initial setup (using CloudFormation or AWS CDK.) You can view the changelog of SIEM here.
Note: When you update SIEM, Global tenant settings, dashboards, etc. will be overwritten automatically. The configuration files and dashboards used before the update will be backed up to aes-siem-[AWS_Account]-snapshot/saved_objects/ in the S3 bucket, so restore them manually if you want to restore the original settings.
If you want to make changes to the OpenSearch Service domain itself such as changing the access policy of OpenSearch Service, changing the instance type, changing the Availability Zone or adding a new one, or changing to UltraWarm, perform the change from the OpenSearch Service console of AWS Management Console.
SIEM on OpenSearch Service saves logs in the index and rotates it once a month. If you want to change this interval or load logs from non-AWS services, see this.
If you have an S3 bucket in the same account and region as the SIEM, you can load logs into OpenSearch Service. Please refer Near-real-time loading from other S3 buckets for the setting method.
You can execute es-loader, which is a python script, in the local environment to load past logs stored in the S3 bucket into SIEM on OpenSearch Service. See Loading past data stored in the S3 bucket for details.
To avoid unnecessary invocation of es-loader, throttle es-loader under the following conditions
- If total free space for the OpenSearch Service cluster remains less than 200MB for 30 minutes and
aes-siem-TotalFreeStorageSpaceRemainsLowAlarmis triggered.- The OpenSearch cluster is running out of storage space. More free space is needed for recovery. To learn more, see Lack of available storage space.
If you want to resume loading logs, set the reserved concurrency of the Lambda function aes-siem-es-loader back to 10 from the AWS Management Console or AWS CLI.
You can also load messages from the dead-letter queue (aes-siem-dlq) by referring to Loading data from SQS Dead Letter Queue.
terraform destroy
This library is licensed under the MIT-0 License. See the LICENSE file.
This product includes GeoLite2 data created by MaxMind, and licensed under CC BY-SA 4.0, available from https://www.maxmind.com.
This product uses Tor exit list created by The Tor Project, Inc and licensed under CC BY 3.0 US, available from https://www.torproject.org