SIEM, Security Logging and Monitoring

4 Nov 2026, 11:2828 min readby ITYantra Admin

Complete guide to SIEM implementation, security logging, log aggregation, correlation rules, threat detection, and security monitoring operations.

# SIEM, Security Logging and Monitoring

Introduction

Security Information and Event Management (SIEM) platforms aggregate, correlate, and analyze security events from across the enterprise infrastructure. Effective security logging and monitoring enable detection of attacks in progress, forensic investigation after incidents, and compliance with regulatory requirements.

This tutorial provides a comprehensive guide to SIEM implementation, log aggregation, correlation rule development, threat detection, and security operations center (SOC) workflows. The principles apply to Splunk, Elastic Security (ELK), Microsoft Sentinel, IBM QRadar, and other SIEM platforms.

Learning Objectives

By the end of this tutorial, you will be able to:

Design log aggregation architecture across distributed infrastructure
Configure log sources (firewalls, servers, applications, cloud)
Develop SIEM correlation rules for threat detection
Create dashboards and alerts for security operations
Implement log retention and tamper-proof storage
Integrate SIEM with ticketing and incident response
Tune rules to reduce false positives

Technical Background

SIEM Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Log       │    │   Log       │    │   Log       │
│  Sources    │───▶│  Collectors │───▶│   SIEM      │
│ (Firewalls, │    │ (Agents,    │    │ (Index,     │
│  Servers,   │    │  Syslog,    │    │  Correlate, │
│  Apps,      │    │  Cloud API) │    │  Alert)     │
│  Cloud)     │    └─────────────┘    └──────┬──────┘
└─────────────┘                             │
                                            ▼
                                     ┌─────────────┐
                                     │   Analyst   │
                                     │   Console   │
                                     │ (Dashboards,│
                                     │  Alerts,    │
                                     │  Reports)   │
                                     └─────────────┘

Log Types and Sources

| Log Source | Log Type | Location | Value | |------------|----------|----------|-------| | Firewall | Connection logs, ACL hits | /var/log/firewall, syslog | Network traffic, blocked connections | | Web Server | Access logs, error logs | /var/log/nginx/, /var/log/apache2/ | Web attacks, suspicious URLs | | Linux OS | Auth, syslog, audit | /var/log/auth.log, /var/log/syslog, /var/log/audit/ | Login activity, privilege escalation | | Windows | Security, System, Application | Event Viewer | Logon events, process creation, policy changes | | Active Directory | Security, Directory Service | Event Viewer | Account lockouts, Kerberos tickets, GPO changes | | Database | Query logs, audit logs | DB-specific | SQL injection, unauthorized queries | | Cloud | CloudTrail, Activity Log | AWS CloudTrail, Azure Activity Log | API calls, resource changes | | Endpoint | EDR logs | CrowdStrike, Defender, SentinelOne | Malware detection, process activity |

Log Levels

| Level | Description | Example | |-------|-------------|---------| | DEBUG | Detailed diagnostic information | Variable values, function entry | | INFO | General operational messages | Service started, user logged in | | WARN | Potentially harmful situation | Connection retry, disk at 85% | | ERROR | Error event, application still running | Failed to connect to database | | FATAL | Severe error, application may crash | Kernel panic, service terminated |

SIEM Processing Pipeline

Collection → Parsing → Normalization → Enrichment → Correlation → Storage → Alerting
    │          │           │               │            │          │
    ▼          ▼           ▼               ▼            ▼          ▼
Raw logs   Structured   Common schema   Threat intel   Pattern    Notifications
           data         (ECS, OCSF)     lookup         matching   Ticket creation

Prerequisites

Understanding of log formats (syslog, JSON, CSV, Windows Event)
Familiarity with regular expressions for log parsing
Knowledge of network protocols and common service logs
Understanding of security events and attack patterns

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | SIEM Platform | Splunk Enterprise, Elastic Security, Microsoft Sentinel, IBM QRadar, Wazuh | | Log Forwarders | Fluentd, Logstash, Filebeat, Winlogbeat, Syslog-ng, NXLog | | Storage | Hot (SSD, 90 days), Warm (HDD, 1 year), Cold (S3/Glacier, 7 years) | | Compute | 8+ vCPU, 32GB+ RAM for SIEM server | | Network | Dedicated log network or VLAN |

Step-by-Step Implementation

Phase 1: Log Collection Architecture

#### 1.1 Define Log Sources and Forwarders

Log Source           | Forwarder      | Protocol    | SIEM Input
---------------------|----------------|-------------|-------------
Linux servers        | Filebeat       | TCP 5044    | Elasticsearch
Windows servers      | Winlogbeat     | TCP 5044    | Elasticsearch
Firewalls            | Syslog         | UDP 514     | Syslog listener
Web servers          | Filebeat       | TCP 5044    | Elasticsearch
Cloud (AWS)          | Lambda → Kinesis | HTTP     | S3 → SIEM
Cloud (Azure)        | Event Hub      | AMQP        | Azure Monitor
Database audit       | Custom script  | TCP 514     | Syslog listener

#### 1.2 Configure Filebeat on Linux

# /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/auth.log
    - /var/log/syslog
    - /var/log/audit/audit.log
  fields:
    log_type: linux-system
    environment: production
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields:
    log_type: nginx-access
    environment: production
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://siem-01:9200"]
  index: "filebeat-%{+yyyy.MM.dd}"

setup.kibana:
  host: "http://siem-01:5601"

sudo filebeat modules enable system
sudo filebeat setup
sudo systemctl enable --now filebeat

#### 1.3 Configure Winlogbeat on Windows

# Install Winlogbeat
.\install-service-winlogbeat.ps1

# Configure C:\Program Files\Winlogbeat\winlogbeat.yml
winlogbeat.event_logs:
  - name: Application
    level: error
  - name: Security
    level: error, warning, information, audit-success, audit-failure
  - name: System
    level: error, warning

output.elasticsearch:
  hosts: ["http://siem-01:9200"]

Phase 2: Centralized Syslog

#### 2.1 Configure rsyslog Server

On the syslog collector (`/etc/rsyslog.conf`):

# Enable UDP and TCP syslog reception
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")

# Template for remote hosts
$template RemoteHost,"/var/log/syslog/remote/%HOSTNAME%/%PROGRAMNAME%.log"

# Route remote logs to separate files
if $fromhost-ip != '127.0.0.1' then ?RemoteHost
& stop

# Forward to SIEM
*.* @@siem-01:514

sudo systemctl restart rsyslog
sudo ufw allow 514/udp
sudo ufw allow 514/tcp

#### 2.2 Configure Client to Forward to Syslog

On each Linux client (`/etc/rsyslog.d/50-siem.conf`):

*.* @@10.0.1.100:514

sudo systemctl restart rsyslog

Phase 3: SIEM Data Parsing and Normalization

#### 3.1 Elastic Common Schema (ECS)

ECS provides a common field naming convention:

| ECS Field | Description | Example | |-----------|-------------|---------| | `@timestamp` | Event timestamp | `2024-01-15T10:30:00.000Z` | | `event.action` | Action performed | `login`, `file_create`, `network_connect` | | `event.outcome` | Outcome | `success`, `failure`, `denied` | | `source.ip` | Source IP address | `10.0.1.50` | | `destination.ip` | Destination IP | `10.0.50.20` | | `user.name` | Username | `admin` | | `host.hostname` | Hostname | `web-server-01` | | `process.name` | Process name | `sshd`, `nginx` | | `network.protocol` | Protocol | `tcp`, `http` |

#### 3.2 Ingest Pipeline (Elasticsearch)

PUT _ingest/pipeline/syslog-parse
{
  "description": "Parse syslog messages",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
        ]
      }
    },
    {
      "date": {
        "field": "syslog_timestamp",
        "target_field": "@timestamp",
        "formats": ["MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
      }
    },
    {
      "remove": {
        "field": ["syslog_timestamp", "syslog_hostname"]
      }
    }
  ]
}

Phase 4: Correlation Rules and Detection

#### 4.1 Common Detection Rules

**Rule 1: Multiple Failed Logins (Brute Force)**

Index: filebeat-*
Query: event.module:ssh AND event.outcome:failure
Group by: source.ip
Threshold: > 5 failures within 5 minutes
Action: Alert + add source IP to blocklist

Elastic Security rule:

{
  "rule": {
    "type": "query",
    "query": "event.module:sshd AND event.outcome:failure",
    "risk_score": 45,
    "severity": "medium",
    "name": "SSH Brute Force Attempt",
    "description": "Multiple failed SSH login attempts from a single source IP",
    "throttle": "no_actions",
    "actions": [
      {
        "id": "alert",
        "config": {
          "group_by": ["source.ip"],
          "threshold": 5
        }
      }
    ]
  }
}

**Rule 2: Impossible Travel**

User logs in from two geographically distant locations within a short time
Example: Login from New York at 09:00, login from London at 09:30

**Rule 3: Privilege Escalation**

Index: winlogbeat-*
Query: event.code:4672 (Special privileges assigned to new logon)
Group by: user.name, host.hostname
Threshold: > 1 within 10 minutes from unusual host

**Rule 4: PowerShell Execution**

Index: winlogbeat-*
Query: event.code:4104 (PowerShell script block logging)
Query: process.command_line:*Invoke-Expression* OR *DownloadString* OR *Execute*
Action: Alert

#### 4.2 Splunk SPL Example

# Detect brute force
index=ssh sourcetype=sshd "Failed password"
| stats count by src
| where count > 5
| eval risk_score = count * 10
| sort - risk_score

# Detect lateral movement
index=windows EventCode=4624 LogonType=3
| stats count by src_user, dest
| where count > 10
| eval risk_level = "high"

#### 4.3 Sigma Rule Format

Sigma is a generic signature format for SIEM rules:

title: SSH Brute Force Attack
status: stable
description: Detects multiple failed SSH login attempts from a single source
references:
  - https://attack.mitre.org/techniques/T1110/
author: ITYantra Security Team
date: 2024-01-15
logsource:
  category: ssh
  product: linux
detection:
  selection:
    event.outcome: failure
    event.module: sshd
  condition: selection
  timeframe: 5m
  threshold: 5
fields:
  - source.ip
  - user.name
falsepositives:
  - Automated scripts with expired credentials
level: medium
tags:
  - attack.t1110
  - attack.credential_access

Phase 4b: Threat Intelligence Integration

#### 4.1 IOC Feeds

Threat Intelligence Sources:
  - AlienVault OTX (free)
  - Abuse.ch SSL Blacklist, URLhaus
  - CISA Alerts
  - Commercial: Recorded Future, Mandiant, CrowdStrike

#### 4.2 Indicator Matching

SIEM Lookup:
  - IP blacklist: Match source/destination IPs against threat feed
  - Domain blacklist: Match DNS queries against known malicious domains
  - File hash blacklist: Match file hashes against malware repositories

Phase 5: Dashboards and Alerting

#### 5.1 SOC Dashboard Panels

Panel 1: Events per Hour (last 24h)
Panel 2: Top 10 Source IPs by Event Count
Panel 3: Top 10 Alerts by Severity
Panel 4: Failed Logins by Username
Panel 5: Blocked Connections by Firewall Rule
Panel 6: Geolocation of Login Attempts (map)
Panel 7: Critical Alerts Requiring Attention
Panel 8: Mean Time to Detect (MTTD) trend

#### 5.2 Alert Severity Levels

| Severity | Description | Response Time | |----------|-------------|---------------| | Critical | Active breach, data exfiltration, ransomware | 15 minutes | | High | Successful attack, privilege escalation | 1 hour | | Medium | Suspicious activity, policy violation | 4 hours | | Low | Informational, policy near-violation | 24 hours |

#### 5.3 Alert Escalation

Level 1 (L1 SOC): Initial triage, false positive filtering
Level 2 (L2 SOC): Investigation, containment recommendation
Level 3 (L3 SOC): Incident response, threat hunting
Level 4 (Management): Executive notification, regulatory reporting

Phase 6: Log Retention and Tamper Protection

#### 6.1 Retention Policy

| Log Type | Hot Storage | Warm Storage | Cold Storage | Total Retention | |----------|-------------|--------------|--------------|-----------------| | Firewall logs | 90 days | 1 year | 7 years | 7 years | | Authentication logs | 90 days | 1 year | 7 years | 7 years | | Application logs | 30 days | 90 days | 1 year | 1 year | | Database audit logs | 90 days | 1 year | 7 years | 7 years | | EDR logs | 30 days | 90 days | 1 year | 1 year |

#### 6.2 Tamper Protection

1. Write-once storage (WORM) for critical logs
2. Digital signatures on log entries
3. Forward logs to separate, hardened SIEM server
4. Restrict delete/modify permissions on log storage
5. Regular integrity checks (hash verification)
6. SIEM admin access via MFA and jump hosts only

Verification and Expected Outputs

| Check | Method | Expected Result | |-------|--------|-----------------| | Log ingestion | Check SIEM index count | Logs from all sources present | | Parsing accuracy | Search for parsed fields | Fields correctly extracted (source.ip, user.name, etc.) | | Correlation rule | Generate test event | Alert fires within expected timeframe | | Alert notification | Trigger test alert | Email/Slack/ticket created | | Dashboard refresh | Open SIEM dashboard | Data updated within 5 minutes | | Log retention | Check oldest available log | Matches retention policy |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Logs not appearing in SIEM | Forwarder not running or misconfigured | Check forwarder status, verify output config | | High SIEM storage usage | Too much logging, long retention | Adjust retention, filter noisy log sources | | False positive alerts | Overly broad correlation rules | Tune rule thresholds, add exclusions | | Delayed alerts | Indexing lag, rule evaluation interval | Increase resources, reduce rule frequency | | Missing fields after parsing | Grok pattern mismatch | Update ingest pipeline, test with sample logs | | Timestamp errors | Timezone misconfiguration | Standardize all logs to UTC |

Security Best Practices

**Log everything, store strategically:** Collect all logs but apply retention based on value and compliance
**Use structured logs:** JSON is easier to parse than unstructured text
**Centralize log collection:** All logs flow to SIEM, not stored locally only
**Protect log integrity:** SIEM servers should be hardened, access restricted
**Alert on silence:** Alert when a critical log source stops sending logs
**Regular tuning:** Review and adjust correlation rules monthly based on false positive rates
**Test detection:** Regularly simulate attacks to verify detection works
**Document detections:** Maintain a detection catalog with rule purpose, MITRE ATT&CK mapping, and owner

Enterprise Considerations

**Multi-region SIEM:** Deploy SIEM collectors in each region, central index in HQ
**Cloud-native SIEM:** Use cloud SIEM (Sentinel, Cloud SIEM) for cloud workloads
**Data residency:** Store logs in the same jurisdiction as the data source
**SIEM as a service:** Consider managed SIEM for organizations without SOC staffing
**Log volume licensing:** SIEM licensing often based on GB/day; optimize log volume
**Threat hunting:** Proactive searching beyond automated alerts
**SOAR integration:** Security Orchestration, Automation, and Response for automated incident handling

Conclusion

SIEM and security monitoring are essential for detecting and responding to security incidents. By aggregating logs from all sources, normalizing them into a common schema, applying correlation rules, and maintaining actionable dashboards, organizations can detect attacks early and respond effectively. The key to SIEM success is not just collecting data—it's tuning detection rules, reducing false positives, and maintaining skilled analysts to investigate alerts.

Frequently Asked Questions

**Q: SIEM vs. log management—what's the difference?** A: Log management focuses on collection, storage, and search. SIEM adds correlation, analysis, alerting, and threat detection on top of log management.

**Q: How much storage does a SIEM need?** A: Depends on log volume (GB/day) and retention period. A medium enterprise might generate 50-200 GB/day and require 90 days hot + 1 year warm storage.

**Q: Can SIEM prevent attacks?** A: No. SIEM detects and alerts on attacks. Prevention requires firewalls, EDR, IAM, and other controls. SIEM integrates with these for automated response.

Official Technical References

Elastic Common Schema (ECS): https://www.elastic.co/guide/en/ecs/current/
Splunk Search Processing Language: https://docs.splunk.com/
Sigma Rules: https://github.com/SigmaHQ/sigma
MITRE ATT&CK: https://attack.mitre.org/
NIST SP 800-92: Guide to Computer Security Log Management: https://csrc.nist.gov/publications/detail/sp/800-92/final