Why the Switch
Graylog served us well as a first SIEM — it's free, runs in Docker, and handles syslog ingestion without much fuss. But as the lab grew, limitations became clear:
- Resource contention. Graylog + OpenSearch + MongoDB shared a Docker host (cainfra01) with LibreNMS, Gitea, MkDocs, and six other services. OpenSearch alone wanted 2GB+ of heap.
- Operational experience. Graylog's web UI is functional but the admin workflow
(extractors, streams, pipelines) doesn't map to how enterprise SIEM tools work. Splunk's
inputs.conf/props.conf/transforms.confmodel is the industry standard. - Search performance. Complex queries on OpenSearch were sluggish with limited resources.
Splunk Enterprise with a Developer License (10GB/day) gives us the real admin experience — native service management, filesystem-based configuration, and proper resource isolation on a dedicated VM.
The Migration
1. New VM: splunk01
Deployed a dedicated Ubuntu 22.04 VM on bighost:
- CPU: 8 vCPU
- RAM: 24 GB
- Disk: 100 GB
- IP: Static on the DMZ subnet
Native install — no Docker. Splunk runs as a systemd service under its own user, with full access to the filesystem for index management and configuration.
2. Input Configuration
Three input types configured to match our existing log sources:
- Syslog (UDP/TCP 1514) — Linux hosts, Proxmox nodes, network devices
- HTTP Event Collector (port 8088) — structured JSON events from applications
- Splunk forwarder receiving (port 9997) — Universal Forwarders on Windows DCs
3. Log Source Redirection
Pointed all existing syslog sources at splunk01 instead of cainfra01:
- cainfra01: rsyslog forwarding updated
- bighost: Proxmox syslog target changed
- GCP VM: rsyslog over WireGuard tunnel redirected
- Windows DCs: Splunk Universal Forwarder installed, replacing Winlogbeat
4. DNS Integration
Added splunk.rpc-cyberflight.com as a DNS record in Active Directory, with nginx
reverse proxy on cainfra01 handling TLS termination.
What We Gained
| Aspect | Graylog | Splunk |
|---|---|---|
| Deployment | Docker (shared host) | Native (dedicated VM) |
| Search | OpenSearch queries | SPL (industry standard) |
| Windows logs | Winlogbeat | Universal Forwarder |
| Config model | Web UI only | Filesystem conf files + UI |
| Resource isolation | Shared with 8+ services | Dedicated 8 vCPU / 24 GB |
| License | Open Source | Developer (10 GB/day) |
Key Takeaways
- Native installs teach more than Docker. Managing Splunk as a systemd service, editing conf files, and understanding the index/bucket lifecycle is the experience that translates to enterprise environments.
- Dedicated VMs for heavy services. SIEM workloads don't play well with shared Docker hosts. Give them their own resources.
- SPL is worth learning. Splunk's search language is the lingua franca of SOC analysts. Time invested here compounds.