Linux Server Anomaly Monitoring, Detection, and Prevention

1. Monitoring Tools

• systemd-journald – logs (journalctl)
• htop, atop – resource monitoring
• vnStat, iftop – network traffic
• Auditd – audit events
• Prometheus + Grafana – metrics and dashboards

sudo pacman -S htop atop vnstat iftop audit

2. Log Monitoring

journalctl -p err..alert --since "1 hour ago"

journalctl -u sshd
journalctl -u httpd
journalctl -u mariadb

ausearch -m USER_LOGIN,USER_CMD

3. Anomaly Detection

• Unusual CPU, RAM, or disk spikes
• Unknown listening ports
• Failed login attempts
• Web server 404/403 spikes (potential scans)
• MariaDB slow queries or access anomalies

ps aux --sort=-%cpu | head
ss -tuln
find /etc /bin -type f -mtime -1

grep "access denied" /var/log/mysql/*

tail -n 100 /var/log/httpd/access_log

4. Prevention Techniques

• Firewall: ufw, iptables
• Keep system and services updated
• Harden SSH
• Use Fail2Ban
• Limit HTTP methods, enable mod_security for Apache
• Harden MariaDB (disable remote root, remove test DB)

SSH Hardening:

PermitRootLogin no
PasswordAuthentication no
AllowUsers youruser
    

Installing and Configuring Fail2Ban:

sudo pacman -S fail2ban
sudo systemctl enable --now fail2ban
    

Example Jail Config (SSH + Apache):

[sshd]
enabled = true
port = ssh
logpath = /var/log/auth.log
maxretry = 3

[apache-auth]
enabled = true
port = http,https
logpath = /var/log/httpd/error_log
maxretry = 5
    

Restart:

sudo systemctl restart fail2ban

Apache Hardening:

sudo pacman -S apache
sudo nano /etc/httpd/conf/httpd.conf

# Add or enable:
ServerTokens Prod
ServerSignature Off
TraceEnable Off

   Options -Indexes

    

Enable mod_security (optional):

sudo pacman -S modsecurity-apache

MariaDB Hardening:

sudo mysql_secure_installation

• Disable remote root
• Remove test DB
• Use strong passwords

Monitor queries:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;
SHOW VARIABLES LIKE 'slow_query_log%';
    

5. DDoS Mitigation

• Rate limiting with iptables or nftables
• Fail2Ban bans IPs sending too many requests
• Use reverse proxies (e.g. Nginx, Cloudflare) if public

Iptables Rate Limiting (Port 80):

iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j DROP
    

Block SYN flood:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
iptables -A INPUT -f -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
    

6. Automating Monitoring

#!/bin/bash
journalctl -p err..alert --since "1 hour ago" > /var/log/hourly_errors.log
fail2ban-client status >> /var/log/hourly_errors.log
vnstat --oneline >> /var/log/hourly_errors.log
    

Place script in /etc/cron.hourly/ or systemd timer.

7. Optional Tools

• Falco – kernel-level anomaly detection
• Wazuh – SIEM
• GoAccess – real-time Apache log analyzer

sudo pacman -S goaccess

goaccess /var/log/httpd/access_log -o report.html --log-format=COMBINED