Logs & Auditing

2 minuti

Logs are the memory of a Linux system: without them, an incident is just a mystery. This note covers where logs live, how to read them, and how to watch files for tampering.

Where logs live

Two systems coexist on modern distros:

  • Traditional text logs in /var/log: auth.log (Debian/Ubuntu) or secure (RHEL/Fedora) for authentication events, syslog / messages for general events, kern.log for the kernel.
  • systemd-journald: the binary log store queried with journalctl.
journalctl -u ssh --since "1 hour ago"   # only ssh, last hour
journalctl -p err -b                     # errors since boot
journalctl -f                            # live tail (like tail -f)

Anatomy of auth.log

grep "Failed password" /var/log/auth.log
 
# Output
Nov  6 09:12:44 web01 sshd[2412]: Failed password for invalid user admin from 203.0.113.7 port 51234 ssh2
Nov  6 09:12:47 web01 sshd[2412]: Failed password for root from 203.0.113.7 port 51240 ssh2

The interesting patterns to hunt for:

  • Failed password / Invalid user bursts → brute force (see SSH hardening in Hardening & Access Control)
  • Accepted password from an unexpected country or at 3 a.m.
  • session opened / sudo: lines → who escalated to root and when
  • authentication failure on su/sudo → local privilege probing

auditd: watch what changes

The Linux Audit Framework records who touched what, independent of application logs.

sudo apt install auditd
 
# watch /etc/shadow for write/attribute changes (persistent in /etc/audit/rules.d/)
sudo auditctl -w /etc/shadow -p wa -k shadow_watch
 
ausearch -k shadow_watch      # search events by key
aureport --auth               # authentication summary

Rule ideas worth keeping: /etc/passwd, /etc/shadow, /etc/sudoers, /etc/cron.d, sshd_config. That covers most of the persistence and escalation paths described in Processes & Persistence.

What to hunt for during an incident

  • login bursts, successful logins after a failure storm
  • new cron jobs or systemd timers (persistence)
  • new SUID binaries (see SUID, SGID & File Capabilities)
  • processes listening on unexpected ports (ss -tulpn)
  • journal entries that suddenly stop → the attacker may be wiping logs

Retention and the “logs can lie” rule

  • logrotate compresses and ages out old logs — check /etc/logrotate.d/ so evidence survives long enough
  • a local attacker with root can rewrite any local log → for anything serious, ship logs off the machine (rsyslog/remote journald → SIEM). Monitoring is a first-class citizen of security operations, and a great excuse to finally read the Linux Forensics Fundamentals note too.