Logs are the memory of a Linux system: without them, an incident is just a mystery. This note covers where logs live, how to read them, and how to watch files for tampering.
Where logs live
Two systems coexist on modern distros:
- Traditional text logs in
/var/log:auth.log(Debian/Ubuntu) orsecure(RHEL/Fedora) for authentication events,syslog/messagesfor general events,kern.logfor the kernel. - systemd-journald: the binary log store queried with
journalctl.
journalctl -u ssh --since "1 hour ago" # only ssh, last hour
journalctl -p err -b # errors since boot
journalctl -f # live tail (like tail -f)Anatomy of auth.log
grep "Failed password" /var/log/auth.log
# Output
Nov 6 09:12:44 web01 sshd[2412]: Failed password for invalid user admin from 203.0.113.7 port 51234 ssh2
Nov 6 09:12:47 web01 sshd[2412]: Failed password for root from 203.0.113.7 port 51240 ssh2The interesting patterns to hunt for:
Failed password/Invalid userbursts → brute force (see SSH hardening in Hardening & Access Control)Accepted passwordfrom an unexpected country or at 3 a.m.session opened/sudo:lines → who escalated to root and whenauthentication failureonsu/sudo→ local privilege probing
auditd: watch what changes
The Linux Audit Framework records who touched what, independent of application logs.
sudo apt install auditd
# watch /etc/shadow for write/attribute changes (persistent in /etc/audit/rules.d/)
sudo auditctl -w /etc/shadow -p wa -k shadow_watch
ausearch -k shadow_watch # search events by key
aureport --auth # authentication summaryRule ideas worth keeping: /etc/passwd, /etc/shadow, /etc/sudoers, /etc/cron.d, sshd_config. That covers most of the persistence and escalation paths described in Processes & Persistence.
What to hunt for during an incident
- login bursts, successful logins after a failure storm
- new cron jobs or systemd timers (persistence)
- new SUID binaries (see SUID, SGID & File Capabilities)
- processes listening on unexpected ports (
ss -tulpn) journalentries that suddenly stop → the attacker may be wiping logs
Retention and the “logs can lie” rule
logrotatecompresses and ages out old logs — check/etc/logrotate.d/so evidence survives long enough- a local attacker with root can rewrite any local log → for anything serious, ship logs off the machine (rsyslog/remote journald → SIEM). Monitoring is a first-class citizen of security operations, and a great excuse to finally read the Linux Forensics Fundamentals note too.