Logging and Audit Trails Worth Turning On

Logging has an unusual property among controls: it is worthless retroactively. Every other measure can be applied when you notice a gap. An audit trail only covers the period after you switched it on.

That is the whole argument for enabling it on day one, before there is anything interesting to see.

The layers

The control-plane audit trail

Records API calls against the account: who created, changed or deleted what, from which address, and when. This is the log that answers the questions that matter most after an incident, and it is the one most often left off.

Enable it across all regions. An audit trail covering one region leaves the rest of the account dark.

Resource logs

Load balancer access logs, storage access logs, database logs. Useful for diagnosis, and sometimes voluminous, so apply retention deliberately.

Operating system logs

Authentication attempts, service starts and failures, kernel messages. These live on the machine and disappear with it, which is the reason to ship them somewhere else.

Application logs

Yours to design. Structured logs with a request identifier are dramatically more useful than free-form text when something goes wrong across several services.

Where logs should live

Two properties matter. They should survive the resource they describe, and they should be hard to alter.

If a server is compromised or deleted, logs stored only on that server are gone precisely when needed. Ship them off the machine. For audit trails, use write-once or restricted-deletion storage where the provider offers it, so that someone who gains access cannot quietly remove the evidence of it.

Retention with intent

Logs cost storage, and verbose logging shipped off-platform also costs transfer, as noted in bandwidth and egress. Pick a retention period for each category rather than keeping everything forever by accident. Audit trails usually justify a longer period than debug logs.

Alerting on the few things that matter

Nobody reads logs continuously. A handful of alerts covers most of the value: root or administrator usage, changes to identity and permissions, disabling of logging itself, and unusual volumes of failed authentication.

The last of those is routine noise on any public server, so alert on unusual rates rather than on occurrence.

On inherited accounts

Enable logging as part of the first hour, and understand that it tells you nothing about what happened before. That gap is one reason to treat every pre-existing credential as compromised, as described in the first hour after taking over a cloud account.