Security Logging: What to Keep, for How Long, and What to Ignore

Every post-incident review contains one of two sentences: 'the logs showed exactly what happened' or 'we could not determine how the attacker got in.' The difference was decided months earlier, by what someone chose to record.

The investigation core

When responders reconstruct an intrusion they reach for the same few streams every time: authentication events (successes AND failures, with source addresses), process execution with parent-child lineage, network flow summaries (who talked to whom, when, how much — payloads rarely needed), and change events on the crown jewels: new users, permission grants, cron and service modifications, configuration writes.

What you can mostly ignore

Debug chatter, load-balancer health checks, and verbose application traces dominate volume while contributing almost nothing forensically. Sample them or drop them at ingest. The goal is not completeness; it is answerable questions.

Retention that matches reality

Breach discovery commonly lags intrusion by weeks or months. Ninety days of hot, searchable logs is a defensible floor for the core streams; a year of cold, cheap archive covers the long tail and most compliance regimes. Whatever the number, write it down — an accidental retention policy is the kind auditors and courts enjoy least.

Protect the logs themselves

Attackers edit history where they can. Ship logs off-host promptly, restrict deletion rights, and let the collector append-only. A modest, tamper-resistant log pipeline beats a lavish one the intruder can prune — because the only log that matters is the one that survives the incident it describes.

Start from the questions you will need to answer

The best way to decide what to log is to imagine the questions an investigation will ask. Who signed in to this system, from where and when? What did they do after signing in? Which processes ran on the affected server, and what started them? Which external addresses did it contact? What changed in configuration, permissions or scheduled tasks? Was data copied out, and how much? Every log source you keep should help answer at least one of these. Sources that answer none of them are candidates for sampling or deletion.

Identity and access logs

Identity is where most intrusions start, so identity logs are the most valuable. Keep sign-in events from your identity provider, email system, VPN and remote-access tools, including failures, multi-factor challenges and the device or network used. Record privilege changes: new administrators, new application permissions, password resets and changes to multi-factor settings. These events are relatively small in volume and enormously useful, so they deserve long retention.

Endpoint and server logs

On servers and laptops, the most useful records describe process execution with command lines and parent processes, the creation of services, scheduled tasks and startup entries, changes to security tools and the loading of unusual drivers or modules. Operating systems record some of this by default; endpoint security tools and system monitoring utilities can add more detail. Collect these events centrally, because local logs are the first thing an intruder with administrative access tries to clear.

Network and DNS logs

Network logs answer the questions of reach and movement. Flow records — source, destination, port, time and volume — show which machines talked to each other and how much data moved, without the storage cost of full packet capture. DNS query logs are particularly efficient: almost every connection begins with a name lookup, so they reveal contact with malicious domains, unusual services and data hidden in DNS traffic. Proxy and firewall logs complete the picture for traffic leaving the network.

Cloud and SaaS audit logs

Cloud platforms and software-as-a-service applications keep audit logs of administrative actions, but defaults vary: some features must be switched on, some logs are kept only briefly and some are available only on higher subscription tiers. Check each important service, enable audit logging everywhere it matters and export the logs to your own storage, where you control retention. Pay particular attention to actions such as creating access keys, changing sharing settings, disabling logging and granting third-party applications access to data.

Make logs usable: time, format and context

Logs are only useful if they can be correlated. Synchronize clocks on every system using a reliable time source, record times with time zones or in a single standard zone, and use structured formats that tools can parse. Add context at collection time where you can: host names, environment, owner and importance of the asset. A consistent naming scheme for systems and accounts saves hours during an investigation.

Privacy and minimization

Security logs inevitably contain personal data — usernames, addresses, activity — so they fall under privacy laws in many jurisdictions. Collect what security genuinely needs, avoid logging content such as message bodies or full form submissions unless there is a specific reason, restrict access to the logs and record who searches them. Define retention by purpose and delete data when it is no longer needed. Good minimization reduces both legal risk and storage cost.

Tiered storage keeps costs sensible

Not every log needs to be searchable instantly for a year. A common design keeps recent data — weeks to a few months — in a fast, searchable system for detection and investigation, and moves older data to cheaper storage where it can still be retrieved when needed. Compress archives, keep them immutable where possible and test that they can actually be searched or restored. This approach lets you keep long histories of the important sources without an unsustainable bill.

Test that the logs answer real questions

Periodically run a small exercise: pick a realistic scenario, such as a stolen password used to sign in and download files, and try to reconstruct it from your logs. Note every question you could not answer and every gap in time or coverage. Those gaps become the next improvements to your logging. The exercise also trains the people who will one day need to perform the investigation for real.

A short policy template

A useful logging policy fits on a page: the sources that must be collected, where they are stored, who may access them, how long each class is retained, how integrity is protected and who reviews coverage. Keep it practical and revisit it when systems change. A clear, modest policy that is actually followed produces far better evidence than an ambitious one that exists only on paper.

Application logs worth keeping

Most application logs are operational, but some events are security records in disguise and deserve the same care as system logs. Keep sign-ins and sign-outs within the application, password and email changes, permission changes, exports and bulk downloads, administrative actions, API key creation and failed authorization attempts. Record who acted, on what, when and from where. These events let you answer the questions customers ask after an incident — was my account accessed, was my data exported — and they are usually small enough to keep for a long time.

Integrity: keeping logs trustworthy

Logs are evidence, and evidence is only useful if it can be trusted. Ship logs off the originating host quickly, ideally in near real time, so that an intruder cannot quietly edit them. Store them where the systems that produce them cannot delete or modify them, using write-once storage or separate accounts with restricted permissions. Record when collection fails, because a gap in the logs is itself an important event. For particularly sensitive records, cryptographic chaining or signing can show that nothing has been altered since collection.

Retention guidelines by source

Retention should follow usefulness and obligation. Identity, privilege-change and audit events are small and valuable, so they deserve long retention, often a year or more. Process and network summaries are larger; keeping a few months searchable and a longer archive is a common compromise. Verbose debug and health-check logs rarely help investigations and can be kept for days or sampled. Check the requirements that apply to you — sector regulations, contracts and data-protection rules — and document the chosen periods so they are applied consistently.

Who should have access

Security logs reveal a great deal about people and systems, so access should be limited to those who need it for detection, investigation and audit. Grant read access through named accounts, log every search and export, and review the list of people with access regularly. Administrators of the systems being logged should not, ideally, be able to delete or alter the logs of their own actions; separating these duties is one of the simplest ways to make logs credible.

When to call in help

Some investigations exceed what a small team can do alone, particularly when legal action, regulators or insurers are involved. Preserving logs carefully from the start keeps those options open. If an incident may become a legal matter, record who collected which data, when and how, and avoid modifying original files; work on copies instead. External investigators can achieve much more when the evidence has been handled this way.

Good logging is unglamorous work done long before anyone needs it. Decide which questions you must be able to answer, collect the few sources that answer them, protect those records and keep them for as long as they are useful. When the day comes, the logs will tell the story instead of leaving it to guesswork.