In May 2026, the federal government reversed five years of its own advice on security log monitoring and told agencies to stop trying to keep everything. The Office of Management and Budget rescinded OMB M-21-31 and told agencies to log by risk instead, after concluding that keeping vast quantities of data "without clear utility" had been neither feasible nor cost-effective for most of them. The reversal is worth sitting with because it puts a name to something security teams have been absorbing for years without saying out loud. The logs you keep to satisfy an auditor and the logs you need to catch an intruder are rarely the same logs, and they behave nothing alike once you account for who reads them, how fast they have to be reachable, and what they cost to keep live.
That gap between the two is the whole subject of this piece. One budget usually funds both jobs, and when they compete for the same money, detection quietly loses out because compliance volume is mandated and predictable, while the sources that catch attackers look optional right up until the morning they would have mattered. This is not a rare edge case. In Strike48's 2026 survey of 100 security leaders, 84% said their current tools cannot reach all of their log data during an investigation, and 80% called the cost of keeping logs hot and searchable a real budget problem. What follows separates the two jobs, shows why funding them as one breaks both, and lays out the architecture that gives each one what it needs.
Security log monitoring is the collection, retention, and analysis of log data for two distinct purposes: proving that required data was maintained with integrity over a defined period and identifying an attacker who is moving through your environment right now. Those are two different activities, even though they share a name and often share a budget line. Compliance monitoring asks a retrospective question about whether the record is complete. Detection monitoring asks an operational question about whether the right data is queryable in time to matter. Logging is the collection-and-retention half; monitoring is the analysis-and-alerting half, and the two consumers weigh them very differently.
NIST SP 800-92 defines the log management process and the distinction between collection, retention, and analysis, and its in-progress Rev. 1 reframes that guidance as a planning playbook. On the detection side, OWASP A09:2021 names insufficient logging and monitoring as a top-ten application security risk. Across enterprise deployments, Strike48 keeps encountering the same issue: a team describes one logging program and then realizes it has been quietly running two.
Retention windows and integrity requirements come first. Compliance owners need log data to be present and unaltered over a defined window. NIST SP 800-92 lays out the log management process and what makes audit evidence hold up, but the retention clock itself comes from whichever framework applies, whether that is PCI DSS, HIPAA, or a federal mandate. On the federal side, OMB M-21-31 spent five years defining a four-tier maturity model (EL0 through EL3) before OMB rescinded it and issued M-26-14 in May 2026, a shift that reporting ties directly to the cost of holding data almost no one was querying. Retention windows still commonly run one to three years, depending on the framework and the sensitivity of the data.
The compliance failure mode is retrospective. This pipeline fails when evidence is missing at audit time because a source was never connected, a window expired, or integrity was not maintained. That gap surfaces months after it was created, during an investigation or an audit, not during the incident it should have documented.
Storage economics is where the tension starts. Compliance owners need data that is retrievable, not queryable in real time, and cold or warm storage meets that bar at a fraction of the cost of hot-tier ingestion. This is where the two jobs first pull apart over money. A compliance archive that gets read once a year, if ever, ends up sitting in the same expensive hot tier as the detection sources an analyst queries fifty times a shift, priced as though both need the same speed. When compliance logs do not need to be hot, paying hot-tier rates for them is the structural mistake everything here keeps coming back to.
Two pipelines share the vocabulary of security log monitoring, and they diverge across every attribute that determines cost. The table below defines both by consumer, SLA, retention window, storage tier, and failure mode, because the split depends on four things being true at once and prose alone flattens that.
Detection coverage is defined by MITRE ATT&CK data sources. Detection engineers do not need every log; they need the specific data sources that MITRE ATT&CK maps to attacker techniques: DNS query data for C2 beaconing, process-creation events for lateral movement, authentication logs from non-priority systems for credential abuse, and network flow records. When budget forces coverage tradeoffs, these tend to be the first sources cut because they are expensive to ingest at volume. The consequence is direct: attacks that exercise those paths generate no alerts, and no alerts mean the investigation never starts.
OWASP A09:2021 names this failure mode. Insufficient logging and monitoring earn a top-ten spot because teams treat monitoring as a single activity and underinvest in the detection half. That is an operational pattern, not a theoretical risk. In the same Strike48 survey, 65% of security leaders said an investigation had stalled for them because the data they needed was trapped in a system their tools could not reach, which puts the detection failure closer to the norm than the exception.
The two SLAs are structurally incompatible. Compliance needs data present when the auditor asks; detection needs it queryable the instant an analyst runs a correlation. Different technical requirements, different storage economics, different failure modes. Run them on one budget, and you get a pipeline that satisfies neither SLA fully.
Compliance logs inflate hot-tier costs without any detection benefit. When compliance-required sources sit in hot storage because the team runs a single ingestion pipeline, their per-GB cost is priced for queryability compliance owners never use. Authentication event logs from a decommissioned application, retained at hot-tier rates to satisfy a three-year requirement, or database audit logs from a low-risk system ingested at full fidelity to clear a checklist: these are common, not edge cases. The budget spent keeping them live crowds out the per-GB budget available for detection.
Detection-critical sources get cut to stay under budget. DNS query logs, process-creation events from non-priority endpoints, and identity logs from SaaS apps outside the primary SIEM scope get cut first, because they are the expensive ones to ingest at volume. They are also the exact MITRE ATT&CK data sources that detection rules for lateral movement, C2 beaconing, and credential abuse depend on. Even the SANS log management policy treats broad, reliable collection as the baseline, and these budget cuts quietly chip away at it. The detection pipeline fails in one specific way. Uncovered sources generate no alerts, so no one ever opens the investigation.
Compliance logs belong in cold or warm storage. Logs that exist to satisfy a retention requirement rather than to be queried in real time should stay in S3, an existing data lake, or another cold-tier store where retention is affordable and integrity is maintainable. The compliance pipeline's SLA is completeness over time, not sub-second latency, so moving those logs off hot-tier ingestion frees the detection budget for the sources that actually require it. No migration into a new system is needed; the logs stay exactly where they are.
Detection sources stay queryable in place. Strike48 provides federated search that queries logs across S3, Splunk, Elastic, and existing data lakes without moving data, through search-in-place connectors that reach sources wherever they already live. Detection-critical sources that were cut for per-GB cost can remain in place and become queryable without a new storage bill. Strike48's agentic layer reaches both the compliance archive and the detection-critical live sources through the same query interface, so teams no longer have to choose which pipeline gets hot-tier budget.
The structural error is a pricing assumption, not a data problem. Treating compliance retention and detection telemetry as a single pipeline on a single storage tier produces the conflict. Splitting them (cold or warm storage for retention, search-in-place for detection) resolves the constraint without adding cost. See what that architecture looks like in practice.
How long do you need to keep security logs?
It depends on the framework, not on a single universal rule. Most compliance regimes last between one and three years, though HIPAA reaches six years for certain records. Detection needs a far shorter live window, usually 30 to 90 days hot with older data pushed to cheaper storage, because analysts rarely correlate against year-old events in real time.
What is the difference between hot, warm, and cold log storage?
Hot storage keeps logs indexed and instantly queryable, which is what detection needs and what costs the most per gigabyte. Warm storage trades some speed for a lower price. Cold storage holds data cheaply for retention and later retrieval, but not for fast search, which is all a compliance archive actually requires. Matching each source to the right tier is most of the budget fight.
Is OMB M-21-31 still in effect?
No. OMB rescinded M-21-31 in May 2026 and replaced it with M-26-14, which moves federal agencies to a risk-based, prioritized logging model. The stated reason was that retaining large volumes of low-value data was not cost-effective, the same economics that push commercial teams to separate compliance retention from detection telemetry.
Do you still need a SIEM?
Yes, for aggregation, correlation, and detection engineering, but a SIEM does not have to be the storage tier for every log you keep. Compliance archives can live in cheaper object storage, and detection sources can be queried where they already sit through federated search, so the SIEM stops being the reason coverage gets capped at budget time.
Which logs matter most for threat detection?
The ones that MITRE ATT&CK maps to real attacker behavior, including DNS query data, process-creation events, authentication logs, and network flow records. These tend to be the first sources cut for cost, which is exactly why attacks that use those paths often generate no alerts at all.