Successful investigation and remediation of a data breach can often depend on log data. Many companies are hard pressed to retrieve log data that is more than a year old, but in a recent Ponemon Institute Study, more than half of the companies surveyed discovered a breach over a year after the attack was launched. That’s a problem for two reasons. First, our cyber resilience team strongly believes that all companies should proactively hunt for signs of a breach so attackers don’t get a significant head start before investigation begins (see “A Cyber Call to Arms”). Until hunting becomes common practice, however, storing log data for a sufficient amount of time is key to effective incident response.
The fact that logs are a critical component of an investigation shouldn’t come as a surprise. After all, logs give insight into actions that have already occurred on a system. Most companies know what logs to keep, but run into problems when deciding how to store them. In my experience, it’s not uncommon for companies to use their Security Information and Event Management (SIEM) platforms as the central repository for log data. On the surface, this makes sense since you need to feed your SIEM most, if not all, of your logs.
But is this really the right place to store your logs? Let’s review.
Security Information and Event Management (SIEM): A platform designed to aggregate events from various sources (Firewalls, IPS\IDS, Anti-virus, OS Logging, etc.) and provide security-focused analytics.
SIEMs help provide context to security events and can uncover issues that might otherwise go unnoticed if security log review is done in silos within different tools. SIEMs do this by parsing out relevant data from each event and storing it in a common database. These parsed fields can then be correlated with fields from different sources that are similarly structured (IP Address, Ports, Applications, etc.) using security-focused logic. After an event has been parsed, the raw data is no longer needed by the SIEM.
SIEMs perform correlation across disparate tools, which reduces blind spots, especially for large companies that have a lot of different sites and tools. But if too much data is stuffed in into a SIEM’s database, performance will be slow and clunky, and limit the effectiveness of the SIEM (and your security team).
Log Management: An application that indexes fields of interest and relates them back to a raw log.
Log management tools act more like search engines than analytics tools. Many log management tools offer reporting and analytics, but they don’t have the security-specific context that a SIEM provides. Log management solutions excel at helping organizations store and access historical log data—this is essential for effective incident response and investigation.
Both tools are valuable – unfortunately, they both also come with hefty price tags. The key is to use each for the unique purpose for which it was intended. For organizations with limited budgets, my advice is the following: A raw log is always valuable. When it comes to incident response, you can implement analytics after the fact, but you can’t get back a raw log entry that hasn’t been properly stored.
To determine if you’re using your SIEM effectively, here are some questions to think about:
- What is the purpose of my SIEM? If the answer is, “log management,” you might want to consider a different architecture.
- What is the role of my SIEM in an incident? If your answer is, ‘log management’, and you’re hesitant to implement a separate log management tool, make sure all of the relevant data fields in all of your logs are being parsed, otherwise your searches and analytics won’t be of much value.
- How long does my SIEM retain data? You don’t want to lose data you’ll need in the event of an incident. The only way to prevent this is to know how your SIEM stores data and for how long.
- What export options exist for my SIEM’s log data? SIEMs often don’t have RAW logs, and if they do, these logs are often incomplete or slow to access. Parsed fields may be exportable, but they must be set up correctly to allow the export to work.
Most security teams know what must be logged and how long to retain these logs. Problems arise when enterprises centralize security logs and mistake this service for storage. To be ready to respond to inevitable attacks, don’t put your data at risk in this way. Implement a log management system for long term storage and access.