Incident Response Preparedness Series: Preparing Linux Systems Before the Breach

Stroz Friedberg is a specialized risk management firm built to help clients solve the complex challenges prevalent in today’s digital, connected, and regulated business world

Linux and its Unix-like siblings are powerful and flexible operating systems that are a growing part of the modern enterprise network. Unfortunately, this power and flexibility has allowed for rapid development of web-based services that potentially provide network intruders with malicious capabilities.

Here are a few low-cost tips you can share with your team that could help stave off an incident and prepare your company to get answers and remediate quickly when an incident inevitably occurs:

  • Keep as many logs as possible, for as long as possible

When an incident occurs, having good log data can go a long way to determining the initial point of compromise and being able to make determinations about what data may or may not have been accessed.

By default, most Linux-based systems regularly create new system logs daily, or when an individual log file reaches a certain size. Older log files are then compressed, and a small number are kept on disk. In the case of a particularly busy server, these older log files can roll off quickly.

Your team should ideally be maintaining logs on a centralized aggregator such as a syslog server like Splunk ( or Logstash ( But even if they aren’t in a position to do so, they can keep copies of logs on an inexpensive external hard drive somewhere else on the network until a log collector is in place.

  • Increase the detail in logs and history files

While the default logging level may be sufficient for general system administration, increasing the verbosity of system and application logging to include informational messages can provide much greater insight about the state of the system and the actions performed during the incident.

Likewise, Bash shell history can be easily reconfigured in a single line to include timestamps, providing valuable information about when commands were executed on the system.

The example screenshot above highlights the advantage of history time stamping. On the left side we see someone downloaded and installed a program from the Internet, but we can’t tell when, and it isn’t clear where the intruder’s commands begin and end. On the right, with time stamping enabled, we can see the intruder started a session late on a Saturday night. The timing of the attack also seems to indicate that this was performed by a user sitting at a keyboard rather than an automated script.

Both of the above examples only require a little storage space, and can mean the difference between guessing what an intruder did on your system, and knowing for sure.

  • Insist on tight controls over system access

Not every user in your organization needs access to all servers, but each user or process that does should have an individual account; credentials should never be shared. Insist that your team replaces username and password-based authentication with key-based authentication, and confirm that they don’t keep any copies of private keys on the system. If keys are going to be interactively used by regular users, add passwords to keys for an added layer of authentication. If you must use password-based authentication for some services, consider adding two-factor authentication mechanisms where possible. Ask your team to ensure that remote SSH logins are only allowed from known systems and networks by adding rules to the sshd_config file, or adding firewall rules to only allow authorized networks to SSH while denying all others by default.

  • Ask your team is they know what ‘normal’ looks like

When something odd happens, a picture truly is worth a thousand words – especially to the people responsible for maintaining servers. But without knowing what normal looks like, everything is a potential problem that warrants further examination and this could lead to delaying resolution of the root cause.

Your team should be monitoring all services and interactions with your servers and make use of graphs so they can easily spot deviations from normal. Data graphing tools like Munin ( and RRDTool ( can provide near real-time and historical data graphing of the state of your systems and processes. Having a visual record at the ready will help everyone involved quickly understand where and when an anomaly occurred and focus the response on the systems of interest.

outbound traffic
In the example above showing the outbound traffic from a server, there’s a sudden and sustained increase between about 2am and 11am. Is this normal or is it worth investigating? Experienced eyes should know and be able to focus the investigation to the time of concern with a single picture.
  • Run a lean system and lock down exposed processes

Production systems should run lean, and only software that is absolutely needed in production should be installed. Making tools available to install packages or compile code on a production system gives an intruder the power. If intruders don’t have these tools and can’t compile their own, they will have a difficult time gaining a foothold on the network, forcing them to look elsewhere or simply go away.

Ask your team if they’re running Internet-exposed processes inside of a chroot jailed environment. This adds additional protection should these processes become compromised because although an attacker may gain control of the process, he will be unable to elevate to higher levels of the system, or access any additional resources outside of the jail.

  • Be ready and able to change the locks on the door

If credentials or keys are compromised, your team needs to be ready and able to securely re-key or issue new passwords quickly, with minimal impact to ongoing business operations. Although this will take a good deal of effort, your team should work proactively with developers and vendors to be ready to safely and securely reissue system credentials on demand.

  • Be prepared for the worst and test your process

Practicing each part of your disaster recovery plan will help to ensure the plan is sound and your team will operate calmly and efficiently to isolate the affected systems, preserve the data, and deploy replacement systems as quickly as possible. Have your team practice the disaster recovery plan periodically, and remediate any processes that don’t work as expected.


Our lawyers don’t want to miss out on the fun and would like you to know that all of the posts are the opinions of the individual authors and don’t necessarily reflect the opinions or positions of Stroz Friedberg. The ideas and strategies discussed herein may not be appropriate for any one reader’s situation and are not meant to be construed as advice.

Risk Areas: Cyber

I am: Super-technical

Tags: incident response, incident preparedness, Linux



Commentary, new discoveries, and innovative ideas right to your inbox.

Stroz Friedberg

Sorry! You are using an older browser which is not supported by this website.

Please download one of these free browsers to enjoy all our website has to offer:
Firefox, Chrome or Internet Explorer.