Linux and its Unix-like siblings are powerful and flexible operating systems that are a growing part of the modern enterprise network. Unfortunately, this power and flexibility has allowed for rapid development of web-based services that potentially provide network intruders with malicious capabilities.
Here are a few low-cost tips you can share with your team that could help stave off an incident and prepare your company to get answers and remediate quickly when an incident inevitably occurs:
- Keep as many logs as possible, for as long as possible
When an incident occurs, having good log data can go a long way to determining the initial point of compromise and being able to make determinations about what data may or may not have been accessed.
By default, most Linux-based systems regularly create new system logs daily, or when an individual log file reaches a certain size. Older log files are then compressed, and a small number are kept on disk. In the case of a particularly busy server, these older log files can roll off quickly.
Your team should ideally be maintaining logs on a centralized aggregator such as a syslog server like Splunk (http://www.splunk.com/) or Logstash (http://logstash.net/). But even if they aren’t in a position to do so, they can keep copies of logs on an inexpensive external hard drive somewhere else on the network until a log collector is in place.
- Increase the detail in logs and history files
While the default logging level may be sufficient for general system administration, increasing the verbosity of system and application logging to include informational messages can provide much greater insight about the state of the system and the actions performed during the incident.
Likewise, Bash shell history can be easily reconfigured in a single line to include timestamps, providing valuable information about when commands were executed on the system.
Both of the above examples only require a little storage space, and can mean the difference between guessing what an intruder did on your system, and knowing for sure.
Related Blog Posts
- Insist on tight controls over system access
Not every user in your organization needs access to all servers, but each user or process that does should have an individual account; credentials should never be shared. Insist that your team replaces username and password-based authentication with key-based authentication, and confirm that they don’t keep any copies of private keys on the system. If keys are going to be interactively used by regular users, add passwords to keys for an added layer of authentication. If you must use password-based authentication for some services, consider adding two-factor authentication mechanisms where possible. Ask your team to ensure that remote SSH logins are only allowed from known systems and networks by adding rules to the sshd_config file, or adding firewall rules to only allow authorized networks to SSH while denying all others by default.
- Ask your team is they know what ‘normal’ looks like
When something odd happens, a picture truly is worth a thousand words – especially to the people responsible for maintaining servers. But without knowing what normal looks like, everything is a potential problem that warrants further examination and this could lead to delaying resolution of the root cause.
Your team should be monitoring all services and interactions with your servers and make use of graphs so they can easily spot deviations from normal. Data graphing tools like Munin (http://munin-monitoring.org/) and RRDTool (https://oss.oetiker.ch/rrdtool/) can provide near real-time and historical data graphing of the state of your systems and processes. Having a visual record at the ready will help everyone involved quickly understand where and when an anomaly occurred and focus the response on the systems of interest.
- Run a lean system and lock down exposed processes
Production systems should run lean, and only software that is absolutely needed in production should be installed. Making tools available to install packages or compile code on a production system gives an intruder the power. If intruders don’t have these tools and can’t compile their own, they will have a difficult time gaining a foothold on the network, forcing them to look elsewhere or simply go away.
Ask your team if they’re running Internet-exposed processes inside of a chroot jailed environment. This adds additional protection should these processes become compromised because although an attacker may gain control of the process, he will be unable to elevate to higher levels of the system, or access any additional resources outside of the jail.
- Be ready and able to change the locks on the door
If credentials or keys are compromised, your team needs to be ready and able to securely re-key or issue new passwords quickly, with minimal impact to ongoing business operations. Although this will take a good deal of effort, your team should work proactively with developers and vendors to be ready to safely and securely reissue system credentials on demand.
- Be prepared for the worst and test your process
Practicing each part of your disaster recovery plan will help to ensure the plan is sound and your team will operate calmly and efficiently to isolate the affected systems, preserve the data, and deploy replacement systems as quickly as possible. Have your team practice the disaster recovery plan periodically, and remediate any processes that don’t work as expected.