Written by Kurt Seifried
Network monitoring / attack detection
One major oversight made by a lot of people when securing their machines is that they forget to create a baseline of the system, that is a profile of the system, its usage of resources, and so on in normal operation. For example something as simple as a "netstat -a -n > netstat-output" can give you a reference to latter check against and see if any ports are open that should not be. Memory usage and disk usage are also good things to keep an eye on. A sudden surge in memory usage could result in the system being starved of resources. Likewise for disk usage. It might be a user accident, a malicious user, or a worm program that has compromised your system and is now scanning other systems. Various tools exist to measure memory and disk usage: vmstat, free, df, du, all of which are covered by their respective man pages.
At the very minimum make a full system backup, and regularly backup config files and log files, this can also help you pinpoint when an intrusion occurred (user account "rewt" was added before the April 4th backup, but isn't in the March 20th backup). Once a system is compromised typically a "rootkit" is installed, these consist of trojaned binaries, and are near impossible to remove safely, you are better of formatting the disk and starting from scratch. There is of course a notable exception to this rule, if you were diligent and used file/directory integrity tools such as L5 you will be able to pinpoint the affected files easily and deal with them.
There are also a variety of tools that do not quite fit under the headings here, but are aimed at attack detection. One is the Linux Intrusion Detection System (LIDS) project, more information is listed here.
So you've secured your machines, and done all the things that needed to be done. So how do you make sure it's actually doing what it is supposed to do, or prove to someone that it is as secure as you say it is? Well you conduct an audit. This can be as simple as reviewing the installed software, configuration files and other settings, or as complex as putting together or hiring a tiger team (or ethical hackers, or whatever buzzword(s) you prefer) to actively try and penetrate your security. If they can't then you did your job well (or they suck), and if they do get in, you know what needs to be fixed (this is also a good method to show the CIO that security is not a one shot affair, it is a constant battle). One thing almost all attackers do is modify system files, once you detect a break in, how do you know which files are ok and which are not? Short of a complete reinstall the only way to be sure (and even then it's not always 100%) is to use software to create signatures of files that cannot be forged so you can compare them later on.
Tripwire is no longer a open source tool. I have absolutely NO problems with commercial software. However, when you expect me to rely on a program to provide security, when I (nor anyone else really) can not view the source (it is available under some special license agreement, probably an NDA) I must decline. Tripwire costs approximately $70 for Linux, and is only available as an RPM package aimed at Red Hat Linux (tripwire is $500 for other operating systems). I feel this is rather on the high side for a piece of software that can easily be replaced with alternatives such as L5 or Gog&Magog. Tripwire is available at: http://www.tripwiresecurity.com/.
AIDE is a tripwire replacement that attempts to be better then tripwire. It is GPL licensed which makes it somewhat more desirable then tripwire from a trust point of view. It supports several hashing algorithms, and you can download it from: http://www.cs.tut.fi/~rammer/aide.html.
There is an alternative to tripwire however, L5, available at: ftp://avian.org/src/hacks/, it is completely free and very effective. I would definitely recommend this tool.
Gog&Magog creates a list of system file properties, owner, permissions, an MD5 signature of the file and so (similar to tripwire). You can then have it automatically compare this and ensure any changed files/etc come to your attention quickly. As well it makes recovering from a break in simpler as you’ll know which files were compromised. You can download Gog&Magog from: http://www.multimania.com/cparisel/gog/.
Sentinel is a program that scans your harddrive and creates checksums of files you request it to. It uses a non patented algorithm (RIPEMD-160bit MAC ), and has an optional graphical front end (nice). You can get it at: http://zurk.netpedia.net/zfile.html.
SuSEauditdisk is a bootable disk with integrity checking tools and the checksums providing a very secure method to check for damage. It ships standard with SuSE and can easily be ported to other Linux distributions, and is GPL licensed. You can get SuSEauditdisk from: http://www.suse.de/~marc/.
ViperDB checks setuid/setgid programs and folders and can notify you (via syslog) of any changes or reset their permissions and ownership to what they should be. ViperDB creates a series of databases (flat text files actually) in the directory root, i.e.: /etc/.ViperDB might contain:
/etc/login.defs,1180,-,root,rw-,root,r--,r--,Apr,15,18:03 /etc/minicom.users,1048,-,root,rw-,root,r--,r--,Mar,21,19:11 /etc/CORBA,1024,d,root,rwx,root,r-x,r-x,Jun,14,16:51 /etc/X11,1024,d,root,rwx,root,r-x,r-x,Jun,14,23:05 /etc/cron.d,1024,d,root,rwx,root,r-x,r-x,Apr,14,17:09
Unfortunately ViperDB doesn’t seem to handle sub directories, so you will have to add them to the viperdb.ini file with something like:
find /etc/ -type d >> /usr/local/etc/viperdb.ini
viperdb.pl has 3 options, -init (creates a set of databases),
-check (checks files against databases, sends any messages to
syslog, and then recreates the databases) and –checkstrict
(checks files against databases, resets permissions if necessary,
sends any messages to syslog, and then recreates the databases).
What this means is if you use –check, you will get a warning
that say /etc/passwd is now world writeable, and since it
recreates the databases the next time you run viperdb you will
NOT get a warning. I would advise running viperdb is
–checkstrict mode only, and make sure you run viperdb with
the –init option after manipulating any file / folder
permissions in protected directories. ViperDB is available for
download from:
http://www.resentment.org/projects/viperdb/.
Sxid checks setuid and setgid for changes, generates MD5 signatures of the files and generally allows you to track any changes made. You can get it at: ftp://marcus.seva.net/pub/sxid/.
nannie is a relatively simply tool that relies on stat to build a list of what files should be like (size, timestamps, etc.). It creates a list containing the filename, inode, link information and so on, it does make a useful, albeit simple burglar alarm. You can get it from: ftp://tools.tradeservices.com/pub/nannie/.
confcollect is a simple script that collects system information such as routing tables, rpm’s installed and the like. You can download it from: http://www.skagelund.com/confcollect/.
Pikt is an extremely interesting tool, it is actually more of a scripting language aimed at system administration then a simple program. Pikt allows you to do things such as killing off idle user processes, enforcing mail quotas, monitor the system for suspicious usage patterns (off hours, etc), and much more. About the only problem with Pikt will be a steep learning tools, as it uses it’s own scripting language, but ultimately I think mastering this language will pay off if you have many systems to administer (especially since Pikt runs on Solaris, Linux and FreeBSD currently). Pikt is available at: http://pikt.uchicago.edu/pikt/.
Something people forget about, but you can compare the current files to old backups, many backup formats (Tape, floppy, CDR, etc.) can be made read only, so a backup of a newly installed system provides a good benchmark to compare things to. The utility “diff” and “cmp” can be used to compare files against each other. See the backup section for a full listing of free and commercial software.
If the last section has you worried you should be. There are however many defenses, active and passive against those types of attacks. The best ways to combat network scans are keep software up to date, only run what is needed, and heavily restrict the rest through the use of firewalls and other mechanisms.
Luckily in Linux these tools are free and easily available, again I will only cover opensource tools, since the idea of a proprietary firewall/etc is rather worrying. The first line of defense should be a robust firewall, followed by packet filters on all Internet accessible machines, liberal use of TCP-WRAPPERS, logging and more importantly automated software to examine the logs for you (it is unfeasible for an administrator to read log files nowadays).
The Deception ToolKit is a set of programs that emulate well known services in order to provide a false set of readings to attackers. The hope is to confuse and slow down attackers by leading them to false conclusions, you can download DTK from: http://all.net/dtk/.
The third component to the Abacus suite, it detects and logs port scans, including stealthy scans (basically anything nmap can do it should be able to detect). Psionic PortSentry can be configured to block the offending machine (in my opinion a bad idea as it could be used for a denial of service attack on legitimate hosts), making completion of a port scan difficult. As this tool is in beta I would recommend against using it, however with some age it should mature into a solid and useful tool. Psionic PortSentry is available at: http://www.psionic.com/abacus/portsentry/.
While this software is not yet ready for mass consumption I thought I would mention it anyways as it is part of a larger project (the Abacus project, http://www.psionic.com/abacus/). Basically Psionic HostSentry builds a profile of user accesses and then compares that to current activity in order to flag any suspicious activity. Psionic HostSentry is available at: http://www.psionic.com/abacus/hostsentry/.
Most firewalls support logging of data, and ipfwadm/ipchains are no exception, using the -l switch you get a syslog entry for each packet, using automated filters (Perl is good for this) you can detect trends/hostile attempts and so on. Since most firewalls (UNIX based, and Cisco in any case) log via the syslog facility, you can easily centralize all your firewall packet logging on a single host (with a lot of harddrive space hopefully).
Wietse's TCP-WRAPPERS allow you to restrict connections to various services based on IP address and so forth, but even more importantly it allows you to configure a response, you can have it email you, finger the offending machine, and so on (use with caution however). TCP_WRAPPERS comes standard with most distributions and is available at: ftp://ftp.porcupine.org/pub/security/.
While mostly obsoleted by TCP-WRAPPERS and firewall logging, klaxon can still be useful for detecting port scans/etc if you don't want to totally lock down the machine. Klaxon is available at: ftp://ftp.eng.auburn.edu/pub/doug/.
NFR (Network Flight Recorder) is much more then a packet sniffer, it actually logs data and in real time detects attacks, scans and so on. This is a very powerful tool and requires a significant investment of time, energy and machine-power to run, but it is at the top of the food chain for detection. NFR is available at: http://www.nfr.com/.
FAQ: Network Intrusion Detection Systems, an excellent FAQ that covers all the major (and many minor) issues with IDS systems. Available at: http://www.robertgraham.com/pubs/network-intrusion-detection.html.
So you've done your homework, you installed tripwire, DTK, and so on. Now what do you do when your pager starts going off at 3am and tells you that someone just made changes on the primary NIS server? Dealing with an attack depends on several factors, is the attack in progress? Did you discover your company plan being sent out by the mail server to a hotmail address? Did you get called in to find a cluster of dead servers? What are your priorities? Restoring service? Ensuring confidential data is safe? Prosecuting the attacker(s)? Several things to keep in mind:
Also before you deal with an attack, you should consult your company policy. If you don't have one consult your manager, the legal department, etc. It's also a good idea to have a game plan to deal with attacks (i.e., the mail server is first priority, checking fileservers is number two, who do you notify, etc) this will prevent a lot of problems when it happens (be prepared). The O'Reilly book “Practical Unix and Internet Security” covers this topic in great detail so I’m not going to rehash it. Go buy the book.
An excellent whitepaper on this is also available, see Appendix D, “How to Handle and Identify Network Probes”.
Packet sniffing is the practice of capturing network data not destined for your machine, typically for the purpose of viewing confidential/sensitive traffic such as telnet sessions or people reading their email. Unfortunately there is no real reliable way to detect a packet sniffer since it is mostly a passive activity, however by utilizing network switches and fiber optic backbones (which are very difficult to tap) you can minimize the threat. There is also a tool called AntiSniff, that probes network devices and sees if their response indicates an interface in promiscuous mode. These tools are also invaluable if your network is under attack and you want to see what is going on. There is an excellent FAQ on sniffing at: http://www.robertgraham.com/pubs/sniffing-faq.html.
The granddaddy of packet sniffers for Linux, this tool has existed as long as I can remember, and is of primary use for debugging network problems. It is not very configurable and lacks advanced features of newer packet sniffers, but it can be useful. Most distributions ships with tcpdump.
My favorite packet sniffer, sniffit is very robust, has nice filtering capabilities, will convert data payloads into ASCII text for easy reading (like telnet sessions), and even has a graphical mode (nice for monitoring overall activity/connections). Sniffit is available at: http://sniffit.rug.ac.be/~coder/sniffit/sniffit.html.
A nice looking network protocol analyzer (a.k.a., a souped up sniffer) with an interface very similar to NT’s network monitor. It allows easy viewing of data payloads for most network protocols (tftp, http, Netbios, etc). It is based on GTK, thus meaning you will probably have to be running gnome to use it. I haven't tested it yet (but intend to). It is available at: http://ethereal.zing.org/.
Snort is a nice packet sniffing tool that can be used to detect various attacks as well. It can watch for activity such as Queso TCP-IP fingerprinting scans, Nmap scans, and the like. Snort is available from: http://www.clark.net/~roesch/security.html.
SPY is an advanced multi protocol sniffer that runs on various platforms. It is not a free program however there is a single user license available for non commercial use with a maximum of 5 hosts. Commercial it costs around $6000 US dollars, but from a quick look at it’s capabilities I would say it is worth it if you need an industrial grade sniffer. You can get it from: http://pweb.uunet.de/trillian.of/Spy/.
packetspy is another libpcap based sniffer. You can get it from: http://www.bhconsult.com/packetspy/.
There are a variety of packet sniffers for Linux, based on the libpcap library among others, here is a short list:
http://www.mtco.com/~whoop/ksniff/ksniff.html
- KSniff
http://ksniffer.veracity.nu/
- Ksniffer
http://mojo.calyx.net/~btx/karpski.html
- karpski
http://www.ozemail.com.au/~peterhawkins/gnusniff.html
- Gnusniff
http://www.xnet.com/~cathmike/mike/Software/
- ipgrab
In theory most operating systems leave tell tale signs when packet sniffing (that is to say their network interfaces respond in certain, non standard ways to network traffic). If the attacker is not to savvy, or is using a compromised machine then chances are you can detect them. On the other hand if they are using a specially built cable, or induction ring there is no chance of detecting them unless you trace every physical piece of network cable and check what is plugged into it.
As mentioned before AntiSniff i a tool that probes network devices to try and see if they are running in promiscuous mode, as opposed to normal modes of operation. It is supposedly effective, and will work against most sniffers. You can get it from: http://www.l0pht.com/antisniff/.
| Written by Kurt Seifried |