I really should take lessons from chrish, because I suck ass trying to be the sys admin my sites. Got a little story for all you like me, wanna-be admins putting up your first apache server to show your friends and family pictures of your kids, pets, or bare body parts:
I used to get hacked once every three months, like clockwork. I’d be sitting at my desk late at night, enjoying an adult beverage and all the sudden my router would light up like a christmas tree. The load on my cpu would spike and the fans would fire up to cool the system down.
Being the sophisticated admin, I yanked the ethernet cord and tried doing an autopsy. Twice, I found nothing other than a couple new users created. Once I found a 1.4 gig flat text file with IP addresses and SMB user name/password combos some automated script kiddie tool that someone was running on my machine was generating.
Only once did a hack do any damage. I was testing some YDL stuff with Mr. Owen, and I lost use of the basic ‘ls’, ‘grep’, ‘ps’, etc commands. With his help, found out some dude/script exploited a hole I created and replaced all the commands with hacked equivalents. The funny part, the hacked commands were x86 binaries! (they don’t run well on a PPC machine)
Ok, so back to the every three month thing. Turns out during an argument with Kai (our CEO) over passwords, someone brought up Jack the Ripper complete with a table of password complexity vs. hack time. Yep, my passwords took ~3 months to brute force. Now my passwords are 20+ character phrases, convoluted to use a little bit of the entire ASCII set. I still don’t dig on random passwords, but am all for complex ones.
So now all my hacks are from exploits that I either created or didn’t keep some service updated. But I’m getting better. Since I think I’ve been hit again, found a cool little posting that is good quick overview of a forensic deconstruction. Enjoy:
http://www.fedora-linux.org/content/view/26/33/
2 comments
Comments feed for this article
Trackback link
http://blogs.ydl.net/billdar/2007/07/01/i-am-a-terrible-sys-admin/trackback/
July 3, 2007 at 10:20 am
cmurtagh
Brute force attacks are *very* rare, unless you happen to have something of high value, such as important data or access to high bandwidth (for use to attack more valuable targets), etc.
The reason why they’re so rare, is due to the fact that the chances of success are very low for decent passwords. This becomes more true if the attacker has to brute force through some public authentication mechanism (like ssh), as this adds connection overhead which adds to the cost of the attack - lowering the chances for success even further.
What is far more common however, is a dictionary attack. This is where the attacker attempts very common usernames with a list of common passwords. It’s hard to have a machine on the net these days without being ‘attacked’ via this scheme.
There are a couple of things you can do to protect yourself from this sort of attack, that are very simple and very effective:
1) In your /etc/ssh/sshd_config allow only users that really need shell to login to your box. Some people are very religious about not allowing root logins, I’m not one of them. Remote exploits are much rarer than local ones, so usually the first attack vector is to get shell, then use a local priv escalation exploit. So, if they don’t need shell, don’t give it to them.
2) Use decent passwords. I know this sounds like a no-brainer, but some people don’t know what that means. A decent password is an alpha-numeric password that is easy to remember and hard to guess. Make it at least 8 chars long, and include at least alphanum and some punctuation. Contrary to some beliefs, ‘decent’ doesn’t mean random, in fact random passwords are usually more of a security problem than good ones. People will write down hard to remember passwords, often in clear text files. This is bad, and becomes the security hole.
3) There is rarely a good reason to have the same root password for two boxes, avoid this. Don’t use a user’s password for root either. Users are more likely to use the same password for unsecure/untrusted things (like blogs, slashdot accounts, etc.). However, if this was their root password, one would hope that they would be less likely to use it elsewhere.
4) Firewall - iptables is your friend. Only allow ports that you absolutely need to be public facing. So, for example, web servers are 80/443 and everything else blocked. If you can and you’re really paranoid, on a very public machine allow port 22 from a specific IP or subnet only. This can be annoying as it means an extra hop if you’re out in the wild and need to get to the machine. A fun iptables trick is to limit the number of port 22 connections per minute from a given IP. So, this reduces the chances of an attack from extremely low to better odds of winning the lottery several times. Here are two lines you can add to your RH based iptables config that will do this:
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name DEFAULT --rsource -j DROP
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --rsource
This uses the conntrack extension of iptables to limit the number of SSH attempts to 4 times per minute.
5) Lastly, if you are hacked, consider *everything* on that box as untrusted. Take it physically off your network ASAP (unplug the network cable). You can try to recover things, but do this while the box is no longer on a network, and never put it back on one until the drives have been formatted or at least booted from another source (optical disk maybe). If you have tools like tripwire installed, you can check md5sums of binaries and such, but most of the time these sorts of things are more time consuming than just formatting the disks and re-installing. Again, this sounds obvious, but I’ve known a couple of people who’ve been hacked and thought they’ve removed the offending software installed only to find later on that a binary was trojaned and they were hacked again shortly afterward. Of course, once you’ve done this, change all of your passwords on all of your other boxes. Since the attacker had access to your /etc/shadow file, their odds of a brute force attack can be much greater if they have access to big iron, since they’re no longer limited by connection overhead, just raw CPU. Even with your crypted passwords, brute force attacks are expensive and not likely to succeed with good passwords.
July 3, 2007 at 12:14 pm
billdar
Thats what I’m talking about. Thanks Chris. I’m going to try out the iptables thing.
I’m going to add one more suggestion you gave me a while back in line with the firewall info. After I was complaining about log file size and other crazy stuff related to ssh and port 22, I moved the port up above 5000. It stopped my logs from getting flooded by automated scans and is fairly trivial.
Just change “Port 22″ to “Port XXX” in /etc/sshd_config