Rookit infections: AARRGH!

Derek Martin invalid at pizzashack.org
Mon May 9 13:13:01 EDT 2005


On Mon, May 09, 2005 at 08:50:33AM -0400, Fred wrote:
> I'm about ready to pull my hair out.
> This is the 2nd time I've had to deal with a rootkit infection, eating
> up my precious time and resources away from being productive.

From reading the whole thread, it's become clear that you have a
number of users on your system besides yourself, and that the
machine(s) in question are directly routable on the Internet (i.e.
they aren't behind a firewall doing NAT).  This is always
problematical.  It gives attackers a direct means of attacking your
machines, and having users provides them with many additional attack
vectors -- unless all of your users are security experts, and take
their paranoia seriously all the time, they're going to do things that
can get you rooted unless you FORCE them not to.  It's really that
simple.

> What I'd like to know is how my systems are being cracked. What is the
> port of entry(!), how are my systems broken into. What's the latest news
> on this.
>
> I am suspicious that they are somehow breaking in through ssh -- my logs
> show lots of suspicious sshd authentication failures. But my root
> password is pretty sound, a near random mixture of numbers and alpha
> characters. They must be breaking in through another account with a
> weaker password. But I'm not sure of this.

This is certainly possible.  Recent posts on bugtraq suggest that
these attacks are being surprisingly successful on account of people
using the same overly simplistic passwords that they've been using
since the beginning of time (and I'm talking the classical age here,
not Unix time).

If it is how they got in, you should see evidence of it in your logs.
You'll probably see a series of failures all from the same IP, and
then a successful login from that same IP.  Buffer overflow attacks
also usually show up in logs with really strange things in the log
messages, like really long, nonsensical user names followed by
garbage, or some such thing like that.  If you're paying attention,
and you've set logging up reasonably (the defaults are usually
sufficient or RH-based systems), breakins almost always leave some
kind of trail for you to follow.

> I have taken countermeasures. Firstly, I have changed the ssh port
> number. Not the most secure approach, granted, but at least their
> automated attacks will be foiled somewhat, since they'll have to do more
> work at hitting all of my ports -- and will probably not bother and move
> on to the next server.

These days, most of the attacks on your box are automated, so this
probably will suffice... BUT, given that you have users and you're
directly connected to the Net, you really ought not to stop there.

One suggestion I'm SHOCKED no one has yet suggested will make it a lot
harder for attackers to log into your machine: DON'T USE PASSWORD
AUTHENTICATION AT ALL!  There's no reason to, and using asymetric
key-based authentication is probably a thousand times more secure.
Disable password auth entirely and force your users to use DSA keys
and SSH2 to log in.

> Secondly, on the infected machines, I use forced RPM installs to
> overwrite everything, then follow up with a run from chkrootkit. This
> seems to work, eliminating the need for me to burn down the box and
> restore everything cleanly. Again, not a perfect solution, but seems to
> work for now.

This simply isn't sufficient, ever.  If your machine has been
compromised, everything from system libraries to the kernel itself
could have been modified to prevent detection tools from working
properly.  If you've been compromised, the only safe way to restore
your system is to wipe it clean and start from scratch.  Period.  This
is said so often on this list that it's hard to imagine that people
still believe that half-hearted efforts are sufficient...

> Thirdly, I have set up chkrootkit to be run daily as a cron job, with
> the results emailed to me.

Which does nothing if the programs or the libraries they depend on (if
not compiled statically) or the kernel hs been modified to prevent
detection.  Note that changes made to the system might not be effected
until after a reboot...  So chkrootkit might have found things
originally, but may stop after rebooting.


On Mon, May 09, 2005 at 09:06:31AM -0400, Brian wrote:
> 2, ONLY allow ssh connections from trusted IPs, not the whole world.

This is a great idea, but usually impractical if users move around a
lot.  Mobile users will never know what IP they're going to be using
in advance.

On Mon, May 09, 2005 at 09:19:30AM -0400, Neil Joseph Schelly wrote:
> On Monday 09 May 2005 09:06 am, Brian wrote:
> > 1, NEVER allow root access via SSH.  You should have to login as a user,
> > and then su - to root, or better yet setup a sudoers file.
> 
> This is one of those best practices I've never really felt had merit.  It 
> seems to me that when people break in through SSH, they are doing it through 
> exploits in the SSH or OpenSSL codebases, not through password guessing.  

Kevin is right on the money here.  No one should ever log in as root,
EVER, unless the system is broken and in need of repair.  If more than
one person has access to the root password, you never know who's
logging in or what they're doing.  Forcing users to log in first
allows them to be identified.  Don't forget that an estimated 70% of
compromises are effected by people on the inside...


On Mon, May 09, 2005 at 09:23:31AM -0400, Tom Buskey wrote:
> If you have a user login to the system, you should *never* need to
> login as root.  If you have sudo setup (with ALL), you shouldn't need
> su either.  root should be able to have a disabled password at that
> point, except for single user login (fsck at boot up...).  I haven't
> tested that of course & there are issues.

Of course, having just written what I did, I'm a big fan of sudo.  But
there are times when we must type lots of commands as root where
preceeding each of them with sudo becomes impractical.  For those
times, I often do "sudo sh".  I think this is perfectly reasonable for
senior administration staff (or the only admin of a machine).

Actually I have an alias that opens an xterm with a different
background color than my usual dark blue, which runs sudo sh for times
when I need to do lots of stuff.  The green background tells me I need
to be careful of what I'm doing...

On Mon, May 09, 2005 at 09:47:36AM -0400, Fred wrote:
> On Mon, 2005-05-09 at 09:23 -0400, Tom Buskey wrote:
> > 
> > 3.  Do not allow SSH v1 protocol.  Only allow v2.  v1 has known,
> > unfixable, vulnerabilities.
> 
> Yes, I have been turning off V1 on the boxes under attack. Though from
> now on, V1 will *always* be disabled. Why it is still left enabled by
> the various distros given the vulnerabilities is beyond me.

It's more universally supported (though this almost isn't true
anymore), and though there are vulnerabilities, they're pretty complex
to attack, and AFAIK there is no known working exploit.  Many people
don't take them very seriously.  I'm not suggesting that's good
policy... but at least for the moment, it seems to be an acceptable
risk.
On Mon, May 09, 2005 at 09:38:05AM -0400, Fred wrote:
> Still, what I could probably do is implement a scheme where visiting a
> particular webpage (and giving proper authentication) would enable that
> IP address for ssh. Come to think of it, that's not such a bad idea
> after all! That will also allow my users to ssh into from their
> locations should they need to.

Interesting idea, but probably not worth the effort.  SSH keys are a
much better way to go, and require virtually NO effort on your part
(except possibly educating your users, which could be quite an effort
indeed ;-).

> I like that idea too, but since these are dedicated servers hosted
> elsewhere, I won't be able to do CDROM. However, I could put a cdrom ISO
> image (or some other filesystem) on the system, chattr +i it, and simply
> mount that by hand when I need to. 

This simply isn't effective.  There's nothing stopping an attacker
from running chattr -i.  In fact, it might even give them a clue to
the importance of that file...  People usually only run chattr -i on
stuff they don't want modified, usually for security reasons or for
some other important reasons.  Files flagged as immutable should
attract the attention of intellgent hackers.

> Better yet, some sort of encrypted filesystem. Hmmm...

A much better idea, but a read-only media is still better.

On Mon, May 09, 2005 at 09:49:48AM -0400, Steven W. Orr wrote:
> On Monday, May 9th 2005 at 09:38 -0400, quoth Fred:
> 
> =>Well, this generated some good ideas, but I could use more. Thanks.
> 
> One more for aftermath cleanup if you're running an rpm-based setup:
> 
> rpm -Va will check every file in the installation for integrity.

And will return completely useless results if the attacker has
modified the rpm command or related libraries.  There do exist
rootkits which do this, IIRC.  As has been pointed out on this list so
many times in the past, relying on the programs on a compromised
system to determine ANYTHING about that system is folly.

On Mon, May 09, 2005 at 10:55:02AM -0400, Bill McGonigle wrote:
> The reason to disable root account ssh login is just an odds game - 
> every unix system is guaranteed to have a root user so it's a good one 
> for password guessers to start with.  Any other account can be renamed. 

This is actually false.  The only thing special about the root account
is the UID of zero.  You can feel free to change the name of the root
acount to bmcgonigle, or bionicle, or iluvphishing, or anything else
you like.  People rarely do change it, but there's no reason not to.
Except maybe that some braindead coders like to hard-code the name of
the root account (and sometimes even the root password) into their
applications.  Sigh.


-- 
Derek D. Martin    http://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail.  Sorry for the inconvenience.  Thank the spammers.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.gnhlug.org/mailman/private/gnhlug-discuss/attachments/20050509/c37622ba/attachment.bin


More information about the gnhlug-discuss mailing list