debian.org compromise - Developer Cleanup Info

With the compromise of some core Debian systems Debian developers need to assume that their debian.org passwords that they use to login to debian.org have been compromised. If ssh agent forwarding was enabled an attacker could have used your ssh private key to log into other machines. Access to all debian.org machines has been disabled to prevent the problem from getting worse. Once things have been cleaned it will be possible to unlock your account and request a new password through the email robot on db.debian.org. In the meantime you need to ensure that this problem can't spread to any non-debian.org machines that you use.

What you need to do

For every machine that,

you may have logged into from a debian.org host
that uses the same passwd or passphrase for access as you used for debian.org
potentially any machine that you connect to from one of the above. As mentioned in the detailed analysis, people's $HOME/.ssh/known_hosts files has been accesses so there's a good chance the attacker knows what hosts to try.

you need to,

run chkrootkit and check for problems
optionally run debsums and cruft
change your user and root passwds
change your ssh key and update any systems with the public half of that key
optionally run the skdetect tool

You need to do this in a systematic way to ensure a clean sweep and no chance of leaving a gap that would allow an attacker to potentially get back in.

start with the system you use the most.
- run chkrootkit, debsums, and anything else you want to confirm that the system is ok
- change your passwd and the root passwd
- generate a new ssh key
for each of the systems you login to use existing credentials to connect
- run chkrootkit, debsums, and anything else you want to confirm that the system is ok
- change your passwd and the root passwd
- install the public half of the key from step #1 and remove the old one. Check and make sure it works.
- if you connect from this system to any others:
  - generate a new ssh key
  - repeat step #2 for the additional hosts
consider doing something more to protect your accounts and systems, see below

Running chkrootkit

Install chkrootkit from unstable. For woody systems you can use the apt sources.list entry below:
```
deb http://lackof.org/taggart/debian woody/chkrootkit main
deb-src http://lackof.org/taggart/debian woody/chkrootkit main
```
This does not have packages for all architectures, so you might need to recompile it for your system. If you already have it installed you might want to do a 'apt-get --reinstall install chkrootkit' to get a new one. If you're really paranoid, you can check the package signature.
Run it `chkrootkit -q`, remove the -q if you want to see what it's doing.

It can get some false positives,
- An ethernet interface is reported as being PROMISC. It will report this if the interface has been PROMISC since boot, so if you run things like tcpdump you may see this. Run `ip link` to see if it's still in PROMISC mode and consider what you may be running to cause a false positive.
- On certain kernels you may get,
```
Checking `lkm'... You have     4 process hidden for ps command
Warning: Possible LKM Trojan installed
```
  You can get more info by running it in expert verbose mode, here is an example,
```
$ chkrootkit -x lkm
ROOTDIR is `/'
###
### Output of: ./chkproc -v -v
###
PID     4: not in ps output
CWD     4: /
EXE     4: /
PID     5: not in ps output
CWD     5: /
EXE     5: /
PID     6: not in ps output
CWD     6: /
EXE     6: /
PID     7: not in ps output
CWD     7: /
EXE     7: /
You have     4 process hidden for ps command
```
  Check the /proc/<pid>/status file for the name of the process. Example:
```
$ cat /proc/[4-7]/status |grep Name
Name:   ksoftirqd_CPU0
Name:   kswapd
Name:   bdflush
Name:   kupdated
```
- The lkm check is known to produce false positives for NPTL kernels (2.6 kernels or 2.4 with NPTL patches). Common multithreaded programs which will show this behaviour are slapd, mozilla and apache2 if you use one of its threading MPMs.
- The lkm check is known to fail on really slow machines. As processess start up and exit, it thinks they are hidden.

Running debsums

Install debsums. If you already have it installed you might want to run 'apt-get --reinstall install debsums' to get a new one. If you're really paranoid, you can check the package signature.
Run it `debsums -s`, remove the -s if you want to see what it's doing.
A lot of packages don't have md5sums and we have no way of knowing if the dpkg md5sums have been tampered with. This test is only good for finding problems, you can't fully trust a positive result but it might make you feel a little better. Don't let it give you a false sense of security though.

Run skdetect

There also is a special suckit dectection tool that is more accurate than the checks done by chkrootkit.

Things you can do to help limit damage

Here are some things you can do to help prevent exploits and reduce the damage that can happen if the system does get exploited.

Always use secure passwords. Anything based on words with numbers added or simple letter->number changes is easily guessable, even if you use a non-English word.
Change your account behaviours(easy). Consider doing the following,
- not ever ssh'ing from one remote host to another
- using unique keys and passphrases for each host
- disabling ssh passwd access and using only keys
- restricting the list of hosts that are allowed to ssh to your systems
In the event of a compromised system, these things will make it harder for the attacker to sniff your passwds/passphrases and use them to attack other systems.
Remove CAP_SYS_MODULE and CAP_SYS_RAWIO from the global set of allowed capabilities(easy for servers). The first prevents userland from changing cap-bound, the second prevents /dev/kmem from being useable. These prevent an attacker that has gained root from installing things to hide processes, files, and do things in the kernel like sniff ptys etc.

You do this by running,
```
echo 0xFFFCFFFF > /proc/sys/kernel/cap-bound
```
This can be done at boot time by adding the following line to /etc/sysctl.conf,
```
kernel/cap-bound = 0xFFFCFFFF
```
Notes:
- lsof and xfree86 will no longer work.
- Once set it cannot be unset without a reboot.
- Loading of kernel modules will no longer be possible, so make sure to compile everything into your kernel or do this after your boot scripts have loaded all necessary modules.
- lilo will also no longer use, on i386 systems it is advised to use grub instead.
- This wouldn't have prevented the debian.org machines from being compromised, but it would have prevented the rootkits from being installed.
Run an exec-shield kernel(easy). Here's a summary,

The exec-shield feature provides protection against stack, buffer or function pointer overflows, and against other types of exploits that rely on overwriting data structures and/or putting code into those structures. The patch also makes it harder to pass in and execute the so-called 'shell-code' of exploits. The patch works transparently, ie. no application recompilation is necessary.

It is available from http://people.redhat.com/mingo/exec-shield/

Notes:
- known to break some lisp stuff
- known to break xfree86
Running some sort of intrusion detection like aide(moderate)
Running SE Linux(hard)
Becoming a luddite hermit(very hard at first, then easy)