debian.org compromise cleanup status
Last updated: Fri, 28 Nov 2003 10:22:05 UTC
After two days we have a reasonable overview of what happened to the various Debian servers and what we need to do to get everything up and running again. This page has an overview of the current status and what will happen in the next few days.
Four machines (gluck, klecker, master and murphy) were compromised. All services on those machines were shut down or moved to different machines so we could take the necessary time to determine what happened and restore the machines.
non-US, security.debian.org and the other websites running on klecker are back online for FTP and HTTP access. Archive maintenance is still down for now, so no new packages will be installed. The non-US and security archives have been verified using logs and MD5 hashes from multiple different trusted mirrors.
murphy has been reinstalled and is processing email for lists.debian.org again.
master has been restored, and debian.org email , the bug tracking system and the list archives are back online.
gluck has been reinstalled, but needs some more changes (chrooting of daemons for example) before it can be put into service.
All password used on quantz (ie all Alioth, arch and svn passwords) have been invalidated. Please use the lost password system to get a new password. Since Alioth routes email for Debian developers via the debian.org domain which is currently not yet functional they can not do this at this moment. We expect debian.org email services to be restored this weekend though. All ssh authorised keys have been removed as well.
All accounts on other machines have been locked as a safety precaution. If you have or had access to a Debian machine and were using the same password on other machines you are strongly advised to change it as soon as possible. When the cleanup is done all passwords will be invalidated and accounts unlocked and people can request a new password through the email robot on db.debian.org .
Since we are dealing with an as yet unknown exploit shell access is currently not being restored on machines. When we are confident we can protected ourselves from the exploit shell access will be restored.