Tuesday, February 19, 2008
Making blogged source code readable
Just a quick note, I've reformatted my source code examples using http://formatmysourcecode.blogspot.com/.
Monday, February 18, 2008
Keeping sensitive data on the crypto disks
Previously, I described how to create one or more crytpographic partitions. The data stored on those partitions is not retrievable without the 32-digit hexadecimal key that protects it, the key being constructed from a passphrase input by the user. It may seem that this is sufficient to protect sensitive data, making sure simply to create and edit your files only in that partition. However, there are some subtle details that have to be kept in mind.
Information stored on an unencrypted ext2 or ext3 partition has an unknown persistence. A file that was stored there, and later deleted, may be partially or fully recoverable at some time in the future. To be sure of the confidentiality of your data, you have to make sure that it has never been stored to an unencrypted partition.
If you start up your favourite text editor, telling it to create a new file in some place, let's call it /crypto/sensitive.txt, and then start typing, you may expect that the data never lands on an unencrypted partition. However, there are at least four things to be careful of:
So, what do we do about /tmp? Well, one simple solution is an overmount. While you normally mount a partition onto an empty directory, it is legal to mount onto a directory that is not empty. The files that were present in that directory are mostly inaccessible after that (a process with access to file descriptors that it opened before the mount will still be able to operate on those files, but they will be invisible to new open operations by pathname).
We're assuming you have at least one cryptographic partition. So, create a directory on that partition, let's say /crypto/tmp. After you have formatted and mounted your cryptographic partition, run this command. You only have to do this once, the first time you set up cryptographic disks.
Now, you can add the following command to the end of the script in the previous post, the script that mounts your formatted disks:
After you've done this, the system will still boot up as usual, using its unencrypted /tmp partition. Then, the root user can run the script from the previous post, now modified to have this extra mount line on the end of it. After entering the passphrase the script will do its work and exit, at which time your /tmp partition will have been replaced with the one in /crypto. Note that if your system starts up in X, with a graphical login screen, you will have to restart it after you have overmounted /tmp, otherwise you will find that X programs fail to work at all. I usually restart X by issuing a simple "killall X" command, and letting the xdm or gdm program start it back up again. This is a lot of trouble, but all manner of things can be stored on your /tmp disk. Firefox will store downloaded files such as PDFs there when there is a helper application ready to use them.
That leaves us with swap. Encrypting the swap space is actually very easy:
This can run unattended during the bootup. It creates a random cryptographic key using /dev/urandom, a device especially designed to produce true random numbers even during a system bootup sequence. This random key is used to create an encrypted interface to /dev/hda6. It is formatted as a swap partition, and then enabled. A new key will be generated each time the system boots, so nothing in swap space will survive a reboot. Note that there do exist suspend-to-disk procedures for Linux that store a memory image on the swap partition. If you intend to use such a suspend system, you will have to ensure that it does not attempt to write to the cryptographic swap partition, or you'll have to defer mounting the swap partition until the root user can enter a specific passphrase, thereby allowing you to preserve the contents across a reboot. If you're supplying a passphrase to handle encryption on the swap space, you should not run mkswap, except the first time you set up the partition (think of mkswap as being a reformat).
The question of how to protect your backup copies of sensitive files is entirely dependent on what system you use for backups. You may be able to pipe your backups through the des binary, or you may be able to store the backups on encrypted filesystems, but there are too many variations for me to offer much advice here. The security of your backups is not something that can be ignored, as has been made all to obvious with the various data disclosure scares that occur with alarming regularity when shipments of tapes or CDs fail to arrive at their destinations.
UPDATE
See my followup article for a warning about a vulnerability in this technique.
Information stored on an unencrypted ext2 or ext3 partition has an unknown persistence. A file that was stored there, and later deleted, may be partially or fully recoverable at some time in the future. To be sure of the confidentiality of your data, you have to make sure that it has never been stored to an unencrypted partition.
If you start up your favourite text editor, telling it to create a new file in some place, let's call it /crypto/sensitive.txt, and then start typing, you may expect that the data never lands on an unencrypted partition. However, there are at least four things to be careful of:
- The editor may store information in your home directory, which may not be on the encrypted partition. It might store some of the file contents there, or it might store file metadata. Your editor may keep a table of filenames recently visited in /home, with information about the line number last visited. Your editor might be configured to store crash-recovery autosave files in a directory under your /home directory.
- The editor may sometimes store the contents of a working buffer to a file in /tmp.
- The computer may come under memory pressure, resulting in some of your data being sent to the swap device.
- Your backups may not be as well protected as the files on the cryptographic disk.
So, what do we do about /tmp? Well, one simple solution is an overmount. While you normally mount a partition onto an empty directory, it is legal to mount onto a directory that is not empty. The files that were present in that directory are mostly inaccessible after that (a process with access to file descriptors that it opened before the mount will still be able to operate on those files, but they will be invisible to new open operations by pathname).
We're assuming you have at least one cryptographic partition. So, create a directory on that partition, let's say /crypto/tmp. After you have formatted and mounted your cryptographic partition, run this command. You only have to do this once, the first time you set up cryptographic disks.
mkdir --mode=01777 /crypto/tmpNow, you can add the following command to the end of the script in the previous post, the script that mounts your formatted disks:
mount --bind /crypto/tmp /tmpAfter you've done this, the system will still boot up as usual, using its unencrypted /tmp partition. Then, the root user can run the script from the previous post, now modified to have this extra mount line on the end of it. After entering the passphrase the script will do its work and exit, at which time your /tmp partition will have been replaced with the one in /crypto. Note that if your system starts up in X, with a graphical login screen, you will have to restart it after you have overmounted /tmp, otherwise you will find that X programs fail to work at all. I usually restart X by issuing a simple "killall X" command, and letting the xdm or gdm program start it back up again. This is a lot of trouble, but all manner of things can be stored on your /tmp disk. Firefox will store downloaded files such as PDFs there when there is a helper application ready to use them.
That leaves us with swap. Encrypting the swap space is actually very easy:
# Encrypt the swap partition
hashed=`dd if=/dev/urandom bs=1 count=64 | md5sum | awk ' { print $1 } '`
dmsetup create SWP <<DONE
0 `blockdev --getsize /dev/hda6` crypt aes-plain $hashed 0 /dev/hda6 0
DONE
mkswap /dev/mapper/SWP
swapon /dev/mapper/SWP
This can run unattended during the bootup. It creates a random cryptographic key using /dev/urandom, a device especially designed to produce true random numbers even during a system bootup sequence. This random key is used to create an encrypted interface to /dev/hda6. It is formatted as a swap partition, and then enabled. A new key will be generated each time the system boots, so nothing in swap space will survive a reboot. Note that there do exist suspend-to-disk procedures for Linux that store a memory image on the swap partition. If you intend to use such a suspend system, you will have to ensure that it does not attempt to write to the cryptographic swap partition, or you'll have to defer mounting the swap partition until the root user can enter a specific passphrase, thereby allowing you to preserve the contents across a reboot. If you're supplying a passphrase to handle encryption on the swap space, you should not run mkswap, except the first time you set up the partition (think of mkswap as being a reformat).
The question of how to protect your backup copies of sensitive files is entirely dependent on what system you use for backups. You may be able to pipe your backups through the des binary, or you may be able to store the backups on encrypted filesystems, but there are too many variations for me to offer much advice here. The security of your backups is not something that can be ignored, as has been made all to obvious with the various data disclosure scares that occur with alarming regularity when shipments of tapes or CDs fail to arrive at their destinations.
UPDATE
See my followup article for a warning about a vulnerability in this technique.
Sunday, February 17, 2008
Cryptographic mounts
Some of the data on my computers is stuff that I'd rather not let into the hands of a random stranger. Work-related files, proprietary data or source code, banking information, or other sensitive files. A laptop can go missing, an entire desktop computer can be carried away. It would be nice if the sensitive data were inaccessible in that event.
This leads us to cryptographic mounts. Partitions whose contents cannot be read without the knowledge of a secret that is not stored in the computer. I use a passphrase, but if you are the kind of person who memorizes 32 digit hexadecimal numbers, you can skip the passphrase. The appropriate features to enable in the kernel, either as modules or compiled directly in, are MD (the same subsystem that controls RAID) and two features in that subsystem, BLK_DEV_MD, and DM_CRYPT. You also need a cryptographic algorithm available. I use AES encryption on my partitions, but there are many others available. I have activated the CRYPTO_AES module, plus the appropriate architecture specific module, CRYPTO_AES_X86_64 for my desktop machine and CRYPTO_AES_586 for my laptop.
So, let's say you have one or more blank partitions that you'd like to set up as a cryptographic partitions, all with the same passphrase. You start with this script:
What this script does is to prompt the user for a passphrase, without echoing it to the screen. Once the passphrase is entered, it is converted to a 32 character hexadecimal string with the MD5 program. I use a here document, marked with the << characters, because that way the hexadecimal string does not appear in the process status list. Simply using echo risks having the secret visible to any user who types ps at the correct moment. Then, the dmsetup program creates the cryptographic mapping, using the hex sequence as the cryptographic key.
You will have to change the values of the $partition and $partition2 variables to correspond to those on your system. Note that volume labels are unavailable, because the system can't read the label off a cryptographic partition before the passphrase has been supplied.
Run this script, entering the passphrase. It's important that you do this through the script, and not manually at the command line, because later you'll modify the script to mount your cryptographic partitions, and you want to ensure that exactly the same code read your passphrase when you created the partitions as will read your passphrase when you try to mount the partitions after a reboot some time in the future.
When the script exits, you will have two new objects appearing in the /dev/mapper directory. In this case, they are /dev/mapper/Crypto1 and /dev/mapper/Crypto2. So, in this example, /dev/sda6 is the encrypted volume, and /dev/mapper/Crypto1 is the decrypted version of the same volume. You do all of your work on /dev/mapper/Crypto1. You format and mount that device, never /dev/sda6.
This command will create an ext3 filesystem with 0 bytes reserved for the superuser.
Now, you can mount /dev/mapper/Crypto1 onto a mount point, and start copying files there as usual. Until you remove the cryptographic mapping, the data is available as a normal mounted partition. So, we now append some code to the script above to allow the partitions to be mounted by the root user after a reboot. Take the script above and add the following lines to the bottom:
This runs fsck on the partitions, if necessary (remember, fstab can't fsck these partitions because it doesn't know the passphrase). Note that if you entered the wrong passphrase, you'll find out at this point, when e2fsck fails to identify the partition as being an ext2 or ext3 partition.
It then manually mounts the cryptographic partitions onto the mountpoints in $mtpt1 and $mtpt2. In the event of a mount failure, it unmounts everything and removes the cryptographic mappings.
The next time the computer is rebooted, the root user will have to run this script and enter the correct passphrase before the data on those drives is readable. If somebody else obtains your laptop, any mounted cryptographic partitions will be unavailable if the computer is rebooted, or the drive removed from the laptop and inserted into another machine.
This is only half the story. In a later post I'll describe the care you have to take to make sure your sensitive data does not wind up as readable plaintext somewhere on your filesystem.
This leads us to cryptographic mounts. Partitions whose contents cannot be read without the knowledge of a secret that is not stored in the computer. I use a passphrase, but if you are the kind of person who memorizes 32 digit hexadecimal numbers, you can skip the passphrase. The appropriate features to enable in the kernel, either as modules or compiled directly in, are MD (the same subsystem that controls RAID) and two features in that subsystem, BLK_DEV_MD, and DM_CRYPT. You also need a cryptographic algorithm available. I use AES encryption on my partitions, but there are many others available. I have activated the CRYPTO_AES module, plus the appropriate architecture specific module, CRYPTO_AES_X86_64 for my desktop machine and CRYPTO_AES_586 for my laptop.
So, let's say you have one or more blank partitions that you'd like to set up as a cryptographic partitions, all with the same passphrase. You start with this script:
#! /bin/sh
#
partition=/dev/sda6
partition2=/dev/sdc6
mtpt=/crypto
mapname1=Crypto1
mapname2=Crypto2
holdIFS="$IFS"
IFS=""
echo -n "Enter the passphrase: "
read -s oneline
IFS="$holdIFS"
{ hashed=`md5sum | awk ' { print $1 } '` ; }<<DONE
$oneline
DONE
dmsetup create $mapname1 <<DONE
0 `blockdev --getsize $partition` crypt aes-plain $hashed 0 $partition 0
DONE
dmsetup create $mapname2 <<ONE
0 `blockdev --getsize $partition2` crypt aes-plain $hashed 0 $partition2 0
DONE
What this script does is to prompt the user for a passphrase, without echoing it to the screen. Once the passphrase is entered, it is converted to a 32 character hexadecimal string with the MD5 program. I use a here document, marked with the << characters, because that way the hexadecimal string does not appear in the process status list. Simply using echo risks having the secret visible to any user who types ps at the correct moment. Then, the dmsetup program creates the cryptographic mapping, using the hex sequence as the cryptographic key.
You will have to change the values of the $partition and $partition2 variables to correspond to those on your system. Note that volume labels are unavailable, because the system can't read the label off a cryptographic partition before the passphrase has been supplied.
Run this script, entering the passphrase. It's important that you do this through the script, and not manually at the command line, because later you'll modify the script to mount your cryptographic partitions, and you want to ensure that exactly the same code read your passphrase when you created the partitions as will read your passphrase when you try to mount the partitions after a reboot some time in the future.
When the script exits, you will have two new objects appearing in the /dev/mapper directory. In this case, they are /dev/mapper/Crypto1 and /dev/mapper/Crypto2. So, in this example, /dev/sda6 is the encrypted volume, and /dev/mapper/Crypto1 is the decrypted version of the same volume. You do all of your work on /dev/mapper/Crypto1. You format and mount that device, never /dev/sda6.
This command will create an ext3 filesystem with 0 bytes reserved for the superuser.
/sbin/mke2fs -j -m 0 /dev/mapper/Crypto1Now, you can mount /dev/mapper/Crypto1 onto a mount point, and start copying files there as usual. Until you remove the cryptographic mapping, the data is available as a normal mounted partition. So, we now append some code to the script above to allow the partitions to be mounted by the root user after a reboot. Take the script above and add the following lines to the bottom:
/sbin/e2fsck -y /dev/mapper/$mapname1 || \
{ dmsetup remove $mapname1 ; echo "" ; echo "fsck failed"; exit 1; }
/sbin/e2fsck -y /dev/mapper/$mapname2 || \
{ dmsetup remove $mapname1; dmsetup remove $mapname2 ;\
echo "" ; echo "fsck failed"; exit 1; }
mount -onodiratime /dev/mapper/$mapname1 $mtpt1 || \
{ dmsetup remove $mapname1 ; dmsetup remove $mapname2 ; \
echo "" ; echo "Failed" ; exit 1 ; }
mount -onodiratime /dev/mapper/$mapname1 $mtpt2 || \
{ umount $mtpt ; \
dmsetup remove $mapname1 ; dmsetup remove $mapname2 ; \
echo "" ; echo "Failed" ; exit 1 ; }
echo ""
This runs fsck on the partitions, if necessary (remember, fstab can't fsck these partitions because it doesn't know the passphrase). Note that if you entered the wrong passphrase, you'll find out at this point, when e2fsck fails to identify the partition as being an ext2 or ext3 partition.
It then manually mounts the cryptographic partitions onto the mountpoints in $mtpt1 and $mtpt2. In the event of a mount failure, it unmounts everything and removes the cryptographic mappings.
The next time the computer is rebooted, the root user will have to run this script and enter the correct passphrase before the data on those drives is readable. If somebody else obtains your laptop, any mounted cryptographic partitions will be unavailable if the computer is rebooted, or the drive removed from the laptop and inserted into another machine.
This is only half the story. In a later post I'll describe the care you have to take to make sure your sensitive data does not wind up as readable plaintext somewhere on your filesystem.
Why do I have so many hard drives?
There are five hard drives in my main computer. There is no RAID setup. Why?
Hard drives fail. I've had the drive holding my root partition fail more than once. When that happens, I used to restore from backup. I would make a backup tape at least once a week, but a badly timed disk failure could still result in the loss of a lot of work.
My solution to this has been to buy my hard drives in matched pairs. I partition them equally, format them the same way, and install them both in the computer. One of them is the live disk, the other is the spare. The spare is kept unmounted and spun down. Every night around 3:00 AM, a cron job spins up the spares drives. Then, one partition at a time is fsck-ed, mounted, and copied to. The shell script uses rdist to synchronize the contents of the two partitions. Finally, I take special care to make the backup drive bootable. I use the LILO boot loader, so, when the root partition is mounted under /mnt/backup, the script executes the command:
which, on my system, writes the LILO boot magic to the backup boot drive, which appears as /dev/sdc when it is the spare in my system. My lilo.conf file, on both the live system and the spare, refer to the boot drive as being /dev/sda, but this '-b' switch overrides that, so that the information is written to the boot block of the current /dev/sdc, but is written so that is appropriate for booting the device at /dev/sda (which it will appear to be should my live boot drive fail and be removed from the system).
Next, I use volume labels to mount my partitions. You can't have duplicate labels in the system, so my spare drive has labels with the suffix "_bak" applied. That means that the /etc/fstab file suitable for the live drive would not work if the spare were booted with that fstab. To solve this problem, the copying script runs this command after it finishes copying the files in /etc:
which has the effect of renaming the labels in the fstab to their versions with the _bak suffix, so they match the volume partitions on the spare hard drive.
OK, that sounds like a lot of work, why do I do it? What does it buy me?
First of all, it gives me automatic backups. Every night, every file is backed up. When I go to the computer at the beginning of the day, the spare drive holds a copy of the filesystem as it appeared when I went to sleep the night before. Now, if I do something really unwise, deleting a pile of important files, or similarly mess up the filesystem, I have a backup that I haven't deleted. If I were to use RAID, deleting a file would delete it immediately from my backup, which isn't what I want. As long as I realize there's a problem before the end of the evening, I can always recover the machine to the way it looked before I started changing things in the morning. If I don't have enough time to verify that the things I've done are OK, I turn off the backup for a night by editing the script.
Another important thing it allows me to do is to test really risky operations. For instance, replacing glibc on a live box can be tricky. In recent years, the process has been improved to the point that it's not really scary to type "make install" on a live system, but ten years ago that would almost certainly have confused the dynamic linker enough that you would be forced to go to rescue floppies. Now, though, I can test it safely. I prepare for the risky operation, and then before doing it, I run the backup script. When that completes, I mount the complete spare filesystem under a mountpoint, /mnt/chroot. I chroot into that directory, and I am now running in the spare. I can try the unsafe operation, installing a new glibc, or a new bash, or something else critical to the operation of the Linux box. If things go badly wrong, I type "exit", and I'm back in the boot drive, with a mounted image of the damage in /mnt/chroot. I can investigate that filesystem, figure out what went wrong and how to fix it, and avoid the problem when the time comes to do the operation "for real". Then, I unmount the partitions under /mnt/chroot and re-run my backup script, and everything on the spare drive is restored. Think of it as a sort of semi-virtual machine for investigating dangerous filesystem operations.
The other thing this gives me is a live filesystem on a spare drive. When my hard drive fails (not "if", "when", your hard drive will fail one day), it's a simple matter of removing the bad hardware from the box, re-jumpering the spare if necessary, and then rebooting the box. I have had my computer up and running again in less than ten minutes, having lost, at most, the things I did earlier in the same day. While you get this benefit with RAID, the other advantages listed above are not easily available with RAID.
Of course, this is fine, but it's not enough for proper safety. The entire computer could catch fire, destroying all of my hard drives at once. I still make periodic backups to writable DVDs. I use afio for my backups, asking it to break the archive into chunks a bit larger than 4 GB, then burn them onto DVDs formatted with the ext2 filesystem (you don't have to use a UDF filesystem on a DVD, ext2 works just fine, and it's certain to be available when you're using any rescue and recovery disk). Once I've written the DVDs, I put them in an envelope, mark it with the date, and give it to relatives to hang onto, as off-site backups.
So, one pair of drives is for my /home partition, one pair for the other partitions on my system. Why do I have 5 drives? Well, the fifth one isn't backed up. It holds large data sets related to my work. These are files I can get back by carrying them home from the office on my laptop, so I don't have a backup for this drive. Occasionally I put things on that drive that I don't want to risk losing, and in that case I have a script that copies the appropriate directories to one of my backed-up partitions, but everything else on that drive is expendable.
There are two problems that can appear with large files.
Hard drives fail. I've had the drive holding my root partition fail more than once. When that happens, I used to restore from backup. I would make a backup tape at least once a week, but a badly timed disk failure could still result in the loss of a lot of work.
My solution to this has been to buy my hard drives in matched pairs. I partition them equally, format them the same way, and install them both in the computer. One of them is the live disk, the other is the spare. The spare is kept unmounted and spun down. Every night around 3:00 AM, a cron job spins up the spares drives. Then, one partition at a time is fsck-ed, mounted, and copied to. The shell script uses rdist to synchronize the contents of the two partitions. Finally, I take special care to make the backup drive bootable. I use the LILO boot loader, so, when the root partition is mounted under /mnt/backup, the script executes the command:
/sbin/lilo -r /mnt/backup -b /dev/sdc
which, on my system, writes the LILO boot magic to the backup boot drive, which appears as /dev/sdc when it is the spare in my system. My lilo.conf file, on both the live system and the spare, refer to the boot drive as being /dev/sda, but this '-b' switch overrides that, so that the information is written to the boot block of the current /dev/sdc, but is written so that is appropriate for booting the device at /dev/sda (which it will appear to be should my live boot drive fail and be removed from the system).
Next, I use volume labels to mount my partitions. You can't have duplicate labels in the system, so my spare drive has labels with the suffix "_bak" applied. That means that the /etc/fstab file suitable for the live drive would not work if the spare were booted with that fstab. To solve this problem, the copying script runs this command after it finishes copying the files in /etc:
sed -e 's|LABEL=\([^ \t]*\)\([ \t]\)|LABEL=\1_bak\2|' /etc/fstab > /mnt/backup/etc/fstab
which has the effect of renaming the labels in the fstab to their versions with the _bak suffix, so they match the volume partitions on the spare hard drive.
OK, that sounds like a lot of work, why do I do it? What does it buy me?
First of all, it gives me automatic backups. Every night, every file is backed up. When I go to the computer at the beginning of the day, the spare drive holds a copy of the filesystem as it appeared when I went to sleep the night before. Now, if I do something really unwise, deleting a pile of important files, or similarly mess up the filesystem, I have a backup that I haven't deleted. If I were to use RAID, deleting a file would delete it immediately from my backup, which isn't what I want. As long as I realize there's a problem before the end of the evening, I can always recover the machine to the way it looked before I started changing things in the morning. If I don't have enough time to verify that the things I've done are OK, I turn off the backup for a night by editing the script.
Another important thing it allows me to do is to test really risky operations. For instance, replacing glibc on a live box can be tricky. In recent years, the process has been improved to the point that it's not really scary to type "make install" on a live system, but ten years ago that would almost certainly have confused the dynamic linker enough that you would be forced to go to rescue floppies. Now, though, I can test it safely. I prepare for the risky operation, and then before doing it, I run the backup script. When that completes, I mount the complete spare filesystem under a mountpoint, /mnt/chroot. I chroot into that directory, and I am now running in the spare. I can try the unsafe operation, installing a new glibc, or a new bash, or something else critical to the operation of the Linux box. If things go badly wrong, I type "exit", and I'm back in the boot drive, with a mounted image of the damage in /mnt/chroot. I can investigate that filesystem, figure out what went wrong and how to fix it, and avoid the problem when the time comes to do the operation "for real". Then, I unmount the partitions under /mnt/chroot and re-run my backup script, and everything on the spare drive is restored. Think of it as a sort of semi-virtual machine for investigating dangerous filesystem operations.
The other thing this gives me is a live filesystem on a spare drive. When my hard drive fails (not "if", "when", your hard drive will fail one day), it's a simple matter of removing the bad hardware from the box, re-jumpering the spare if necessary, and then rebooting the box. I have had my computer up and running again in less than ten minutes, having lost, at most, the things I did earlier in the same day. While you get this benefit with RAID, the other advantages listed above are not easily available with RAID.
Of course, this is fine, but it's not enough for proper safety. The entire computer could catch fire, destroying all of my hard drives at once. I still make periodic backups to writable DVDs. I use afio for my backups, asking it to break the archive into chunks a bit larger than 4 GB, then burn them onto DVDs formatted with the ext2 filesystem (you don't have to use a UDF filesystem on a DVD, ext2 works just fine, and it's certain to be available when you're using any rescue and recovery disk). Once I've written the DVDs, I put them in an envelope, mark it with the date, and give it to relatives to hang onto, as off-site backups.
So, one pair of drives is for my /home partition, one pair for the other partitions on my system. Why do I have 5 drives? Well, the fifth one isn't backed up. It holds large data sets related to my work. These are files I can get back by carrying them home from the office on my laptop, so I don't have a backup for this drive. Occasionally I put things on that drive that I don't want to risk losing, and in that case I have a script that copies the appropriate directories to one of my backed-up partitions, but everything else on that drive is expendable.
There are two problems that can appear with large files.
- rdist doesn't handle files larger than 2 GB. I looked through the source code to see if I could fix that shortcoming, and got a bit worried about the code. So I'm working on writing my own replacement for rdist with the features I want. In the mean time, I rarely have files that large, and when I do, they don't change often, so I've been copying the files to the backup manually.
- Sometimes root's shells, even those spawned by cron, have ulimit settings. If you're not careful, you'll find that cron jobs cannot create a file in excess of some maximum size, often 1 GB. This is an inconvenient restriction, and one that I have removed on my system.
Breaking packages
I've used the term "breaking packages" a few times. As I said, I maintain my Linux boxes without a package manager. So, how did I get these Linux boxes?
My main Linux computer has over half a million files in its filesystem, and over 3000 separate executables. Where did they all come from? You need some way to start out, your computer isn't going to do much without a kernel, a shell, and a compiler.
In 1994, I installed Slackware on a 486-based computer. This computer had about 180 MB of hard drive space (nowadays that wouldn't even hold half of the kernel source tree) and 16 MB of RAM. At that time, Slackware didn't really have a package manager. It had packages, just compressed tar files of compiled binaries, grouped by function. If you weren't interested in networking, you didn't download the networking file. If you weren't interested in LaTeX, you didn't download that file. There were only a few dozen "packages", because of this very coarse granularity. The functions like "upgrade", "install", "find package owning file" weren't present. An upgrade was the same as an install, just install the new package into the filesystem, and it would probably replace the old package. To find out which package provided a certain file, you could look in per-package lists of files.
So, I never really had a package manager on that system. When I needed new programs, I downloaded the source code, compiled it, and installed it. When I moved to a new system, I brought backup images or a live hard drive to the new computer. I didn't start with a blank hard drive, I started with the hard drive from the old computer I was replacing. Over the years, I have replaced every executable that was installed in 1994 (I know this is the case because all of the files installed then were a.out format, and I have only ELF binaries on my computer now).
Sometimes, though, I've started with a computer that had a distribution installed on it. At a previous job, my laptop came with Mandrake Linux installed on it. I tried to keep the distribution alive for a while, but eventually got impatient with the package management system and broke the packages.
So, if you give me a new Linux computer and tell me it's mine to modify, a good first step for me is to kill the package manager. On an RPM-based system, that's generally achieved by recursively deleting the directory /var/lib/rpm. After that, the rpm command will stop working, and I have the finer control and more difficult task of managing the box myself.
My main Linux computer has over half a million files in its filesystem, and over 3000 separate executables. Where did they all come from? You need some way to start out, your computer isn't going to do much without a kernel, a shell, and a compiler.
In 1994, I installed Slackware on a 486-based computer. This computer had about 180 MB of hard drive space (nowadays that wouldn't even hold half of the kernel source tree) and 16 MB of RAM. At that time, Slackware didn't really have a package manager. It had packages, just compressed tar files of compiled binaries, grouped by function. If you weren't interested in networking, you didn't download the networking file. If you weren't interested in LaTeX, you didn't download that file. There were only a few dozen "packages", because of this very coarse granularity. The functions like "upgrade", "install", "find package owning file" weren't present. An upgrade was the same as an install, just install the new package into the filesystem, and it would probably replace the old package. To find out which package provided a certain file, you could look in per-package lists of files.
So, I never really had a package manager on that system. When I needed new programs, I downloaded the source code, compiled it, and installed it. When I moved to a new system, I brought backup images or a live hard drive to the new computer. I didn't start with a blank hard drive, I started with the hard drive from the old computer I was replacing. Over the years, I have replaced every executable that was installed in 1994 (I know this is the case because all of the files installed then were a.out format, and I have only ELF binaries on my computer now).
Sometimes, though, I've started with a computer that had a distribution installed on it. At a previous job, my laptop came with Mandrake Linux installed on it. I tried to keep the distribution alive for a while, but eventually got impatient with the package management system and broke the packages.
So, if you give me a new Linux computer and tell me it's mine to modify, a good first step for me is to kill the package manager. On an RPM-based system, that's generally achieved by recursively deleting the directory /var/lib/rpm. After that, the rpm command will stop working, and I have the finer control and more difficult task of managing the box myself.
What do we have running on that box?
As I mentioned in my first post, when you install a distribution you sometimes have programs running without your knowledge. Because some users may need these features, you often get them as well.
Last week at the office, somebody came up to me to ask if I could figure out why it was no longer possible to log into one of the servers. This server has a history of flakiness, there's probably a bad memory module on the board, and sometimes it becomes unresponsive. So, my co-worker, upon realizing that he couldn't log in, had rebooted the computer. However, even after the reboot, he still couldn't log on, either as his regular user through SSH, or as root on the console.
The first step, before getting out of my chair, was to telnet to port 22 on the box. I got a "connected" message, and a text string indicating that I was attached to an SSH daemon. This told me that the kernel was alive, it was accepting new connections and passing them to the appropriate processes, which were themselves able to make forward progress. So, the box wasn't wedged. I went to the console, and tried to log in through the getty running on the text login screen. I entered 'root' at the username, and got a password prompt. When I entered the password and pressed ENTER, the getty process froze, and did not present me with a shell.
So, we have two very different authentication schemes that are failing to allow logins. The console doesn't allow root logins. Something seemed to be interfering with the general activity of authentication. The first thought is that this might be a PAM problem, but it would be a strange one. We didn't get authentication failure messages, we got a hang after authentication. Root's credentials were stored on the local drive, so it wasn't an LDAP issue, and in any case, the machine was on the network, and there weren't LDAP problems anywhere else in the office.
When multiple independent programs fail together, the next thought is that there's probably a full disk somewhere. If you fill up /tmp your system can start to behave very strangely. The login problems were a symptom of something, as yet unknown. So, the next thing to do is to check the hard drive to see if we had any full partitions. Because I didn't know what else might be misbehaving, I wanted to avoid all of the startup jobs, so I rebooted the machine with an appended kernel parameter, "init=/bin/bash". Instead of running the usual /sbin/init, and all of the various scripts under /etc/init.d, the computer would start up the kernel and then drop immediately to a root shell. No logins, no passwords, no startup scripts. I could then run 'df' at the prompt, and confirm that there were no partitions within 5% of full (remember that a default ext2 format will reserve 5% of the blocks for root, so a disk that's 96% full could actually be entirely full for some users). Checking with 'df -i' showed that we had not run out of inodes either.
So, what's next? I decided to reboot the machine into single-user mode so that I could easily modify files on the disk but still get onto the computer without a password. This is done by appending the parameter "S" on the kernel boot line. Again, I get a shell, but this time the disks are read-write mounted, and various services have started up. So, I modify the inittab. I replaced the getty on tty1 with /bin/bash. That means that when the computer is rebooted into multi-user mode, tty1 has a root shell while the other ttys are still running their gettys.
Reboot into the usual multi-user mode. I have a root shell on tty1. I run "ps ax", and find the PID of the getty on tty2. Then, I run the command
at the shell prompt of tty1. Changing virtual consoles to tty2 with the usual command, CTRL-ALT-F2, I am presented with a login prompt. I enter the username 'root', and enter the password. The program hangs. So, I change back to tty1 to see what strace has to say about the program. The last things the program did are on the screen. It opened a device called /dev/audit, did some things with it, then issued an ioctl() on the file descriptor. That ioctl was not returning to the caller, so the program was blocking waiting for a response from something associated with /dev/audit.
None of us had heard of /dev/audit, so it was time to do a bit of research. It turned out to be a package that was included in the RHEL distribution installed on that computer. There is communication between the device and a daemon. That daemon keeps logs, so I went to its logging directory to see what was there. I found 4 GB of data there. Apparently that had reached some sort of internal limit, and the daemon responded by forbidding further auditable actions until some of the logs were removed by the administrator. Logins, being auditable actions, were blocked.
So, delete all of the logs in the directory, and reboot the computer. Everything returned to normal.
Now, a logging function like this is very useful for some users. There are some people who must know exactly who logged into the machine, what database entries they accessed or modified, and so on. We are not such people. A service we never knew about, enabled for all because it is useful by some, wound up locking us out of our own machine.
It's a security feature that logins are forbidden until the logs have been inspected and removed. If you're going to design a function like this, then this is the correct way to go about it. Of course, it was very easy for me to overcome this security feature with access to the console, but that's generally true. I probably would have set it up so that gettys are permitted to log in as root even when an audit failure occurs, but that level of flexibility may not be available, if the behaviour is driven by a special PAM module or a patched glibc.
Last week at the office, somebody came up to me to ask if I could figure out why it was no longer possible to log into one of the servers. This server has a history of flakiness, there's probably a bad memory module on the board, and sometimes it becomes unresponsive. So, my co-worker, upon realizing that he couldn't log in, had rebooted the computer. However, even after the reboot, he still couldn't log on, either as his regular user through SSH, or as root on the console.
The first step, before getting out of my chair, was to telnet to port 22 on the box. I got a "connected" message, and a text string indicating that I was attached to an SSH daemon. This told me that the kernel was alive, it was accepting new connections and passing them to the appropriate processes, which were themselves able to make forward progress. So, the box wasn't wedged. I went to the console, and tried to log in through the getty running on the text login screen. I entered 'root' at the username, and got a password prompt. When I entered the password and pressed ENTER, the getty process froze, and did not present me with a shell.
So, we have two very different authentication schemes that are failing to allow logins. The console doesn't allow root logins. Something seemed to be interfering with the general activity of authentication. The first thought is that this might be a PAM problem, but it would be a strange one. We didn't get authentication failure messages, we got a hang after authentication. Root's credentials were stored on the local drive, so it wasn't an LDAP issue, and in any case, the machine was on the network, and there weren't LDAP problems anywhere else in the office.
When multiple independent programs fail together, the next thought is that there's probably a full disk somewhere. If you fill up /tmp your system can start to behave very strangely. The login problems were a symptom of something, as yet unknown. So, the next thing to do is to check the hard drive to see if we had any full partitions. Because I didn't know what else might be misbehaving, I wanted to avoid all of the startup jobs, so I rebooted the machine with an appended kernel parameter, "init=/bin/bash". Instead of running the usual /sbin/init, and all of the various scripts under /etc/init.d, the computer would start up the kernel and then drop immediately to a root shell. No logins, no passwords, no startup scripts. I could then run 'df' at the prompt, and confirm that there were no partitions within 5% of full (remember that a default ext2 format will reserve 5% of the blocks for root, so a disk that's 96% full could actually be entirely full for some users). Checking with 'df -i' showed that we had not run out of inodes either.
So, what's next? I decided to reboot the machine into single-user mode so that I could easily modify files on the disk but still get onto the computer without a password. This is done by appending the parameter "S" on the kernel boot line. Again, I get a shell, but this time the disks are read-write mounted, and various services have started up. So, I modify the inittab. I replaced the getty on tty1 with /bin/bash. That means that when the computer is rebooted into multi-user mode, tty1 has a root shell while the other ttys are still running their gettys.
Reboot into the usual multi-user mode. I have a root shell on tty1. I run "ps ax", and find the PID of the getty on tty2. Then, I run the command
strace -f -p <PID>
at the shell prompt of tty1. Changing virtual consoles to tty2 with the usual command, CTRL-ALT-F2, I am presented with a login prompt. I enter the username 'root', and enter the password. The program hangs. So, I change back to tty1 to see what strace has to say about the program. The last things the program did are on the screen. It opened a device called /dev/audit, did some things with it, then issued an ioctl() on the file descriptor. That ioctl was not returning to the caller, so the program was blocking waiting for a response from something associated with /dev/audit.
None of us had heard of /dev/audit, so it was time to do a bit of research. It turned out to be a package that was included in the RHEL distribution installed on that computer. There is communication between the device and a daemon. That daemon keeps logs, so I went to its logging directory to see what was there. I found 4 GB of data there. Apparently that had reached some sort of internal limit, and the daemon responded by forbidding further auditable actions until some of the logs were removed by the administrator. Logins, being auditable actions, were blocked.
So, delete all of the logs in the directory, and reboot the computer. Everything returned to normal.
Now, a logging function like this is very useful for some users. There are some people who must know exactly who logged into the machine, what database entries they accessed or modified, and so on. We are not such people. A service we never knew about, enabled for all because it is useful by some, wound up locking us out of our own machine.
It's a security feature that logins are forbidden until the logs have been inspected and removed. If you're going to design a function like this, then this is the correct way to go about it. Of course, it was very easy for me to overcome this security feature with access to the console, but that's generally true. I probably would have set it up so that gettys are permitted to log in as root even when an audit failure occurs, but that level of flexibility may not be available, if the behaviour is driven by a special PAM module or a patched glibc.
Breaking packages on the MythTV box
As I mentioned earlier, I have a MythTV computer, installed from packages, but I've broken some of the packages. Here are some of the issues that I had with the packages, and how I solved them.
Two of the package manager drawbacks I've mentioned previously appear here: the one-size-fits-all approach to software packaging, and the failure to receive timely updates.
The MythTV box is on old hardware. Because it has hardware assistance for both MPEG encoding and decoding, I didn't need a new computer with a fast CPU. The fact that this is old hardware, with a 7-year old BIOS, may be why I had problems, but I found it easier to break the packages than to try to solve the problems under the constraints of the package system.
First, the MythTV box controls an infra-red LED attached to its serial port, allowing it to change the channels on a digital cable box. This requires the use of the LIRC package, and the lirc_serial kernel module. Well, at the time I set this up, the lirc_serial module was having problems with the SMP kernel. The system would generate an oops quite regularly when it wanted to change channels. Looking at the oops logs, I could see that there were problems specifically with SMP. My MythTV box has only one CPU, so I didn't need an SMP kernel, but because some users will have SMP computers, the KnoppMyth distribution ships with an SMP kernel. I tried to find a non-SMP kernel for the system, without success. So, the easiest way to fix the problem was just to download a recent kernel source tree from kernel.org, copy the configuration file from the Knoppix kernel, and reconfigure it as non-SMP. The spontaneous reboots stopped occurring. The package manager still believes that it knows what kernel is running on the computer, but that isn't what is really installed.
When I installed the MythTV box, the software was still a bit immature, and a stability fix in the form of version 0.20 came out several months later. I waited a few weeks with no update to the distribution, and no word of when an update might become available. Eventually, I grew impatient and downloaded the source code of 0.20 myself, recompiled it on the MythTV box, and installed it over top of the existing programs.
There was one other impact of the one-size-fits-all approach that caused difficulties with the MythTV box. I was regularly recording a television show between 6:00AM and 6:30AM. A few minutes before the end of the show, the recording would have problems. The audio would break up, and the video would jump. It appeared that the program was losing frames of data, either because it was losing interrupts, or because it couldn't get the data to the disk quickly enough. Because it happened at about the same time every day, I expected it was probably a cron job. I got a root shell on the box, and asked for the list of all root-owned cron jobs with the command "crontab -l". This reported that there were no root-owned cron jobs. I mistrusted this result, and did more investigation. As I mentioned in the first post, distribution packagers often break up a configuration file into a set of separate files. They did that with cron jobs, which means that the command-line tool that ought to tell you all about root-owned cron jobs didn't report the full set of such processes. A bit of digging around in /etc showed that the slocate database update was being run at that time. This process scans the entire disk, making a list of the files on it. While probably useful in a general context, this is an unnecessary operation on an appliance box that isn't changing, particularly when it results in so much bus traffic that the primary function of the box is degraded. My solution was to change the /etc/crontab file (which is, itself, not viewed by "crontab -l") so that a cron job would be skipped if there were any users (reported by the 'fuser' command) of either of the two video input devices, /dev/video0 and /dev/video1.
Two of the package manager drawbacks I've mentioned previously appear here: the one-size-fits-all approach to software packaging, and the failure to receive timely updates.
The MythTV box is on old hardware. Because it has hardware assistance for both MPEG encoding and decoding, I didn't need a new computer with a fast CPU. The fact that this is old hardware, with a 7-year old BIOS, may be why I had problems, but I found it easier to break the packages than to try to solve the problems under the constraints of the package system.
First, the MythTV box controls an infra-red LED attached to its serial port, allowing it to change the channels on a digital cable box. This requires the use of the LIRC package, and the lirc_serial kernel module. Well, at the time I set this up, the lirc_serial module was having problems with the SMP kernel. The system would generate an oops quite regularly when it wanted to change channels. Looking at the oops logs, I could see that there were problems specifically with SMP. My MythTV box has only one CPU, so I didn't need an SMP kernel, but because some users will have SMP computers, the KnoppMyth distribution ships with an SMP kernel. I tried to find a non-SMP kernel for the system, without success. So, the easiest way to fix the problem was just to download a recent kernel source tree from kernel.org, copy the configuration file from the Knoppix kernel, and reconfigure it as non-SMP. The spontaneous reboots stopped occurring. The package manager still believes that it knows what kernel is running on the computer, but that isn't what is really installed.
When I installed the MythTV box, the software was still a bit immature, and a stability fix in the form of version 0.20 came out several months later. I waited a few weeks with no update to the distribution, and no word of when an update might become available. Eventually, I grew impatient and downloaded the source code of 0.20 myself, recompiled it on the MythTV box, and installed it over top of the existing programs.
There was one other impact of the one-size-fits-all approach that caused difficulties with the MythTV box. I was regularly recording a television show between 6:00AM and 6:30AM. A few minutes before the end of the show, the recording would have problems. The audio would break up, and the video would jump. It appeared that the program was losing frames of data, either because it was losing interrupts, or because it couldn't get the data to the disk quickly enough. Because it happened at about the same time every day, I expected it was probably a cron job. I got a root shell on the box, and asked for the list of all root-owned cron jobs with the command "crontab -l". This reported that there were no root-owned cron jobs. I mistrusted this result, and did more investigation. As I mentioned in the first post, distribution packagers often break up a configuration file into a set of separate files. They did that with cron jobs, which means that the command-line tool that ought to tell you all about root-owned cron jobs didn't report the full set of such processes. A bit of digging around in /etc showed that the slocate database update was being run at that time. This process scans the entire disk, making a list of the files on it. While probably useful in a general context, this is an unnecessary operation on an appliance box that isn't changing, particularly when it results in so much bus traffic that the primary function of the box is degraded. My solution was to change the /etc/crontab file (which is, itself, not viewed by "crontab -l") so that a cron job would be skipped if there were any users (reported by the 'fuser' command) of either of the two video input devices, /dev/video0 and /dev/video1.
Subscribe to:
Posts (Atom)