Hot-replacing a failing disk that is a part of Linux Software RAID and ZFS pools

Disks break: not "if", "when".

Yes, that's what they do. I run a 4-disk setup that hold one Linux Software RAID6 array, and two ZFS RAIDZ2 pools. 

Clouds in the sky

As of a few days ago, one of the disks started to fail, which was apparent by the syslog entries like these:

[1318523.293294] ata2.00: failed command: READ FPDMA QUEUED

[1318523.304015] ata2.00: cmd 60/01:00:8f:da:14/00:00:4d:00:00/40 tag 0 ncq 512 in

[1318523.304021]          res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)

[1318523.346321] ata2.00: status: { DRDY ERR }

[1318523.356810] ata2.00: error: { UNC }

[1318523.367279] ata2.00: failed command: READ FPDMA QUEUED

[1318523.377664] ata2.00: cmd 60/3f:08:60:ad:14/00:00:4d:00:00/40 tag 1 ncq 32256 in

[1318523.377670]          res 41/40:00:98:ad:14/00:00:4d:00:00/40 Emask 0x409 (media error)

[1318523.419883] ata2.00: status: { DRDY ERR }

[1318523.430424] ata2.00: error: { UNC }

[1318523.440904] ata2.00: failed command: READ FPDMA QUEUED

[1318523.451164] ata2.00: cmd 60/01:10:95:29:00/00:00:4e:00:00/40 tag 2 ncq 512 in

[1318523.451169]          res 41/40:00:00:00:00/00:00:00:00:00/00 Emask 0x9 (media error)

[1318523.492656] ata2.00: status: { DRDY ERR }

[1318523.503246] ata2.00: error: { UNC }

As I did not have a spare disk on hand (tsk, tsk, tsk, yes, I know...) I immediately ordered one, even before sending the old disk for RMA. Initially, as I ran a zpool scrub on the pools, there would be only these messages, but the zpool itself did not notice trouble. 

Thunderstorms in the sky

As of yesterday, errors started making it to the zpool layer:

$ sudo zpool status

  pool: data

 state: ONLINE

status: One or more devices has experienced an unrecoverable error.  An

attempt was made to correct the error.  Applications are unaffected.

action: Determine if the device needs to be replaced, and clear the errors

using 'zpool clear' or replace the device with 'zpool replace'.

   see: http://zfsonlinux.org/msg/ZFS-8000-9P

 scan: scrub repaired 356K in 3h25m with 0 errors on Sun Oct  7 15:16:16 2012

config:

NAME                                                 STATE     READ WRITE CKSUM

data                                                 ONLINE       0     0     0

 raidz2-0                                           ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part1  ONLINE       0     0  422K

   ata-WDC_WD2002FYPS-[serial]-part1  ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part1  ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part1  ONLINE       0     0     0

errors: No known data errors

  pool: ttank

 state: ONLINE

 scan: scrub repaired 0 in 0h56m with 0 errors on Fri Oct  5 11:53:31 2012

config:

NAME                                                 STATE     READ WRITE CKSUM

ttank                                                ONLINE       0     0     0

 raidz2-0                                           ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part3  ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part3  ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part3  ONLINE       0     0     0

   ata-WDC_WD2002FYPS-[serial]-part3  ONLINE       0     0     0

By now, the drive not only had read errors; it even started to return faulty data (despite claiming that said data is ok). Fortunately, ZFS is built from the ground up to never trust hardware, so that its checksumming mechanism detected the faulty data. Clearly, it was now time to replace that disk. Fortunately, the spare drive just came in by mail.

Taking the old disk offline

I run my disks in an IcyBox Hotplug backplane, so I wish to replace the disk without even so much as rebooting the server. One first needs to know which disk this is, of course. Since I use the disk-ID links, just looking at the symlinks in /dev/disk/by-id tells me that the disk in question is /dev/sdb.

To be safe, I read a gigabyte of data off the disk, to physically inspect which drive light switches on as I do so:

# dd if=/dev/sdb of=/dev/null bs=1048576 count=1024

Visual inspection tells me that this is the top drive in the IcyBox. Good.

As for ZFS, there is nothing special that one needs to do. For Linux Software RAID, one needs to tell the system to fail, and subsequently remove the disk from the array:

# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 

md0 : active raid6 sde2[5] sdb2[0] sdc2[4] sdd2[2]

      409996800 blocks super 1.2 level 6, 256k chunk, algorithm 2 [4/4] [UUUU]

      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices:

Fail the disk:

# mdadm /dev/md0 --fail /dev/sdb2

mdadm: set /dev/sdb2 faulty in /dev/md0

# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 

md0 : active raid6 sde2[5] sdb2[0](F) sdc2[4] sdd2[2]

      409996800 blocks super 1.2 level 6, 256k chunk, algorithm 2 [4/3] [_UUU]

      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices:

Remove the disk:

# mdadm /dev/md0 --remove /dev/sdb2

mdadm: hot removed /dev/sdb2 from /dev/md0

# cat /proc/mdstat 

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 

md0 : active raid6 sde2[5] sdc2[4] sdd2[2]

      409996800 blocks super 1.2 level 6, 256k chunk, algorithm 2 [4/3] [_UUU]

      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices:

At this point, one could yank the disk out, but it's better to tell Linux that you are going to do so. Switching off the disk and detaching it from the system is done as follows:

# echo 1 > /sys/block/sdb/device/delete

The syslog will tell you that the device indeed went offline:

[1734127.293861] sd 1:0:0:0: [sdb] Synchronizing SCSI cache

[1734127.331629] sd 1:0:0:0: [sdb] Stopping disk

[1734127.768141] ata2.00: disabled

As this point, the tray can be taken from the Hotplug backplane, and the old disk can be replaced by the new one.

Bringing the new disk online

After physically taking out the tray, removing the old disk from the tray, and adding the new disk to the tray, I replaced the tray. The kernel detects the disk:


[1743181.511929] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[1743181.512460] ata2: irq_stat 0x00000040, connection status changed
[1743181.512883] ata2: SError: { CommWake DevExch }
[1743181.513215] ata2: hard resetting link
[1743187.276049] ata2: link is slow to respond, please be patient (ready=0)
[1743190.860073] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[1743190.998197] ata2.00: ATA-9: WDC WD20EFRX-[serial], max UDMA/133
[1743190.998206] ata2.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[1743190.998836] ata2.00: configured for UDMA/133
[1743190.998855] ata2: EH complete
[1743190.999097] scsi 1:0:0:0: Direct-Access     ATA      WDC WD20EFRX-[serial]
[1743190.999679] sd 1:0:0:0: [sdf] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[1743190.999691] sd 1:0:0:0: [sdf] 4096-byte physical blocks
[1743190.999705] sd 1:0:0:0: Attached scsi generic sg1 type 0
[1743191.000185] sd 1:0:0:0: [sdf] Write Protect is off
[1743191.000197] sd 1:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[1743191.000328] sd 1:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[1743191.014153]  sdf: unknown partition table
[1743191.014902] sd 1:0:0:0: [sdf] Attached SCSI disk
[1743415.817135]  sdf: unknown partition table

Obviously, there are no partitions on the disk yet. In order to create them, I simply copy them off one of the other drives:

# sfdisk -b /dev/sdc | sfdisk /dev/sdf

This is readily picked up by the kernel:

[1743415.817135]  sdf: unknown partition table
[1743416.227972]  sdf: sdf1 sdf2 sdf3

Resilvering the arrays

The first array I decide to resilver is the most important one: the primary data pool:

# zpool replace data /dev/disk-by-id/ata-WDC_WD2002FYPS-[serial]-part1 /dev/disk/by-id/ata-WDC_WD20EFRX-[serial]-part1

This is going to take a long time: more when this is done.

No-downloading inconveniences in the digital age

My country does not allow me to download music/movies for personal use!

While a good number of countries (e.g., The Netherlands, Switzerland) have relatively sane laws that allow the downloading (though not uploading) of music and movies, there are a good number of other countries where even the downloading of music and movies for personal use is forbidden.

Even if one does live (or operates a server) in one of the latter countries, these restrictions are but small inconveniences that are easily worked around.

Case in point here is an Ubuntu Linux server in one of these countries to which somebody wants to download content from the Giganews Usenet provider, where one sets up OpenVPN himself. Note that although this article is written in terms of Linux and Giganews, the general principles readily carry over to other situations.

The solution: OpenVPN

The solution in this case is to hide the fact that your are perusing the service from the country with the backward laws that you happen to be in. A simple mechanism to do this is to use OpenVPN: ones creates an encrypted VPN tunnel over which one tunnels the connections to Giganews.

If one already has an account at Giganews, Giganews offers a branded deal through VyprVPN where you get OpenVPN access for $5 per month.

Step 1: Apply for OpenVPN access at Giganews

Just follow the steps on their website: you can't go wrong there.

Step 2: Install OpenVPN

sudo apt-get install openvpn

(easy enough)

Step 3: Install the VyprVPN root certificate

sudo wget -O /etc/openvpn/ca.vyprvpn.com.crt http://www.giganews.com/vyprvpn/ca.vyprvpn.com.crt

This allows your OpenVPN client to ascertain that it is indeed talking to VyprVPN, and not to some man-in-the-middle attack box your government may have put in place.

Step 4: Create a configuration for your VyperVPN

The easiest way to do this is to create two files: one that contains your Giganews username and password, and one that contains the OpenVPN client configuration. The names are arbitrary, but I happen to use these:

/etc/openvpn/vyprvpn.pass contains:

gn123456

abcd1234

(replace the red content with your actual username and password).

/etc/openvpn/vyprvpn.conf contains:

client

dev tun

proto udp

remote eu1.vpn.giganews.com 1194

resolv-retry infinite

nobind

persist-key

persist-tun

persist-remote-ip

ca ca.vyprvpn.com.crt

tls-remote eu1.vpn.giganews.com

auth-user-pass vyprvpn.pass

comp-lzo

verb 1

(you could replace the eu1 part with several other options, but eu1 is in the Netherlands, where downloading is legal).

Step 5a: Fire and forget

Open boot, your server will now automatically start up your VyperVPN, and route all traffic through it. You can also force it right now by issuing:

sudo /etc/init.d/openvpn restart

If that is not what you want, e.g., because you use the box for other purposes, too, the next step will describe how to route just your Giganews traffic through the VPN.

Step 5b (optional): Route just Giganews traffic through the VPN.

If this is what you want, this is possible, too. Simply add the green content to your /etc/openvpn/vyprvpn.conf file:

client

route-noexec

route-up /etc/openvpn/vyprvpn-route-up.sh

down /etc/openvpn/vyprvpn-route-down.sh

script-security 2

dev tun

proto udp

remote eu1.vpn.giganews.com 1194

resolv-retry infinite

nobind

persist-key

persist-tun

persist-remote-ip

ca ca.vyprvpn.com.crt

tls-remote eu1.vpn.giganews.com

auth-user-pass vyprvpn.pass

comp-lzo

verb 1

The route-noexec option tells OpenVPN to not directly use all route pushes it gets from the VyprVPN server, but to pass options via environment variables to scripts in which you are in control of what happens.

In my case, I wanted to use news-europe.giganews.com for downloading. I used whois to figure out that their IP range in Europe is 216.196.96.0/19. The two scripts mentioned above now contain:

/etc/openvpn/vyprvpn-route-up.sh:

#!/bin/bash

# Route Giganews Europe (216.196.96.0/19), and ONLY Giganews,

# through VyprVPN.

ip route add 216.196.96.0/19 dev $dev

(note that $dev is passed in the environment by OpenVPN).

/etc/openvpn/vyprvpn-route-down.sh:

#!/bin/bash

# Remove routing for Giganews Europe (216.196.96.0/19).

ip route del 216.196.96.0/19

Step 6: Check that things work

Quickly check that your routing to Giganews indeed goes through the VPN:

traceroute news-europe.giganews.com

traceroute to news-europe.giganews.com (216.196.109.144), 30 hops max, 60 byte packets

 1  10.25.0.1 (10.25.0.1)  14.601 ms  14.606 ms  14.611 ms

 2  * * *

 3  vl304.gw1.ams.giganews.com (216.196.108.218)  15.268 ms  15.309 ms  15.274 ms

 4  news-europe.giganews.com (216.196.109.144)  14.964 ms  15.195 ms  15.210 ms

Here, the first hop being on a private subnet (10.25.0.1, on 10.0.0.0/8, which is private) tells you that traffic is routed correctly.

Happy networking!

Local-disk encryption to protect against casual privacy loss

Like many others, I store a lot of privacy-sensitive information on the disks of my local server: photos, scanned documents, and more. I do not feel the need to protect that data from those who have physical access to the machine, let alone to protect that data from authorities, should those ever come along with a (mistaken) warrant. No, the protection I seek is much simpler:

The protection I would like is against those who get one of my disks, for example when I exchange a disk under warranty. It would not be the first time that such a disk is resold, or that the friendly shop personnel scan the disk for interesting data. Also, my other server, which sits in a remote datacenter, should not leak information when a disk is exchanged.

The simple mechanism by which I now do this is by accessing the underlying disks (or partitions) of my data disks through dm_crypt , and to create zpools, mdraid, or simple filesystems on top of those dm_crypt mapped block devices. The normal way to do this is to add the required entries to /etc/crypttab, but I find that Ubuntu sets up these devices too late in the game. Therefore, I created my own script.

On my remote server, I have a script in /etc/init.d/local-cryptsetup , which contains:

#!/bin/bash

/sbin/cryptsetup -d /etc/mydevs/passwd.dat create zloop0 /dev/disk/by-id/[NAME_DISK1]

/sbin/cryptsetup -d /etc/mydevs/passwd.dat create zloop1 /dev/disk/by-id/[NAME_DISK2]

In /etc/rc2.d, /etc/rc3.d, /etc/rc4.d, and /etc/rc5.d, I symlink a link called S05local-cryptsetup to the above script. I chose the number S05, as I use these mappings are underlying devices for a ZFS ZPool, and the ZFS subsystem is started at S20. As S05 < S20, this ensures that the mappings are available before ZFS attempts to start using them.

Initializing the ZPool once was easy enough:

# zpool create tank mirror /dev/mapper/zloop0 /dev/mapper/zloop1

I ensures that the pool, and all data in it, successfully survive a reboot.

Hot-removing a SATA drive, and copying partition tables

In a multi-disk setup, one sometimes needs to replace a drive. If one has a hot-swap bay, this is surprisingly easy, but before actually pulling the drive out, one MUST tell Linux that one is about to do so:

# echo 1 > /sys/block/sdX/device/delete

You can then pull the drive out and replace it with a new one. Note, though, that the new drive will generally get a new device name (e.g., /dev/sdZ), until the next reboot.

Now, say that you want the drive to have identical partitions to another drive (say, sdY) in your system, then you simple copy the partition tables:

# sfdisk -d /dev/sdY | sfdisk /dev/sdZ

This can all be done without ever rebooting the system :-)

A _seriously_ close shave with Linux software RAID

I am running a 4-disk home server that uses the first partition on each of the four drives as a single RAID6 array using Linux mdadm software RAID. As I freed up some partitions on the remainder of each of the drives, I wanted to extend the size of the first partition on each drive so that I could first grow the RAID6 array, and then grow the filesystem.

That sounded easy enough: my data drivers are /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde, so I simply first checked what the starting sector of each partition was, using:

# fdisk -c -u -l /dev/sdb (and for c, d, and e, too).

This showed that the first partition started on block 2048. Fair enough: I ran

# fdisk -c -u /dev/sdb

deleted (d) partition 1, created a new partition 1 with a new end block (higher than before), and set the type to Linux Raid Autodetect (fd).
I did the same on /dev/sdc, /dev/sdd, and /dev/sde, too.

Of course, since the array (/dev/md0) was still active and mounted, the kernel refused to re-read the partition table. That was fine: a reboot would solve that.


I rebooted, and to my dismay I found out that the array was no longer recognized! It turned out that my little fdisk adventure removed the RAID superblock, and I did so on ALL RAID6 members. That is slightly problematic: having even as little as one superblock still available is enough to use "mdadm --examine --scan" to get things up and running again, but I had NONE left.

You can imagine my sinking feeling as I realised that I might just have lost 1.2 TB of private data... What to do? All options of mdadm --assemble would not work, for lack of superblocks, and completely recreating the array would destroy all data, right?

Right?...

Wrong! It turns out that mdadm has a few nice cards up its sleeve... If you create an array with the bare minimum number of devices (N-2 for RAID6, N-1 for RAID6, 1 for RAID1), there is nothing to sync, and mdadm will not do so. Now, in RAID6 (and RAID5), the order of the devices is important (because of the data/parity-block rotation), so with my bare minimum of 2 devices, I made a list of all possibilities. If you call the two devices /dev/sd${X}1 and /dev/sd${Y}1, I had the following possibilities:

X Y
----
B C
B D
B E
C B
C D
C E
D B
D C
D E
E B
E C
E D

For each of these combinations, I ran:

# mdadm --create /dev/md0 --verbose /dev/sd${X}1 /dev/sd${Y}1 missing missing

(note the two missing devices at the end)

If that succeeded, I tried to run a filesystem check:

# fsck.ext4 /dev/md0

For most of the options, the block ordering would be wrong, so fsck.ext4 would not find a filesystem, so I would delete the array again using:

# mdadm --stop /dev/md0

I thus went through all the options, becoming more and more nervous, until the LAST option (seriously!) was right! :-) The filesystem was nicely checked, and then I could mount it, too:

# mount /dev/md0 /data

Of course I was running at the bare minimum of devices now, so I added the other members back in:

# mdadm /dev/md0 --add /dev/sdb1 /dev/sdc1

Linux does this one new device at a time, which takes 10 hours per device (it is 1.3 TB per device). I let it run overnight. When I looked in the morning, adding the first device had succeeded (putting me in the safety of N+1 redundancy already), and Linux was resynching the last device.

That was a close shave, in fact, _way_ too close for comfort! I'll be looking at a hardware RAID HBA next.

Put an IcyBox into my server

Because my server had become a bit of a mess on the inside (4 SATA drives), I bought an IcyBox IB-553SK to at least fit 3 of the drives neatly into the 2 5.25" bays that the server box has.

This looks a lot better now.

Installing Kubuntu/Ubuntu 9.04 on an Acer Aspire 3000

Yesterday, I installed Kunbuntu 9.04 on a family member's Acer Aspire 3000 laptop; my family member was fed up with Windows constantly crashing and being slow.

Things worked out-of-the-box, except for two small things:

1. The display colors were garbled.
2. The wireless network card did not work.

Both were easily solved though:

1. Edit /etc/modules, and add a line containing "sisfb"
2. sudo apt-get install b43-fwcutter

Reboot, é voila, everything works!