Upgrading to FreeBSD 9.x with a whole-disk gmirror

This post is very old and likely contains information that is no longer accurate and links which no longer work. Proceed with caution.

My FreeBSD desktop workstation has been on 8.x since I built it just over two years ago. When I made it, I bought two 1TB HDDs and setup a RAID-1 volume with gmirror, following what was at the time the recommended instructions from the FreeBSD Handbook After delaying the upgrade to 9.x repeatedly over the last several months I finally decided to give it a try after 9.1-RC1 was announced. Read on for how it went wrong, since FreeBSD 9.x more correctly handles partition integrity checks.

I also used it as an opportunity to switch my /usr/src tree over to Subversion instead of CVS The buildworld and buildkernel went fine (After I re-created my kernel config file that I accidentally removed when clearing out /usr/src to move to svn!), and rather than my usual tactic of doing the installworld in multi-user mode, I decided to be cautious and boot the 9.x kernel into single user mode like we’re supposed to. It’s a good thing I did!

Upon booting, I was greeted with two fun things:

• GEOM_PART: integrity check failed (mirror/freebsd0, MBR)
• A mountroot> prompt.

So I was a little panicy at this point, because I have about 700GB of data on this array. Some Googling around later on my laptop and I found out about 9.x’s stricter integrity checks that don’t play nicely with a whole-disk gmirror made using the handbook’s instructions. As this mailing list post indicates the real solution is to switch to doing a per-slice mirror using GPT rather than the old MBR/glabel way. Since that mailing list post was a little light on details, I kept searching and found an excellent article by Vick Khera (who, incidentally, also contributes on the pfSense mailing list occasionally!) that steps through the whole process in an easy-to-follow manner.

Much of that article was spot on but there were a few differences for me, mostly because of the differences in our setup – his was a server in a datacenter with no local access, and mine was a desktop sitting right in front of me. The differences for me were:

• I had to make sure I was booted from a 9 kernel+world to do gpart properly (9.1-RC1 memstick works great for this) – I initially discovered the gmirror issue after trying to single user boot after only doing an installkernel, so my world+kernel did not match when booting the 9.x kernel. If I tried gpart in this state, it did not function.
• I had to set kern.geom.part.check_integrity=0 before I did a kldload geom_mirror on 9.x or it wouldn’t touch the mirror, claiming it was corrupt. I was able to access the mirror fully with this sysctl set, and it’s possible I may have been able to function fully indefinitely this way, but part of me kept thinking it was a ticking time bomb and there was no better time than now to fix it properly.
• If you do the partitioning and dump/restore from the fixit boot environment, you don’t have to worry about rebooting or fixing fstab in the middle of the process, you need only fix fstab at the end.
• If you have a large filesystem and do this from fixit, you’ll need to mount a bigger /tmp than the fixit system provides. For me I was able to dump/restore my /tmp slice, then forcefully unmount the mfs /tmp, and mount the ‘old’ /tmp as the fixit /tmp - this allowed the dump/restore on my 700+GB /usr to complete without much fuss
• Doing the whole process from fixit did make for some slight variances in the process, mostly pointing dump at the (old) mirror slices directly rather than mount points and not using L since it wasn’t mounted.

The dump/restore phase took several hours for the large /usr slice, I don’t have an exact count since I let it run overnight and didn’t note the start time. There were a couple of ATA errors but it worked past them. (I ordered a spare disk just in case this is a bad sign.) The next day after the mirror was back up, the mirror synchronization also took a few hours, but I had left the house and it was done by the time I returned.

The end result is a much more modern disk layout, and mirrored partitions rather than a completely mirrored disk. It won’t be quite as convenient to rebuild when a disk fails, but I also won’t have to worry about overwriting metadata with real data. I do need to keep a record of my slice ordering and the commands to re-create it somewhere.

Now my desktop is happily running FreeBSD 9.1-PRERELEASE, and is currently busy recompiling all 1132 of its ports. Hopefully when the time comes the transition to 10.x will not be quite so painful.