A bizarre linux "bug", what happens when UUIDs are not unique!

[richard.cs]

Short story - I did something weird and things broke. Still, having fought with this for the last month and finally realised what was going on I thought I would share it.

So, I had a slightly elderly Ubuntu system (12.04, a long-term support release) that was nagging me to upgrade. Ok thought I, I'll do a backup first because I've had poor upgrade experiences in the past. I have two 10 GB partitions, one with the active system on and one spare that tends to contain some older release so I booted a live CD and DD'd my currently active partition to the spare one. Right, I've used this method before, I've got a backup, run the upgrade, it worked and everything was peachy.

I noticed a day or two later that sometimes the old OS install (which now only exists on the spare partition) booted. The problem seemed to occur at random, graphics were a bit broken but I could always ctrl-alt-F1 to a terminal and reboot safely allowing me to keep using this somewhat annoying system and try to track down what I assumed was a grub problem (perhaps I'd confused it's OS list generation with my backup method). No luck. I put up with this for an month and today it was being really annoying. On the "broken" install I noticed /boot (which I had previously renamed in a failed attempt to stop this system loading) had re-appeared. On finally getting into the right OS I noticed GParted listed both /dev/sda6 and /dev/sda7 mounted as root. df thinks only /dev/sda6 is mounted. I mount /dev/sda7 on a temporary directory in /mnt and sure enough, if I write a file too /mnt/old_partition/test.txt is appears there and as /test.txt - this really isn't good. I check /etc/fstab and there is only one entry for / (good!). Still, they're listed by UUID, a nasty thought occurred...

The output from blkid:
/dev/sda1: UUID="34002D4C002D1700" TYPE="ntfs"
/dev/sda5: UUID="9682d8ab-c10c-4ea3-9b79-b9daa46087c8" TYPE="swap"
/dev/sda6: UUID="ba55c6cb-e9fb-4f6f-98fc-a3877d4293bf" SEC_TYPE="ext2" TYPE="ext3"
/dev/sda7: UUID="ba55c6cb-e9fb-4f6f-98fc-a3877d4293bf" TYPE="ext3"
/dev/sda8: UUID="599818de-d6ac-4145-8a0b-2002441d0c2a" TYPE="ext3"
/dev/sdb1: UUID="c7485de2-21ea-44ca-a3ae-9493ad4ef4c3" TYPE="ext3"
/dev/sdc1: UUID="0dc109f7-ff37-40d0-9297-fefd19567502" TYPE="ext3"


Oh dear, two partitions with the same ID. It would seem both are being mounted as root but which system actually loads is the outcome of some kind of bizarre race condition. It really shouldn't be possible, right? Anyway, I'm about to try and fix it, wish me luck!

[mrflibble]

To avoid such fun in the future, take a look at LVM. Instead of shuffling partitions and dd-ing you can clone logic volumes without any risk of duplicate UUID's. Oh and a bunch of other handy features as well.

As for how to fix your current situation, wilfred's tune2fs suggestion should do the trick. Also, man blkid.

[richard.cs]

I think I've got it fixed now by changing the UUID of the unwanted partition.* I've thought for a while that using dd in that was was related to the problems I was having but I only realised the UUID thing this evening. What I found interesting was the system behaviour under those circumstances, directories seem to have got merged in some cases, it was unclear what determined which version actually loaded, etc. Essentially I would have expected it to always pick the same one (computers being deterministic after all), and the "two partitions both mounted at /" thing was really unexpected.

Thanks for the suggestion about LVM.

*I actually changed the wrong one first of all and had to dig out a knoppix cd to fix it. 

[mrflibble]

*I actually changed the wrong one first of all and had to dig out a knoppix cd to fix it. 

That's traditional! 

[SirNick]

Are your boot scripts run in parallel?  If that's the case, it won't be deterministic at all.

Just thinking out loud here...  There might ought to be a safety check for mounting multiple volumes with the same UUID (the only real "bug" evident here), but mounting over root is common during boot-up, so that in particular shouldn't (....) cause FS consistency problems.  I wonder if maybe the second volume was being mounted after files were already opened on the first.  I don't know what would happen in that case.  Linux is awfully clever about surviving cosmetic changes to a file while it's open in another process.

Interesting find. 

[richard.cs]

Are your boot scripts run in parallel?  If that's the case, it won't be deterministic at all.

Just thinking out loud here...  There might ought to be a safety check for mounting multiple volumes with the same UUID (the only real "bug" evident here), but mounting over root is common during boot-up, so that in particular shouldn't (....) cause FS consistency problems.  I wonder if maybe the second volume was being mounted after files were already opened on the first.  I don't know what would happen in that case.  Linux is awfully clever about surviving cosmetic changes to a file while it's open in another process.

Interesting find. 

I would have expected that a mount-by-UUID line in fstab would only mount one partition and the same one each time. That doesn't mean everything would work but the problem would be more obvious and the chances of breaking something rather less. Clearly that's not what happened suggesting that either 1) someone considered that was desirable behaviour or 2) no-one considered the possibility of duplicate UUIDs. I suspect the latter, if you explicitly wanted the behaviour you could put it in as two lines.

I am curious about the seemingly random selection of which system actually loads.