Stable Linux kernel hit by ext4 data corruption bug - Update

Linux kernel developer Theodore "Ted" Ts'o has released a series of patches for what he has called "a Lance Armstrong bug" in the kernel, meaning behaviour that does not trip up tests but nevertheless makes the kernel work differently than intended. A user had reported a problem that caused data loss; the kernel developers quickly narrowed this down to a fault in the ext4 implementation that was introduced with the release of Linux 3.6.2 just over a week ago. Apparently, the data corruption bug was hard to track down as it only manifests itself if a system is rebooted twice in a relatively short period of time.

The bug only occurs when the filesystem's starting block is zero. This will cause the kernel to truncate the journal when the filesystem is unmounted. This situation arises if the filesystem is mounted and unmounted so quickly that the journal log does not have a chance to be written completely. The first time this happens, the ext4 driver can recover the journal which will not lead to any ill effects. Should the same situation arise twice, data from the newer mount session will get written to the journal before data from the older one, leading to metadata blocks that "can end up getting very scrambled indeed", according to Ts'o.

The kernel developer cited users who shutdown and boot their laptops instead of hibernating them as a possible group which could be hit by this bug, leading to data loss as described by the original reporter of the problem. Ts'o apologised to affected users for not having spotted the bug in the original commit and released a patch. When another kernel developer doubted that the patch completely fixed the problem, Ts'o published a second follow-up patch.

Beyond affecting Linux 3.6.2 and later versions, the problematic commit was also backported to the stable 3.5.x branch of the kernel. It is currently not known when it will be fixed in that release, although the Linux developers are working to resolve this as well.

Update 25-10-12:

Theodore Ts'o has continued his investigation of the bug and has found that the problem was more esoteric than was first thought. The user who reported the problem was using umount -l, which immediately unmounts the filesystem without waiting for it to stop being busy. The bug is now thought to be caused when the machine is being shut down while it is in the process of unmounting the filesystem with an already compromised journal.

The developers are still working to pinpoint the exact problem and it might actually involve more kernel components than just the ext4 drivers. In any case, it has become clear that the bug needs a very specific configuration to surface and is unlikely to affect most users.

(fab)