Leap second: Linux can freeze

The leap second that was inserted on Saturday night can cause permanent high CPU loads on Linux computers. Among other examples, the behaviour is documented in the blog of the Mozilla Foundation, where strange peak loads on one Mozilla server were observed from the time the leap second was added. The developers said that the problem is easily solved by resetting the date or rebooting the system.

The root of the behaviour is located in the Linux kernel, where a leap second that is triggered by the NTP subsystem results in a deadlock situation. The problem appears to affect all kernel versions from 2.6.26 up to and including 3.3.

The current release of the Debian Linux distribution ("Squeeze") seems to be affected by the problem in a similar way to the Mozilla servers. In a post mortem analysis at serverfault.com, the authors write that several computers at a data centre no longer responded to pings and displayed a blank screen after the leap second was inserted. System administrators were reported to have saved the day by stopping the NTP daemon and executing a Perl script that reset the leap second bit in the kernel.

Initially it was believed that this problem was related to a similar leap second bug that occurred in the Linux kernel in 2009; that is not the case, however. A developer by the name of John Stultz, who is involved with the development of the timer functions in the Linux kernel has now analysed the problem and is working on two patches.

In view of the recurring leap second disruptions in its own server farms, Google took the motto "prevention is better than cure" to heart and established a "leap smear" process: using modified NTP servers, Google ensures that every NTP update on the day of the leap second inserts a few milliseconds that add up to a second by the time the leap second is actually inserted. Such minor system time differences are usually tolerated by operating systems.

Update - The leap second issue is also causing data centres to use massive amounts of extra power. Details about this and fixes are in a new story on The H.

(fab)