Leap second bug in Linux wastes electricity

Rise of power consumption at Hetzner Online's data centres
Source: Hetzner Online

Power consumption at Hetzner Online's data centres jumped by "about a megawatt" in the night from 30 June to 1 July due to a kernel bug which resulted in many Linux systems reaching 100% processor usage in dealing with the leap second added at midnight. The problem is described in an email sent by the web hosting company to customers, asking them to check their systems' CPU usage and, where necessary, to restart their systems in order to restore processor usage to normal levels.

It is unlikely that Hetzner is the only company to have experienced this problem – many recent Linux distributions are affected, so that Linux systems in many other data centres are likely to be running CPUs at 100% usage. The problem is caused by a bug in the kernel code for high resolution timers (hrtimers). Since they are configured using the CONFIG_HIGH_RES_TIMERS option and most systems manufactured in recent years include the High Precision Event Timers (HPET) supported by this code, these timers are active in the kernels in many recent distributions.

The kernel bug means that the hrtimer code fails to set the system time when the leap second is added. The result is that the hrtimer representation of the time taken from the kernel is a second ahead of the system time. If an application then calls a kernel function with a timeout of less than a second, the kernel assumes that the timeout has elapsed immediately after setting the timer, and so returns to the program code immediately. In the event of a timeout, many programs simply repeat the requested operation and immediately set a new timer. This results in an endless loop, leading to 100% CPU utilisation.

John Stultz, who has worked on the kernel's timer code for many years, has been working on a fix for the problem, a third incarnation of which has been available since Sunday evening. He has asked that the patch be merged into the main Linux kernel development tree, where Linux 3.5 is currently undergoing development. He will then work on fixes for previous kernel versions. According to Stultz and a SUSE knowledgebase entry, setting the system time using a command such as

date -s "$(LC_ALL=C date)"

should be sufficient to cure a system without having to restart it. Linux machines which are running at 100% CPU usage as a result of the bug do not therefore necessarily need to be restarted.

A bug in inserting a leap second previously caused problems on Linux systems at the turn of the year in 2009. In that case, the problem was located elsewhere and was subsequently fixed. As Dave Jones reported in a mid June blog posting, a number of kernel developers have been performing testing in recent months to see whether this leap second insertion was likely to cause problems, finding and fixing several bugs in the process.

Update 05-07-12: The graph in this article has been replaced with with another version from Hetzner Online which shows the power consumption measured in watts instead of as a percentage of data centre CPU load.

(fab)