Login Register

Vodia News: Leap Second

Date: 7/1/2015

July 1st was a remarkable day. Not only because of the things happening in Greece, but also because of a widely unnoticed other event: The leap second.

Most people know what a leap year is. Every couple of years, February has an extra day, so that summer stays hot and winter stays cold (or the other way around if you happen to live in the southern part of the planet). A leap second is something similar. Depending on ships, airplanes, other planets and all sorts of stuff moving on or around the earth, the spin of the earth is not always exactly the same. In order to keep noon where it should be (in the middle of the day), people need to insert or remove a second every now and then. Well, that was what happened yesterday: We had one more second.

From a sleep deprivation perspective, I thought it would not matter much. But it did, in a way I did not anticipate.

When we upgraded our server yesterday, the registrations did not come up as they should. At first I thought that some new feature has caused the hiccup. Most of you know the risk up upgrading a system that otherwise works well. But this time, there was really only a small change that could not have caused such a major problem. It turned out that the internal clock for callbacks got screwed up, and that they came a lot earlier than they should. So the registrations timed out before they had even started. Not good. A couple of restarts of the PBX it turned out, the problem is not inside the PBX; it must be outside: The operating system.

We are using Linux on that server, and I have to admit it is not the latest one. I remember listening to the radio yesterday when they were joking about what they would do with the extra second, started my search engine of choice and soon found out that we are not the only one with problems. In desperation, and after 730 days uptime, I decided to reboot the operating system. And voila, the problem was solved. Unfortunately it took too long to get my usual amount of sleep.

Now the morning after we get a better picture on what happened. It was not only for me a sleepless night, it was also for others. Those who got my email yesterday should have saved a lot of time for troubleshooting and kept their services running smoothly. But I got emails back that said that phones also had similar problems. After all, they also run Linux.

I am currently reading a book about the future of crime. They are talking about IoT, robots and all sorts of stuff; of course the worry is that this part of the infrastructure can be attacked and cause big harm. Now thinking about such a minor glitch like the time being offset for a second, and this can already cause devices crap out? What if someone is able to change the time servers and add ten seconds? I thought that the monolithic time was not vulnerable to that, but it seems to be. It makes me a little worried.

I hope that keeping the operating system fixes that vulnerability. We have to check how we can upgrade our server without major service interruptions to make sure that when the next leap second comes or even someone purposely, we don't have the same effect.

So if you experience problems with your servers, check if your Linux is older than March 2012 (see http://www.wired.com/2012/07/leap-second-glitch-explained/): short-term fix is to reboot the system, mid-term fix is to upgrade the OS.


Your Vodia Team

© 2018 Vodia Networks Inc. All rights reserved.