How a leap second caused havoc with the Internet (2021)

40 points by frantic2821 1 year ago | 12 comments
  • nytesky 1 year ago
    A couple years ago we had issues caused by a 0 (zero) leap second, which honestly I couldn’t find documented anywhere but hacker news! Something about the system expects a leap second every 17 years and if it doesn’t get one it’s more troublesome than the 0 leap second.

    https://news.ycombinator.com/item?id=27944776

    • gertrunde 1 year ago
      One of the more unseen side-effects of this particular bug back in 2012 was datacenter internal temperatures spiking by ~5℃ due to the increased CPU usage iirc.
      • ahazred8ta 1 year ago
        We've already discussed the Leap Smear, where clocks run 0.0012% slower for a day.

        https://news.ycombinator.com/item?id=28047714

        https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

        • zokier 1 year ago
          And inevitably there is also Leap Unsmear: https://news.ycombinator.com/item?id=22223591

          By Google. Who started this whole trend of leap smearing in the first place.

          • tinix 1 year ago
            Google didn't start this... It's standard practice to speed up or slow down the clock during transition in ERP systems.

            VAX/VMS are way older than Google.

            • ahazred8ta 1 year ago
              Actually, before 1972 they would run the radio time signals several parts per million fast or slow to keep in step with the earth's rotation.
          • gregw2 1 year ago
            I recall this hitting our startup’s various MySQL Linux servers in 2012.

            They were slow/maxed out for a while till we figured it out. The DBA rebooted a few but didn’t know why and it kept cropping up on others; I drilled into root cause with some google-fu.

            Not mentioned in the article is that I think this issue occurred again in 2015. I don’t recall if it hit us because we hadn’t patched/upgraded some servers since 2012 (!) and had just relied on resetting the clock/rebooting, or there were more Linux bugs (see LWN below).

            Good technical writeup from Linux weekly news: https://lwn.net/Articles/504744/

            And more on Linux patching in 2015: https://lwn.net/Articles/648313/

            IANA canonical list of leap seconds: https://data.iana.org/time-zones/tzdb-2018a/leap-seconds.lis...

            • ooterness 1 year ago
              > Hany explains: “Due to physics, at certain times of the year, the Earth moves faster along its orbit than others. How do we base time off of this, then?”

              Sorry, but no. This is so wrong I don't know where to start.

              • zokier 1 year ago
                It's not wrong really, Earth's orbital velocity does indeed vary. It just doesn't have anything to do with leap seconds
              • 1 year ago
                • frantic2821 1 year ago
                  TLDR: a leap second overloaded computer systems due to issues in how Linux handles time.
                  • ripe 1 year ago
                    Time to plug my old explanation of leap seconds:

                    https://robotsinplainenglish.com/e/2022-11-20-stopwatch-time...

                    • zokier 1 year ago
                      > As you can see, UTC is not a proper, continuous time stream. It has breaks in it wherever leap seconds have been added

                      I can not see that. 23:59:58 - 23:59:59 - 23:59:60 - 00:00:00 is perfectly continuous, proper, and even monotonic, or at least as much as 23:59:58 - 23:59:59 - 00:00:00 is.

                      > When you receive a package with a computer-generated time stamp, it is showing this fudged UTC number, not the actual number of seconds according to the atomic stream.

                      I think you are mixing up UNIX timestamps (which are their own thing) and UTC. UTC and TAI both indicate exactly same thing, there is perfect 1:1 mapping between UTC timestamps and TAI timestamps (ignoring pre-1972 era).