Ask HN: What's behind all the UK IT failures this month?

26 points by 0898 1 year ago | 50 comments
This month we've had McDonald's, Sainsbury's and now Greggs forced to shut temporarily because their payment systems have gone down.

Surely these can't be unconnected? Can anyone shed any light on what's happening here behind the scenes?

  • nonrandomstring 1 year ago
    Politicisation of technology.

    It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.

    That, plus a mixture of crony contracts and bad project management.

    We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.

    A telling remark by a Labour politician to the question "What is the greatest concern about a 'cashless economy' was "Making sure people are not left behind".

    In other words, we're forcing this on people whether they want it or not, and whether it works or not. The idea that there might be "risks" or necessary safeguards was simply not conceivable.

    That's what happened with the Post Office Horizon system and it's what's happening with other new systems pushed out "for our own good"

    That's not a remark against technology or modernisation, it's a criticism of bloody-minded recklessness and anti-democratic hubris.

    Good technology requires care. We talk about it a lot here [0].

    [0] https://cybershow.uk

    • bilekas 1 year ago
      > Politicisation of technology. > It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.

      > That, plus a mixture of crony contracts and bad project management.

      > We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.

      That is a LOT of VERY sweeping statements to a pretty specific question it seems of payment processors, can you give some examples of what you're talking about ?

      • nonrandomstring 1 year ago
        > can you give some examples of what you're talking about ?

        PLease see the podcast linked above where we have spent many hours researching, analysing, writing and presenting on specific cases. In episodes to come we're working on joining the dots, finding common factors in these cases. Enjoy, and please give feedback if you feel able.

    • jjgreen 1 year ago
      McDonalds, Sainsbury, Tesco, barely matters. But Greggs, that's some serious shit depriving people of pies. We need answers.
      • gadders 1 year ago
        The south is organising food drops to the North of England as we speak.
        • jjgreen 1 year ago
          Poncey southern bastards. Still, pies come before pride ...
          • drcongo 1 year ago
            "Pies before pride" would be a great t-shirt.
            • gadders 1 year ago
              Having said this, though, my nearest Kent town has 2 Greggs, 3 if you count a garage on the outskirts of town.

              Even Canary Wharf has a Greggs now.

              • buggeryorkshire 1 year ago
                Pukka Pies can pissoff!
            • 0898 1 year ago
              Is Putin hitting us where it hurts?
              • gertrunde 1 year ago
                Quite probably.

                This could be an attack on national security, after all, Greggs sausage rolls are the fuel that runs the UK police forces, much like donuts in the US ;)

                • abirch 1 year ago
                  As an American an attack on donuts should trigger NATO article 5
                • fifteen1506 1 year ago
                  Please watch "The Undeclared War", on UK-Russia Cyberwar [fictional TV series] so the 2nd season gets produced.
                  • timthorn 1 year ago
                    Hannah Khalique-Brown taught herself Python and C++ as well as playing with IDA Pro to prepare for her role in that.
              • password54321 1 year ago
                Work is getting outsourced to India. Most CS grads don’t know how a computer works. People get hired for reasons other than merit. And ChatGPT…
                • smcl 1 year ago
                  Only 3 comments into a story about a UK problem and here's a comment blaming it on foreigners. Incredible.
                  • soco 1 year ago
                    I take the spirit of that comment that they didn't blame the foreigners, they blamed the outsourcing. Because even if the provider was local the result would have likely been similar (and we do have recent examples, Horizon and not only)
                    • smcl 1 year ago
                      They didn't say "Work is getting outsourced" they said said "Work is getting outsourced to India"
                      • mtmail 1 year ago
                        They gave 4 reasons, and I think all are guesses without knowledge of the IT failures in question.
                      • bilekas 1 year ago
                        A great quote I love.

                        Being British is about driving a German car to an Irish pub for a Belgian beer, then traveling home, grabbing an Indian curry or a Turkish kebab on the way, to sit on Swedish furniture and watch American shows on a Japanese TV.

                        All while being suspicious of foreigners.

                        • alephnerd 1 year ago
                          > Belgian beer

                          Completely unrelated, but I love a classic British ESB. The microbreweries at attempt them here in the US just suck. I miss a nice Spitfire XPA.

                        • 1 year ago
                        • ozzcer 1 year ago
                          Considering chat GPT is a few years old it seems unlikely those who used it in university and are now probably just Junior Developers are responsible for wide spread outages at major companies
                          • dkdbejwi383 1 year ago
                            I worked for one of the affected companies as a rare example of a permanent, in-house engineer.

                            They had a _lot_ of contractors, some independent based in the UK, some contracts with companies with a local presence who would work from the office alongside us, and some in India.

                            All were equally shit at worst, with only the independent contractors proving the exception on occasion.

                            I recall being stuck with “senior” colleagues who couldn’t manage to upgrade a dependency even while being handheld.

                          • DarkNova6 1 year ago
                            All of these enterprises are examples where IT is a "cost center", not a "value center". So you have this conundrum where everything goes smooth and the higher ups ask "why do we spend so much money on this without anything in return?". And when something does go wrong they are likely to repeat said question.
                            • thinkingemote 1 year ago
                              Sainsburys said it was due to an error with "an overnight software update". Perhaps they all use the same software but updated at different times?

                              https://twitter.com/sainsburys/status/1768972295622553900

                              • abulman 1 year ago
                                Maybe all the interns that were left in charge didn't properly do what the AI had said?
                              • i_have_an_idea 1 year ago
                                Random chance and years of underinvestment and tech debt
                                • usui 1 year ago
                                  McDonald's wasn't unique to the UK as it was global. McDonald's Japan was affected.
                                  • jl6 1 year ago
                                    They could definitely all be unconnected. It’s not that suspicious. I guess we’ll find out if they ever publish post mortems.
                                    • phtrivier 1 year ago
                                      Do you have sources about the "outages" ? (did not make the headlines internationally..)

                                      It they all shut down because of "a payment system", then a simple explanation would be a failure of... the same payment system that they all happen to use ?

                                      (It's not like Greggs is going to fully develop a payment system in house without relying on any infra, I suppose ?)

                                      • oakesm9 1 year ago
                                        Sainsbury's and Tesco: https://www.theguardian.com/business/2024/mar/16/sainsburys-...

                                        Greggs: https://www.theguardian.com/business/2024/mar/20/greggs-shop...

                                        The Sainsbury's and Tesco ones happened on the same day (last Saturday), but the scale was very different. Sainsbury's were unable to take card payments nationwide. My local one said it was due to the internet connection in their store not working. It also affected planned deliveries, likely because the scanners they use to fetch and pick the orders didn't have a connection.

                                        Sainsbury's blamed this on a botched "software update".

                                        Greggs happened today and was card payment related but doesn't seem to be nationwide (I've not been to one today so not 100% sure if that's true).

                                      • ksec 1 year ago
                                        McDonald was worldwide. Sainsbury, Tesco and Greggs seems to be related somehow? At least they are all payment related.
                                      • damvigilante 1 year ago
                                        Mandatory cert rotation forced by a dependency that had a leak, and they all didn’t handle it properly. (Just a guess)
                                        • dsattt 1 year ago
                                          Probably leetcode.