Backblaze S3 API is having some downtime

32 points by doctorfoo 4 years ago | 24 comments
  • atYevP 4 years ago
    Yev here - we're investing the S3-endpoint issue - apologies for delay in response here, it took them a bit of time to wake me up.

    *Edit -> we've identified the fix and are rolling it out, you should see S3-Endpoint service resuming shortly!

    • mahesh_rm 4 years ago
      Happy to know who to bother the next time it happens! :P EDIT: But really, get a simple b2s3api status page up, doesn't need to be anything fancy..
      • atYevP 4 years ago
        Well, hopefully there won't be a next time - but also, bothering me doesn't help debug or resolve the problem, our Ops/Tech team do that :D
        • OJFord 4 years ago
          I was baffled that I couldn't find a status page at all, but I assumed I was just failing to find it.

          > Well, hopefully there won't be a next time

          Are you seriously saying that not only is there not one, but you're against having one because you hope the status is always green?

      • doctorfoo 4 years ago
        Thank you, and I do love your service. It would be nice to see a post mortem on what went down. /goes to sleep
      • nhoughto 4 years ago
        Looks like Europe is down too?

        https://s3.eu-central-003.backblazeb2.com/

        • koreanguy 4 years ago
          https://bgp.he.net/dns/backblazeb2.com#_website

          This site can’t be reachedCheck if there is a typo in backblazeb.com. If spelling is correct, try running Windows Network Diagnostics. DNS_PROBE_FINISHED_NXDOMAIN

          • nhoughto 4 years ago
            But non s3, b2 api is up

            https://f002.backblazeb2.com/file/blah

            https://f003.backblazeb2.com/file/blah

            • akchou 4 years ago
              Today's task: starting working on moving back to aws. Storage savings cant compete with the cost of developer time and downtime even when we store a petabyte.
              • OJFord 4 years ago
                Yes! Glad it's not just me. And no status page. (?!)
                • jsiepkes 4 years ago
                  I also can't find a status page and not so much as a peep from Backblaze anywhere. I pinged them on Twitter [1] to no avail. I like Backblaze but this has been going on for 3 hours now. You can't be down for 3 hours without any communication about it...

                  [1] https://twitter.com/JasperSiepkes/status/1329347119774375937

                  • OJFord 4 years ago
                    From the support page, I get the impression nothing will happen until 9am Pacific. Tickets have a 1d response time, and chat is Pacific working day with a lunch break.

                    Bottom of https://www.backblaze.com/help.html

                    I think they're all sound asleep, blissfully unaware of the day they'll wake up to!

                    • slice-beans 4 years ago
                      It doesn't reflect well on their monitoring/on-call setup that everyone else was paged before them, assuming they are indeed unaware.

                      Perhaps the first they'll know of the problem is when they read it on HN

                    • doctorfoo 4 years ago
                      yup, here desperately migrating to the regular b2 API...
                    • zhan_eg 4 years ago
                      Well, it looks like it is in the time frame of their scheduled maintenance https://www.backblaze.com/scheduled-maintenance.html -- Disregard, I got AM and PM confused :(
                      • OJFord 4 years ago
                        No, I saw that, but that says 3 pm not am.

                        Pacific time is several hours behind me in the UK and it's 11am here.

                        I don't think it was even Thursday yet Pacific time when this started.

                        • jsiepkes 4 years ago
                          Can't be because this has been going on for over 3 hours now. The timeframe is only 1 hour.
                      • mahesh_rm 4 years ago
                        Argh. I believed I was going insane. How can this be the only B2 status page?
                        • chima3ra 4 years ago
                          well, what can u expect , 4 times cheaper than AWS, 4 times less stability and professionalism.

                          I am now wandering if my data is safe there. I think it will be smart to keep a copy in aws glacier.

                          • doctorfoo 4 years ago
                            Feeling kind of the same... I just completed moving my entire app and all user data from AWS. And here we are!
                          • manojlds 4 years ago
                            How prominent is usage of Backblaze and the S3 API in specific? Is HN going to be status page for any random service?