Hetzner Cloud – Data loss incident (redux)

21 points by asgrdz 1 year ago | 6 comments
Dear Customer

We regret to inform you that there has been a loss of snapshots from your following Storage Box that were created in the period from 15 May 2023 11:00 CEST and 12 July 2023 13:00 CEST. Snapshots created before or after this period and the contents of your Storage Box are not affected!

On 25 July 2023, we discovered that due to a software error, snapshots created for Storage Boxes were partially deleted. This affected both the snapshots created manually (via web interface or Robot Web Service) and those created automatically as part of snapshot plans. Unfortunately, we were unable to restore the snapshots deleted during the above-mentioned period.

The protection of your data is important to us. We take this matter very seriously and have already rectified the problem and immediately reported the incident to the Bavarian State Office for Data Protection Supervision.

We apologise for this inconvenience and ask for your understanding.

If you have any questions, please do not hesitate to contact us. Please use the contact form in your administration interface under the menu item “Support”.

Kind regards

Hetzner Online ---- * Previous incident, 15 months ago: https://news.ycombinator.com/item?id=31015840

  • pizzafeelsright 1 year ago
    Data loss is terrifying. I am building a tool to help in data transfers.

    I always assume my data is more secure in S3 than my on prem NAS. I also assume that 0.00001% accounts for anything from one file to an entire data center worth of bytes.

    • nik736 1 year ago
      This is bad, but storage boxes are already a backup target and the content of the backup target (actual files stored in in box) are not affected... "only" snapshots.
      • mydriasis 1 year ago
        Almost a month of snapshots. Yikes.
        • whatsthatabout 1 year ago
          you mean almost two months…
          • elliewithcolor 1 year ago
            Apparently only if you changed the config and also only snapshots that are older than 14 days.

            Still Bad but less impact than described in their first email. (Asked support)

            • mydriasis 1 year ago
              Eek! Even worse!

              And now I know I can't count!