How to take down production with a single Helm command

44 points by hddherman 1 year ago | 16 comments
  • psini 1 year ago
    Tangentially related, cannot help but be reminded of the "Ohnosecond": https://en.wikipedia.org/wiki/List_of_humorous_units_of_meas...
    • silisili 1 year ago
      I've seen enough of these over the last two decades, between Ubuntu/Debian and RedHat, that I use and inherently trust other distros more. Just compile the source packages. If there is some issue, file a bug. Monkey patching everything silently leads to things like this, which I've never experienced with distros like Arch, Void, or Solus.
      • sneak 1 year ago
        Debian is very very diligent about patching out phone-home and expiration timers and other such “developer knows best” misfeatures. Some apps also include autoupdate (aka solarwinds style RCE) which they also remove.

        Overall they do way more good than harm. Trashing them because they sometimes make mistakes is probably incorrect.

        • p_l 1 year ago
          But when they get it wrong, they can get it catastrophically wrong - like breaking ssh-keygen so that for given set of parameters there existed only 2^32 keys.
        • vbezhenar 1 year ago
          For helm and most other golang apps, you just need to download and put binary into your $PATH. They usually put them onto github releases, so it's really low friction way to install a genuine application.

          Now to keep them updated is another story.

          • silisili 1 year ago
            Last sentence. I love Go's install and run story, but keeping it up to date is a pain.

            We trust, or used to trust, distros for this. Why are they messing with the source?

            • xavxav 1 year ago
              My understanding is that they've more or less always done this for various reasons: security patches, compatibility, dependency versioning. I understand the historical reasons that led to this structure for package management, especially with how brittle C dependencies seem to be, but I truly hate this practice. It seems to make it exceptionally difficult for authors of major software to establish any sort of invariants or security boundaries.
        • ofrzeta 1 year ago
          That's a good argument for not running IaC tools on your own workstation but on a dedicated machine where the environment is also somehow vetted.

          On the other hand I am not sure how to actually vet all used tools. For instance if you take Helm there are many pointers to repos for various Linux distros and other OSs: https://helm.sh/docs/intro/install/

          Who to trust? While the Helm page says "members of the community" are in charge of the Ubuntu repo the Fedora repo is called an "official repository". Presumably judging from the OP that means "official Fedora repo" opposed to "official Helm repo".

          So essentially there are no repos of the Helm maintainers which makes installing and auto-updating it on any Linux distro virtually impossible.

          • adrianmsmith 1 year ago
            Explanation here: https://github.com/helm/helm/issues/12681#issuecomment-19593...

            Looks like it's a bug in Helm, but actually isn't Helm's fault, the issue was introduced by Fedora Linux.

            (This issue is linked from the article, but it took me some time to find it.)

            • olabyne 1 year ago
              why would someone use Fedora in production, tho ?
              • vbezhenar 1 year ago
                helm runs on the operator machine, not server. While fedora server might not be very popular, fedora workstation seems to be pretty popular.
            • rad_gruchalski 1 year ago
              > Apparently the version of Helm packaged in Fedora Linux included a patch that introduced this issue

              Let's translate: how to take down whatever by living on the bleeding edge.

              • readthenotes1 1 year ago
                And increase the bloodshed by, apparently, configuring your staging environment significantly different than your production environment. An instigating the bloodshed by doing something first in production and then testing it in staging (sure, the first time staging wouldn't have caught it, but maybe then you could look to see where you misconfigured staging...)
                • number6 1 year ago
                  Fedora is hardly bleeding edge
                  • zer00eyz 1 year ago
                    I always thought of Fedora as QA for red hat.
                    • rad_gruchalski 1 year ago
                      Apparently bleeding enough in this case.
                  • 1 year ago