Leaving the Basement

80 points by timf 2 years ago | 50 comments
  • dang 2 years ago
    Recent and related:

    Post mortem on Mastodon outage with 30k users - https://news.ycombinator.com/item?id=33855250 - Dec 2022 (101 comments)

    (Offtopic meta note: Alert users will note that that thread was posted later than this one. This is because the second-chance process (https://news.ycombinator.com/item?id=26998308) has a race condition: the events "story makes front page" and "moderator puts story in second-chance pool" sometimes diverge and can happen in any order.)

    • Enderboi 2 years ago
      Ah yes.

      I've also seen NFS/ZFS on Linux have very... bizzare... issues with locking, latency, and poor handling of errors bubbled up from the block layer taking down clients or even the host.

      All of these went away when we redeployed everything into a Solaris-based distro (still exporting ZFS shares to Linux clients via NFS). It does seem something specific to the interaction of these two components under load on a Linux kernel.

      Unfortunately, it also only happens under real-world production load and was impossible to create reliable test-case with simulated stress tests or benchmarking :(

      • sydbarrett74 2 years ago
        Did you ever evaluate FreeBSD? My hesitation with Illumos and downstream distros is the small number of people maintaining the ecosystem (not that any of the BSD's have huge dev teams by comparison, but still).

        That said, I think OpenSolaris is technically superior in most ways to any of the BSD's.

        • Enderboi 2 years ago
          Yeah, FreeBSD was an improvement with regards to ZFS/NFS integration not having any major issues.

          Unfortunately we had some strange HBA issues with our disk shelves, which went away with an Illumos downstream. Since our use case for this was basically an isolated box that just supplied NFS shares, the limited ecosystem wasn't a major concern more so than stability :)

        • hossbeast 2 years ago
          As someone storing important stuff on ZoL on my desktop, this is concerning to read. OTOH I'm 4+ years in, I've never had an issue, and I have backups (outside of the zfs volume itself).
          • lakomen 2 years ago
            Yeah I've had nothing but bad experiences with ZoL
          • rglullis 2 years ago
            As someone running a commercial provider for Mastodon (and Matrix, and XMPP...), I am somewhat envious of these posts. "Wow, 30000 users! If I had that many users on my service paying the $0.50/month I am charging, it would be enough to pay myself a full salary!".

            But then I realize that they are only getting these many people because they are not driven by commercial interests: even with donations, I can bet they are not collecting enough to keep things afloat and they only keep going because they don't mind spending all this time, money and resources of their own on this project. They can treat it as a (relatively expensive) hobby, and they can keep it running as long as it satisfies them.

            The problem is that I think that this is harmful in the long run. Yes, people now are finally seeing the issue with ad-funded social media. But if we want to have a healthy alternative, we need to understand TANSTAAFL, we need to accept that we need to give real money to the people working on this and to have the servers available 24/7 to store and distribute the hot takes and stupid memes that we so bizarrely crave every day.

            I worry that if we don't change the mindset quickly, the whole Twitter drama would be a wasted opportunity and Mastodon (and the Fediverse in general) will go back to the status quo, where surveillance capitalism is the norm and truly open systems are just a geeky curiosity.

            I wish I could fund a tech-equivalent of the "buy local and organic" campaign. I wish I had more people thinking "ok, I will pay $5/month to this guy and I will bring 10 people to this instance" because it is the ethical thing to do.

            • buovjaga 2 years ago
              • rglullis 2 years ago
                You lost me at "call for moderators and volunteers". Unless these people are actually paid for this taxing and stressful work, I don't believe it can be called "sustainable".
                • abraae 2 years ago
                  Wikipedia would be unsustainable using that test. Yet it's been around for two decades and shows no signs of going anywhere - if that's not sustainable then no web business is.
            • watchdogtimer 2 years ago
              Kris described hachyderm's infrastructure operating in her basement on the Oxide and Friends podcast in mid-November. Kudos to her for being able to keep it going there so long!
              • CharlesW 2 years ago
                As someone who'd hoped to start an industry-focused instance, I found Kris' Medium articles about Hachyderm's growth really interesting: https://medium.com/@kris-nova

                The latest post, "Yelping: Action Through Criticism", includes links to additional Hachyderm-related content near the top, then talks about they handle an obnoxious online behavior they've experienced because of the recent popularity of Hachyderm.

              • cyberpunk 2 years ago
                > "We can then leverage Mastodon’s S3 feature to write the “hot” data directly back to Digital Ocean using a reverse Nginx proxy."

                How does that work?

                • josteink 2 years ago
                  Digital Ocean offers S3 compatible storage.

                  I guess they use Nginx to reroute traffic which by default is targeting aws.amazon.com?

                  • evankanderson 2 years ago
                    I think the default is "files-off-disk" image serving (hence NFS, etc). This can be replaced by S3 storage, with the URLs remaining the same (or maybe redirected, if you don't want to pay the bandwidth bill twice).

                    Disclaimer: I have this setup using Humanmade/S3-uploads for WordPress, but am not currently running Mastodon.

                • convolvatron 2 years ago
                  "In other words, every ugly system is also a successful system. Every beautiful system, has never seen spontaneous adoption." - not only is this logically fallacious, its pretty offensive about the general notion of software quality
                  • nmyk 2 years ago
                    Where's the fallacy? She's saying the only systems that are beautiful are the ones that haven't been forced by massive spontaneous adoption to scale faster than the developer(s) can come up with and implement a beautiful design to meet the new requirements.

                    Maybe you think if the original system were _really_ beautiful and of high quality, it would have scaled with the adoption on its own, with no need for ugly patches... but in that case the original system would have had the capacity to do a lot more than what was originally required. It would have been overengineered, in other words, and it would have been more beautiful if it had met its original requirements more cheaply.

                    The notion that a sudden change in requirements that must be dealt with quickly results in an uglier system seems fairly straightforward to me, and certainly not offensive.

                  • bluedino 2 years ago
                    Why weren't the disks just replaced?
                  • lakomen 2 years ago
                    If Mastodon is the new cool, where all the climate activists hang out, why does it use node and not Go, Rust or C++?
                    • musk_micropenis 2 years ago
                      I would like to understand why Mastodon requires such a huge amount of hardware for mediocre traffic volumes. Not just the lazy "it's Rails" answer - I know Rails is a resource hog, but that doesn't go far enough to explain the extreme requirements here.

                      As a point of reference, look at what Stack Overflow is run on. As a caveat, SO is probably more read-heavy than Mastodon, but it also serves several orders of magnitude more volume (on a normal day in 2016 they would serve 209,420,973 HTTP requests[0]). They did this on 4 DB servers and 11 web servers. And in fact, it can (and has) worked serving this volume of traffic on only a single server.

                      With this setup SO was not even close to maxing out their hardware (servers were under 10% load, approximately). SO also listed their server hardware[1] in 2016. I don't know enough about server hardware to assess the difference, but to my eye they look similar on the web tier with similar amounts of memory, similar disk, etc.

                      I'm not saying Hachyderm is doing anything wrong, but it makes me wonder if there's a fundamental problem with the design of Mastodon. And to be clear I understand that this particular issue was caused by a disk failure, but that they even had this hardware in place running Hachyderm is surprising to me.

                      [0] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

                      [1] https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...

                      • BeetleB 2 years ago
                        I would like to point out that you asked this same question a few days ago and got several answers:

                        https://news.ycombinator.com/item?id=33855686

                        I would recommend people read that thread before responding with the same answers.

                        • rglullis 2 years ago
                          Every post and every reply generates a request to all servers you are federating to: https://aeracode.org/2022/12/05/understanding-a-protocol/
                          • spankalee 2 years ago
                            This ultimately polynomial direct-connection approach is clearly fatally unscalable.

                            The fediverse needs to figure out a hubs-and-spokes or supernodes pattern so that service providers can scale up syncing, indexing etc.

                            ie, my personal instance should be able to offload most of the message passing to an supernode intermediary that lots of other instances use for federation so that my instance only needs one connection, and the supernodes only need to connect to each other and their local network.

                            • rglullis 2 years ago
                              They do have it for some things already. There is this concept of "relays", which you can use as a feed of the data from the larger instances. But AFAIK it's used only as a content source and it's not something that you can set up now as a way to help with scalability.

                              I am also closely following https://github.com/nostr-protocol/nostr to see how they go along, because I am growing weary of the "tech elite" that is moving to Mastodon and is pushing for "moderation by committee". I've gotten myself with discussions already with people who actually want server operators that want only to open federation for those that abide by some "Covenant". This seems rooted in good intentions, but it reeks of something that might lead to a corporate copout of a network which is supposed to be open.

                              • masukomi 2 years ago
                                > This ultimately polynomial direct-connection approach is clearly fatally unscalable.

                                Is it? I mean, let's assume you post something popular, and 17,000 servers request it. How many people do 17,000 servers cover? Like even if we assume only 10 people per server. That's 170,000 people. How many people have 17,000 followers, nevermind 170,000 ? And 10 per server seems implausibly low for an average.

                                17,000 hits is... not particularly notable from a server perspective, triply so when they're all requesting the same "just posted" item which is still cached.

                                Sure if like, someone with multiple millions of followers is on your server you're going to have issues, but seriously, twitter had issues with that too.

                                Also keep in mind that there's no algorithm pushing people towards the same "popular" posts. Things grow organically.

                                So, at what point are you suggesting that this becomes "fatal" and is that a point that anyone that isn't hosting literal superstars is going to encounter?

                              • musk_micropenis 2 years ago
                                I guess this goes some way to answering my thought,

                                > but it makes me wonder if there's a fundamental problem with the design of Mastodon.

                                I also note the article says,

                                > During the month of November we averaged 36.86 Mbps in traffic with samples taken every hour

                                That seems like a large amount of bandwidth to service 30,000 users (who knows what fraction of them are actually active at any given moment). But I guess there's going to be a lot of video and image content. I have tried searching all of their linked blog posts about scaling but can't find any number that might map to requests per second without making huge assumptions.

                              • maxsilver 2 years ago
                                > I would like to understand why Mastodon requires such a huge amount of hardware for mediocre traffic volumes.

                                There's some inherent overhead in a federated model (vs a single-source one), and the ActivityPub protocol Mastodon happens to use, wasn't necessarily designed to be the lightest possible thing in all use-cases.

                                Also, there's just a lot more traffic. My instance said, after Twitter's major struggles, they saw something like 30x more traffic and 20x more daily registrations. For instances that, prior to the influx, were running by volunteers in spare time out of people's bedrooms or small cheap VPS's and such.

                                These instances weren't necessarily ideally performance-tuned prior to the influx (and even if yours was, the remote ones your users might need to hit to fetch content from may not have been)

                                • Cyberdog 2 years ago
                                  Interesting username you have there.

                                  I don't see why you don't accept "it's Rails." There are other issues as sibling comments have pointed out, but by starting with an ecosystem known to have performance limitations, this sort of outcome is inevitable, is it not? I'm sure the Mastodon team were never expecting the degree of usage which has been thrust upon its larger instances, but now that it has happened and the limitations have become apparent, I'd encourage people who are interested in setting up fediverse/"Mastodon network" instances to consider the alternatives to Mastodon, however paltry they currently are.

                                  I know that the Pleroma front end and its forks are written in something called Elixir, which I have no idea about but I can't imagine it could be much worse than Ruby. What I'd really like to see is something written in a language known to be actually fast, though - PHP or Lua.

                                  • bscphil 2 years ago
                                    I vouched for this comment because it's a good question, although I'm worried that your account is not long for this world with an inflammatory username like that.

                                    The problem probably starts with the inefficiency of RoR, as you've guessed. Mastodon is a very dynamic site which limits the amount of caching that can be done, and there are hot code paths like filtering streams using a user's block lists and word filters that are not particularly optimized - all this happens in Ruby.

                                    But there are other inefficiencies, compared to SO:

                                    1. Mastodon is a media heavy site, with a lot of uploading by users. Mastodon has to convert user-uploaded media to standardized representations (e.g. JPEG and h.264), which takes a lot of CPU time.

                                    2. Mastodon has a "firehose" feed which is available in the UI and actually used by many users. Filters apply to the firehose feed as well. Obviously this requires quite a lot of bandwidth and processing.

                                    3. Federation is a weakness when it comes to traffic. If user X has an account on server A, and at least one user on 1000 other instances follow user X, server A has to immediately send any posts to all 1000 other instances, regardless of whether anyone on the other end will ever deliberately view them. (Of course, some users may view them in their instance's firehose feed.) The instance then has to duplicate this traffic when sending it to the actual subscribed users. By this standard both large non-federated "servers" (like Twitter) and widely federated pull-only servers (think RSS) are more efficient than ActivityPub (the open standard Mastodon uses).

                                    4. Federation is a weakness when it comes to trust. Instances do not (and must not) fully trust each other, except for things like "@x@thisinstance said 'P'". So for example, the little Open Graph based preview cards you're used to seeing on Twitter and elsewhere have to be generated for links per instance. The first time a Mastodon server sees a link, it must fetch that link and generate a preview card itself. Because new posts by popular accounts are syndicated immediately, this is a burden on websites as well. https://www.jwz.org/blog/2022/11/mastodon-stampede/ (note: copy link or disable sending referrers from HN for this site)

                                    5. Scaling is not really a solved problem yet for Mastodon, because in practice it hasn't had to be. It's easy to pass the buck to instance operators, who end up needing a $20/month VPS to run a small instance rather than $5/month. Even the very biggest servers are scarcely larger than 1M users. At that kind of scale you can patch over performance problems by just throwing more hardware at the problem - and e.g. mastodon.social has the funds from Mastodon (the org) to do that. Note that Hachyderm, AFAIK, is an obvious example of this; it was started by a tech worker in Seattle with much better access to expensive hardware than most casual instance operators can dream of. It's not surprising that they can pull the funds together to scale up before they start seeing performance issues.

                                    • kibwen 2 years ago
                                      In practice #3 is the only one that matters. For reducing dynamism/increasing caching potential, it would be fairly easy to run a fork of the site with the more dynamic features excised (donate $1 a month to get access to dynamic features like filtering). For media transcoding, that's a textbook case of a CPU-bound operation that you could offload to an isolated Rust component for a CPU savings of 99% compared to Ruby (not an exaggeration). But the exponential nature of the network scaling will still kill you despite all this, and needs to be addressed at the protocol level ASAP.
                                      • bscphil 2 years ago
                                        The media transcoding components aren't written in Ruby, it shells out to ffmpeg or Imagemagick.
                                  • imtringued 2 years ago
                                    That basement hardware didn't last long. If you don't know how big your userbase is going to be it would be better to avoid committing money to specific hardware.
                                    • ocdtrekkie 2 years ago
                                      Note that Nova didn't feel an actual hardware capacity level was hit here. However, the setup lacked the redundancy to handle hardware outages for something like drive replacements without a significant outage. And I believe one of the main considerations in moving to a cloud service was actually limited connectivity options, because only so much fiber capacity was even available.

                                      > Our limiting factor in Hachyderm had almost nothing to do with the amount of users accessing the system as much as it did the amount of data we were federating. Our system would have flapped if we had 100 users, or if we had 1,000,000 users. We were nowhere close to hitting limits of DB size, storage size, or network capacity. We just had bad disks.

                                      • lantry 2 years ago
                                        hacker news: you don't need the cloud! you can just run a couple machines in your basement!

                                        also hacker news: why would you try to run something in your basement? Just use the cloud!

                                        • CleverLikeAnOx 2 years ago
                                          There are multiple people here and their opinions vary.
                                          • lantry 2 years ago
                                            There are multiple people here and their opinions generally skew towards contrarian one-upsmanship
                                          • 2 years ago
                                          • nix0n 2 years ago
                                            > committing money to specific hardware

                                            Note that Dell R620 and R630 servers have been discontinued for a couple of years now, were probably bought used, and can probably be re-sold.

                                          • 2 years ago