This completely reverses the typical market dynamics. The company is more focused on catering to investor’s wet dreams than actually solving business problems, and “engineering playgrounds” with 3 different programming languages, microservices, AI & blockchain netting them a 5-figure monthly AWS bill is something that appears to please investors more than a rock-solid, “boring” backend. Maybe the complex architecture succeeds at obfuscating the fact there’s no business model nor plans for one?

I think there is a tension between this kind of actively guiding anti-complexity management and hiring "top talent".

The very best developers are capable, and avoid complexity. The next best developers are capable, and love complexity. The worst developers are not capable.

There aren't enough of the very best developers for a company to plan around hiring only those. So, if you want to hire developers who are at least capable, you have to give them some leeway to make things overcomplicated. Yes, that incurs a real cost to the business. You can think of that as just being another part of their compensation package.

It was a hot time for NoSQL and document DBs. Having investigated using Mongo myself to little avail, I asked why they didn't just use Postgres. If I recall correctly, a couple years later they published a Mongo at Etsy postmortem which concluded they should have just stayed with Postgres.

> The industry constantly mints senior engineers who have been bitten by complexity, but doesn't want to hire them, or listen to them.

Again, in JavaScript land that is not what it sounds like. The industry has minted a bazillion expert beginners who have never moved out of their parent’s basement and had to live on their own (in a technology sense). They are fanatically invented here fearing original code more than job termination and now they have somehow risen to make architectural decisions about the future health of your software and business.

I guess they failed to understand that choosing boring software is different than depending on a package manager to write all your software.

If your engineering team is the one pushing in that direction I'd reckon the company was in a bad spot to begin with to have hired that team because it strongly indicates that the management layer (head of tech/CTO) has no technical clue.

Hire strong Lead Developers with a proven track record of delivering value to companies they worked at and you'll be mostly fine.

Also there's not much to study, in 99% of the cases in a web based startup if your stack deviates from a monolith with one of PHP/Ruby/Python/.NET/Go + Mysql/Postgres/MSSQL you're doing it wrong.

Sounds like the bored engineers need to be allowed to go home early, or have some 20% projects.

Also, as John Gall teaches us with his tounge-in-cheek, yet never-the-less true principles[1] -- a principle so obvious most never give it any thought:

"New System, New Problems"

Can someone please just ask "what do we expect some of the new problems to be?" If you get blank stares and no good answers, then you know they haven't thought it through.

[1]: https://en.wikipedia.org/wiki/Systemantics

For example, you need a search feature. ElasticSearch is big in search, there's lots of article about people implementing ElasticSearch. Very infrequently do I meet people who just starts out with the full text search in their database, or maybe just try something extremely simple, like Sphinx, even if it would solve their problem quicker, safer and cheaper.

It's honestly starting to become a bigger and bigger issue. During the last few weeks I've talked to one customer who is think: Kubernetes. They don't have the funds, staff or need for Kubernetes. What they do need to do here and now is to focus on smarter utilisation of their cloud providers features and reduce the reliance on expensive VMs.

Another customer is going all in and wants to implement all the things. We're talking Kubernetes, Kafka, ElasticSearch and more. They'r are currently running on just a small number of VMs. While their stack does need updating, maybe start smaller?

Had this literally happen to me, but I was a low level manager and this was happening in another team. One thing I've taken from it, at the time the feeling around Sr Management was that we needed to allow this or we would lose the engineers. They allowed it, and after the conversion, those engineers left to start their own company. The remaining engineers had to deal with undocumented OCaml code and keep it running and were resentful.

I have seen this with React vs Vue, where an engineer not liking React, just did his code in Vue. 'We have to let him do this or he'll leave', but he left on his own accord.

Lesson, stick up for your codebase, and if engineers don't like it, let them leave or make them leave. The other engineers on your team will like it, and some of them will become your new Sr Engineers.

The first Sid came in and wanted to rewrite in C++.

Then the second Sid wanted to rewrite in Java.

The whole time the HTML is 25% space chars, served, sent, received, discarded, because the PHP guy likes deep indentation, and the DB is constantly burning like the sun because all the business logic is in stored procedures.

(That was the problem, not which abstraction the servers are written in, since all they do is pass data back and forth to the fiery inferno of the database.)

If there was a reliable general algorithm to make good software managers would hire lousy engineers then tell them to execute the algorithm. There isn't, they can't. The fallback is to be as picky as possible about who has influence on the software.

You can immediately spot a misaligned engineering culture when every team has its own tech stack and its own ops as it means that none of the teams trusted each other for anything and had to resort to federation. On the flip side you can se bad engineering cultures where decisions are made based on pure conformity with what was previously built regardless of the problem being solved.

There's a happy middle-ground, a company like etsy with 200-400 engineers can happily afford a small team of 2 engineers trying out scala/mongo for something, it might work out, and nuking it from orbit won't cost that much in the grand scale of things.

But more often than not, this feedback comes from engineers that a) have never been (as you say) bitten by complexity, or b) they aren't in the position to deal with all the negative consequences for those decisions.

There's probably some wisdom in letting your direct reports experience the kinds of failure in making these decisions, so they develop that sort of empathy, but the cost of that failure is sometimes just unacceptable for the business; especially in periods of cash runway constraints.

Because you may know how to hire the best or have the skill to know who the best are, but you don't know how to be the best, so how can you judge their work if by definition you're not as good as they are?

I’m writing a fairly large-scale app, right now.

It’s written in Swift (frontend), using IB (classic UIKit), and PHP/MySQL/Postgres (backend).

It does not use SwiftUI (shiny), or Rust (shiny, but a bit more dusty), or some form of NoSQL.

I picked standard UIKit, because I like SwiftUI, but the app has a fairly intricate and non-simple storyboard. I am not confident that SwiftUI is up to the task, and I know that IB can do it.

I’ve been writing in Swift since the day it was announced, so I already know it is up to the task, despite being a fairly new kid on the block.

I picked PHP, because I’m already quite decent with it, and, despite the hate, it is a perfectly good, performant, proven, and supported enterprise language. There’s a better than even chance the server will be swapped or rewritten in the future, so it’s a good idea to use my implementation as a PoC and architecture model, anyway. It will need to run the system during the nascent phase of the project, so it needs to be solid and secure. There’s no way I will take the risk of writing such a critical system, in a language I barely know (See this scar? I’ve done that -long story).

I picked MySQL and Postgres, because they are proven, robust databases, and can be installed on most low-cost hosting solutions (the app is for an NPO). I used PDO to interact with the databases, for security and reliability, anyway, so it’s entirely possible to add support for more databases, in the future.

Also, backend is not my specialty. What I did, was design a layered architecture that will allow future “shiny” engineers a path to replacing the engine. I wrote an abstraction layer into the server, allowing a pretty wholesale subsystem replacement. The app communicates with the server through a classic REST-like JSON API, so there’s another place for a swap. I’m not married to a system like GraphQL, with the need for dependencies; but the layered architecture allows use of GraphQL, anyway, if people really want it (it is cool and robust, but is difficult to use without some big dependencies).

Speaking of dependencies, I do everything in my power to eliminate them. I have been badly burned, in the past (not too distant, either -I had to do an emergency dependencyectomy, just a couple of weeks ago), by over reliance on dependencies. It means some extra work, on my part, but not crippling.

Speaking of boring, few things are more boring than documentation, testing and quality coding techniques. My testing code usually dwarfs my implementation code. I spend many boring hours, running tests, and examining results.

In my experience, I don’t think I’ve ever written a test that wasn’t necessary. They always expose anomalies. I just went through that, in the last week or so, as I was refactoring a legacy system for the app I’m writing. I actually encountered and fixed a couple of issues (including a really embarrassing and scary security bug) that have been in the backend for two years.

But that’s just me. WFM. YMMV.

In many ways it’s an argument for MongoDB considering if you’ve built a JS based application MongoDB minimizes any additional learning and having to translate between the objects on the front end and in the backend. Non relational DBs are also easier to scale horizontally without requiring any application changes.

The OP is an argument against introducing new technology without a significant clear benefit. It’s basically saying that simply having a new tech can add significant complexity, unknown unknowns, and require much more maintenance and other costs.

So if your web application is currently running on MongoDB and it’s running well then this is an argument to stick to MongoDB instead of say migrating to postgresql going forward.

Good luck...

Also product managers (myself) who hold the P&L bag and tell folks like them to get fucking real when they are smoking dope.

Or more like it's the scarcity, by definition new tech still doesn't have many experts in it. So if you're one of the few that learns it you're all of a sudden pretty differentiated compared to your peers. Even if the new tech turns out to be pure garbage down the road it doesn't matter because in the meanwhile you can land the hippest jobs and win the admiration of your peers by being so far ahead of everyone else.

It sounds like you're saying the fear is unwarranted. It very much is a real fear as long as people interviewing them actually count that against them and value new tech.

Reward employees a direct and substantial cut of the profits and incentivize to them to stay 5-10 years and these behaviors should disappear.

The loss of job security, frequent job hopping has created more incentives to optimize for the next job switch and not value add.

The explosion of startups also contributes to this. They often have to attract employees by offering the promise of autonomy. Most startup employees know they aren’t getting rich. So they milk the startup for maximum resume points and move on.

The VCs unload these bloated companies into inflated stock markets and the cycle continues.

DHH runs really small companies and pays his employees really well and doesn’t work them too hard. Employees have no real reason to leave. They see a direct link between the low overhead and their job security and work life balance. Aligning incentives fixes this.

I find it interesting that these technologies are considered "state of the art" (SOTA). What does SOTA mean in this context? I could see an argument for postgresql and rails/django being SOTA as I think many believe them to be fairly mature, secure, and feature complete.

That app could have been written in the mid-1990s using WebObjects in just a few months.

Technologies like MongoDB, React, GraphQL, Microservices etc exist because modern, real-world apps are generally far more advanced than just a glorified CRUD app. Consumers simply have higher expectations and more demands for what web apps should be able to do.

This is 100% how we did things for the last 5 years. We used the exact same basic tools & APIs, but iterated on how we integrated them together by way of our code.

We took what most people would call a "toy" stack, consisting of just C#/AspNetCore/SQLite (and recently Blazor), and turned it into something that can serve thousands of users in production across several B2B customers. We don't use any sort of containerization or cloud tech. Our software can be deployed by unzipping a file on the target environment and running a single command. You would be surprised at how much speed you can get out of SQLite when you have had 5 years to live with and experiment with its capabilities. On top of its extensive internal testing framework, I have several testaments to its stability sitting in our customers' environments right now. We've got OLTP SQLite databases that are getting close to 1TB in production without any signs of distress.

So, instead of focusing all of our energy on shiny things, we focused on building a 100% integrated vertical with (mostly) boring tech so that we can fully manage the lifecycle of our software product with the product itself. We have a management web interface built out for everything we need to do for building & operating the solution. We are very close to being able to partner with other organizations who can run the implementations on our behalf because the software is so easy to manage now. This is the type of real-world horizontal scalability the CEOs are after.

It's sad that most places - and by definition the largest places also - end up being a meat grinder and people just hop companies and teams within companies every year or two. By the time you start understanding the domain you move on. It takes years to internalize a problem and understand it deeply.

It also applies to managing your life, personally. Know what things you do, what your personal goals are, and don't let yourself get distracted by the latest and greatest social media trends or stuff your friends are doing.

Focus. Mature. Achieve.

The trap most engineers fail is that they only think about scaling. It's obviously an interesting problem, but until a company becomes successful it's not really something you should worry about, and for the most part things that (allegedly) scale well are more expensive, slower and harder to maintain.

But there are so many ways to innovate that's not just about scaling: can you make your application faster? What are things you never even considered because you have subconsciously internalised as a physical limit of reality when actually it's not?

One of the examples I'm currently exploring is the idea of moving a large amount of data in memory. I remember decades back when Google announced that it's search indexes are now fully in memory (they proudly announced that any given search query might run through a thousand computers). I cannot imagine how many possibilities it enabled for their product that were not possible before. The experimentation with new technology in pursuit of completely new ways of exploring your problem space should always be encouraged, and if boring technology cannot do it then that's when you give up on it.

I've spoken with 70+ different devs working on 70+ different projects of all sizes on my Running in Production podcast[0] and the choose boring tech phrase came up a whole bunch of times, and especially the idea of using innovation tokens. If it helps folks build and ship their app in a quick and stable way, that seems like a big win to me.

[0]: https://runninginproduction.com/podcast/

I do agree. Although the point of the article is to _lean_ more on "boring technology" side of, and paying extra effort when considering adopting newest flashy things.

Having read the article 3-4 time in the last years, I don't think they say "don't use new things", just "not too many new things at the same time"

For what it’s worth, it’s a great read that I would recommend to anyone in the industry.

They aren’t forcing technologies on you, but driving home the true cost of long term maintenance and investing in the “core stack” that you already have instead of adding N technologies to solve N business problems. This is good stuff.

Maybe read it before commenting next time.

I wish HN had a feature where it could detect that you clicked the link and disallowed commenting before that. At the very least you’d have to click the link, even if you just immediately click back without reading, and you’d know what you were doing was circumventing the spirit of the place.

More broadly, the software industry "freeloads" over the work of all other workers who create a safe environment where food and energy is easily available. But the whole attraction of creating a society is to make life better eventually, otherwise we might as well have never left the savannah.

Ed: I'm an idiot.

If your query was exposed as a REST API you could have easily accessed it via a micro-service. And MongoDB has a pretty powerful and easy to use query language. I could've answered your query in about 5s.

Sounds more like a business or process limitation.

> I chose “boring technology” as the pithy SEO-friendly title for this content, and I regret it most days. It’s kind of distracting. “Boring sounds bad, why is he saying it’s good?” Et cetera. It’s a real shitshow.

> But what I’m aiming for there is not technology that’s “boring” the way CSPAN is boring. I mean that it’s boring in the sense that it’s well understood. It’s bad, but you know why it’s bad. You can list all of the main ways it will let you down.

> It’s important to master the tools that you do pick.

This view feels so strange to me.

As an engineer, I find myself the happiest when I manage to build the simplest and the most maintainable thing for the business requirements. Extra happy, if it turns out we didn't paint ourselves into a corner when those requirements increase or change.

I just want to minimize my own tears in the future.

> If you are an engineer and you want to try a new technology, go for it. Even if it doesn't make sense.

Imagine how immoral this advice sounded like if it came from the mouth of a carpenter, an auto mechanic or a doctor.

Maybe this just activates my PTSD acquired from working in a startup with a long history of codecampers doing CV-Driven Development.

Yes, by all means, have fun, at HOME, not on company time.

If you had a choice, isn't it better to build cool stuff with boring technology instead of building boring stuff with cool technology?

I think his point was to focus more on the product not on technology, and by choosing a boring technology you will likely have more time and fewer problems shipping awesome products.

In my opinion, it also works in the other direction, where the managers push things onto the developers. "Can we use this blockchain thing?", "we need to do AI", "we should use this cross platform mobile app tool I heard and read 3 minutes about".

If you're a responsible adult you'll probably be the one who has to ship the stuff you started with the "flashy framework X" - boring technology is about maximising happiness - it just takes a while for this to sink in.

Don't put them on the critical path of your company without an exit plan.

Trying out new things makes a ton of sense. But trying them out in a domain where unforeseen consequences could potentially destroy you is not a smart move.

https://news.ycombinator.com/item?id=23444594

https://news.ycombinator.com/item?id=9291215

Just like modern web apps are far more scalable, highly-available, secure and performant then they were 15 years ago.

Most dev work is "blue collar" work, following top down business decisions. The opportunities for devs to push impactful bottom up projects for a company business are rare at best.

I have always preferred organizations where business problems-to-solve were pushed top-down rather than executive decisions. It encourages business understanding, initiatives and ownership.

Choosing boring technology doesn't have to be a top down thing, although it probably needs to have some mechanism for aligning a whole team (since 10 different "boring technologies" are no longer boring and bring along all the problems of using shiny new toys).

Typically how this goes, few of the existing software migrate to the new stack (it's too much work to migrate and it doesn't work as well). After the guy is gone, you can deprecate the new stack immediately and don't bother supporting it. I've seen it happen a lot.

What you don't say is how large the company is. For startups and small companies, my experience is that every generation of developers (couple years) will rewrite most of everything. That's just how things go in startup, filled by eager young folks who have no experience and build their resume. That's only possible because there isn't that much code in the first place.

Larger companies can be stuck with multiple stacks because there's too many software to migrate and no sucker to do it. Older projects are stable and have no active developers working on them, they're not going to be rearchitectured. Other departments/developers know that they can't trust the shiny new tool from your department, that's going to be deprecated next year, they don't adopt in the first place (a great example of why large companies resist changes).

It's just the state of the industry I guess.

IMO it's crazy how few people working on tools/frameworks can inflict migrations on a hundred developers/projects (thousands in larger companies), often for no benefits.

I realize this doesn't answer your question, how to help?

No idea. Would be nice to cancel projects early before they reach that stage. It's probably too late now.

What you can improve is your perception. If it makes you feel better, one pro of all this churn is that there are many more jobs, doing rewrites over and over again. It's nice to have a job in the current climate.