How to become good at system design?
64 points by Tim25659 3 years ago | 27 comments- cppr 3 years agoDesigning Data-Intensive Applications, Martin Kleppmann
https://www.youtube.com/c/SystemDesignInterview
https://www.educative.io/courses/grokking-the-system-design-...
- anyonecancode 3 years agoIn addition to the responses talking about getting better with software specifically, I think it's helpful to work get better about thinking in a systems way, generally. There are systems all around us, providing plenty of opportunity to practice systems thinking. For instance right now looking out my window at my backyard I see a drainage basin with frozen water there. I can start asking questions like "where did that water come from," "how long will it stay in that drainage basin", "where does it go when it leaves," etc. Or looking at the next house over and seeing the steam coming out a chimney -- where does that steam come from? What is the heat source? How did that heat get to that house? Etc.
System design is, primarily, a way of thinking -- identifying how given things are actually connected, thinking through how they are connected, what happens up and down those connections as things change. Software systems are just one particular example of this (and, as a bonus, when you start to include non-software parts, such as the humans actually using the software, your software system design will be stronger).
I'm sure others will cite this too, but I don't see it yet at the time I'm writing this, so for a specific book let me recommend Thinking in Systems by Donella Meadows[0]
[0]https://www.chelseagreen.com/product/thinking-in-systems/
- anyonecancode 3 years agoAdding one more thing to my comment -- I think an important skill is the ability to put details in boxes and then kind of forget about them. Eg when thinking about a software system, being able to look at it with the appropriate "zoom level." The shape of the boxes, and how they relate to each other, are what you need to look at. Crucially, though, this doesn't mean details aren't important, it means they're not important _right now_. That's why I picked the analogy of the boxes; you will at some point need to open up the boxes again.
Or another way to try and explain it -- you can't keep everything in your mind's "RAM" -- when trying to think systematically, you need to offload the details onto disk for later retrieval. Don't throw them away! People who are "vision people" and brag about not sweating the details become really annoying to work with as a software engineer -- but you won't be able to grasp the system if your mental RAM is cluttered up with those details all the time.
- captainredbeard 3 years agoI want to second the book recommendation, it changed how I viewed the world and politics.
- apineda 3 years agoI only got through about halfway, stocks and flows is all I got out of it.
- captainredbeard 3 years agoThat's the gist of it really
- captainredbeard 3 years ago
- apineda 3 years ago
- xyzzy21 3 years ago2nd (3rd?) the recommendation on the book
- anyonecancode 3 years ago
- thecleaner 3 years agoPlease don't bother with the whole charade of you have to do it to learn it. This kind of advice places more importance on people working at internet companies which have achieved scale. If you are at these companies then you have nothing to worry, experience will teach you. If you are not, follow @cppr's advice and read through DDIA. It will give you good enough introduction to data systems - not just databases. Then I would suggest to learn how message queues work since that part isn't covered in the book. Do a lot of mock designs and attack your designs with different load parameters - how much data do I store, how many machines (if more than one is necesary), can I do it simpler. Honestly if you can estimate storage and throughput well then you are fine. Pick up some distributed systems papers if you are very interested. Once you have these down you would be able to pick more topics yourself. Good luck. Keep learning.
- porker 3 years agoOne topic I haven't seen widely discussed in books but happens in the real world is back pressure. I think you're getting to this with "attack your designs", though I find it hard to predict.
Have you any thoughts, books or resources you turn to, to make sure the whole pipeline is balanced - or doesn't fall over when one part is overloaded?
- thecleaner 3 years agoYou need to see what happens in you data processing chain. The pipeline is only "balanced" based on what Rps you want. Throughput calculations are easier to do. Latencies are harder to predict but honestly latency calculations without sensible benchmarks are horseshit.
Back pressure is refusing to let the queue unbounded or take in more requests than slots available. That's pretty much it.
- thecleaner 3 years ago
- porker 3 years ago
- aristofun 3 years agoSorry, there is no way to be good at something without doing it.
After some initial reading try to get your hands dirty with any large scale application you can get access to.
- Jensson 3 years agoBut there are many ways to fail at getting good at something while doing it a lot.
- Jensson 3 years ago
- bwh2 3 years agoThe book System Design Interview is quite good and pragmatic. Personally, I found Designing Data-Intensive Applications less useful because it works on a much lower level. Like if you want to know how database internals work, then read DDIA. If you want to understand how to author a scalable API with layers of caching and distributed data storage, read SDI.
- Tim25659 3 years agoSDI?
- Tim25659 3 years ago
- 3 years ago
- jacobsimon 3 years agoIn addition to the great resources already mentioned here, we’ve been making hands-on videos and lessons at Exponent that show you system design in practice, with interview prep as the main focus but advice and discussions with people across the industry.
Check out some of the free videos on our site[0] and YouTube channel.[1]
- taubek 3 years agoI think that good analytical skills are prerequisite. You must know how each component/module will interact /interfere with other components. I don't know if there are any labs that might help you. Which existing application do you consider to be large scale application?
- sharps_xp 3 years agomy advice is that you need to both consume and produce system design.
don’t rely on interviewing people to test yourself. you have to practice talking or writing and stop settling with the feeling that you just learned something and will have the judgement to know when to use xyz when problem abc comes in front of you.
i built enumerable.co so that i can evaluate myself, to practice producing the knowledge i’m consuming, and reach the edge of my understanding.
there’s no database and i built over a few days, so it’s just a single player way to figure out how much you really don’t know
- pshirshov 3 years agoLearn ho to modularize and how to keep cohesion high and coupling low. Avoid horizontal dependencies, don't mix abstraction layers, learn to isolate, apply tagless final.
Use formal API definitions.
- aristofun 3 years agoAlthough these are in general great advice, it means nothing to someone without handson experience, way too generic and abstract.
- aristofun 3 years ago
- sumosudo 3 years agoVMWare.
GNS3.
Docker.
Remember that large scale, complicated systems that work invariably start out as small simple systems that work, so build it virtually first, where you can break and re-build to your hearts content.
- Tim25659 3 years agopeople who works on CRUD applications will stay far.. far.. away from the system design and distributed systems.