The Great Re-shard: adding Postgres capacity (again) with zero downtime (2023)
30 points by unchar1 4 months ago | 8 comments- winrid 4 months agoWonder what hardware they are using and what the actual iops numbers are
- tucnak 4 months agoI wouldn't be surprised if it's some stock AWS instance type, probably not even NVMe-enabled, or some kind of metal instance bottlenecked by EBS. The downside of Cloud push in SaaS is that they were unable to tap into the ever-changing storage landscape, NVMe-oF, and so on. This is mostly gun-in-foot matter more than anything.
- jitl 4 months agoI work at Notion.
It’s stock RDS on EBS, which as you say is super slow compared to Genuine Local Storage. Any large scan or large result set query that doesn’t fall into the Postgres cache or the underlying EBS cache is frustratingly slow.
However almost all of our reads are “get row by ID” lookups that hit memcached. Our query pattern is mostly recursive graph traversal which Postgres is bad at, with our table size even a 20x faster disk wouldn’t make it feasible to do in SQL. We also copy paste “notion database” data in to a stateful caching service, but that also needs specialized indexing for user defined schema, another thing Postgres can’t handle (my last project was an improvement to that service).
The place we really suffer from the EBS slowness is with full table scan data migrations, stuff like “for all blocks in Notion, block = f(block)”. For this we use “Big Data” approach reading from DB dumps in S3 instead of Postgres directly.
Ultimately I’d prefer to serve every read from your device’s local storage, which is my current project.
- choilive 4 months ago> Ultimately I’d prefer to serve every read from your device’s local storage, which is my current project.
Is that for native desktop/mobile clients only or web as well?
- tucnak 4 months agoMuch appreciated, & good luck with your project!
- winrid 4 months agoYeah I figured it was EBS :) the default max iops is like 16k... an i3en can usually destroy 10 default-config EBS backed instances :P
Thanks for the info and GL with your project!
- choilive 4 months ago
- jitl 4 months ago
- tucnak 4 months ago