quick question for advice - we have been evaluating minio for a in-house deployed storage for ML data. this is financial data which we have to comply on a crap ton of regulations.

so we wanted lots of compliance features - like access logs, access approvals, short lived (time bound) accesses, etc etc.

how would you compare garage vs minio on that front ?

That's very cool; I didn't expect Garage to scale that well while being so young.

Are there other details you are willing/allowed to share, like the number of objects in the store and the number of servers you are balancing them on?

I would settle for first-class support for object hashes. Let an object have metadata, available in the inventory, that gives zero or more hashes of the data. SHA256, some Blake family hash, and at least one decent tree hash should be supported. There should be a way to ask the store to add a hash to an existing object, and it should work on multipart objects.

IOW I would settle for content verification even without content addressing.

S3 has an extremely half-hearted implementation of this for “integrity”.

That's how we use S3 in Peergos (built on IPFS). You can get S3 to verify the sha256 of a block on write and reject the write if it doesn't match. This means many mutually untrusting users can all write to the same bucket at the same time with no possibility for conflict. We talk about this more here:

https://peergos.org/posts/direct-s3

Garage splis the data into chunks for deduplication, so it basically already does content addressed storage under the hood..

They probably don't expose it publicly though.

Take a look at https://github.com/n0-computer/iroh

Open source project written in Rust that uses BLAKE3 (and QUIC, which you mentioned in another comment)

you might be interested in https://github.com/perkeep/perkeep

Have you seen https://github.com/willbryant/verm?

Something related that I've been thinking about is that there aren't many popular data storage systems out there that use HTTP/3 and/or gRPC for the lower latency. I don't just mean object storage, but database servers too.

Recently I benchmarked the latency to some popular RPC, cache, and DB platforms and was shocked at how high the latency was. Every still talks about 1 ms as the latency floor, when it should be the ceiling.

Sounds a little like Kademlia, the DHT implementation that BitTorrent uses.

It's a distributed hash table where the value mapped to a hash is immutable after it is STOREd (at least in the implementations that I know)

But you don't really handle blobs in real life: they can't really be handled, they don't have memorable name (by design). So you need an abstractly layer on top of it. You can use zfs that will deduplicate similar blobs. You can use restic for backups that will also deduplicate similar parts of a file also in an idempotent way. And you can use git that will deduplicate files based on their hash

You might also be interested in Tahoe-LAFS https://www.tahoe-lafs.org/

Have a look at LakeFS (https://docs.lakefs.io/understand/architecture.html).

Files are stored by hash on S3. Metadata is stored in a database. I run it locally and access it just like an S3 store. Metadata is in a Postgres DB.

Check also SeaweedFS, it has some interesting tradeoffs made, but I hear you with wanting some of the properties you're looking for.

IPFS like "coordination free" local S3 replacement! Yes. That is badly needed.

The RADOS K/V store is pretty close. Ceph is built on top of it but you can also use it as a standalone database.

I read that curl recently added sigv4 for what that’s worth[0]

0: https://how.wtf/aws-sigv4-requests-with-curl.html

Implementing v4 on the server side also requires the service to keep the token as plain text. If it's a persistent password, rather than an ephemeral key, that opens up another whole host of security issues around password storage. And on the flip side requiring the client to hit an endpoint to receive a session based token is even more crippling from a performance perspective.

You can implement S3 V4 signatures in a few hundred lines of code.

https://github.com/Peergos/Peergos/blob/master/src/peergos/s...

It makes sense to tap into the existing ecosystem of AWS S3-compatible clients.

Check something like PicoS3 or https://github.com/sentienhq/ultralight-s3

There is a few "very minimal" sigv4 implementations ...

Sending your api key in the header is equivalent to basic auth.

Tried this for my own homelab, either I misconfigured it or it consumes x2(linearly) memory(working) of the stored data. So, for example, if I put 1GB of data, seaweed would immediately consume 2GB of memory constantly!

Edit: memory = RAM

Looks awesome. Been looking for some flexible self-hosted WebDAV solutions and SeaweedFS would be an interesting choice.

Does your file system have search? Mine doesn’t. Instead I have software that implements search on top of it. Does it support filtering? Mine uses software on top again. Which an S3 api totally supports.

Does your remote file server magically avoid network latency? Mine doesn’t.

In case you didn’t know, inside the bucket you can use a full path for S3 files. So you can have directories or folders or whatever.

Some benefits of this system (KV style access) is to support concurrent usage better. Not every system needs it, but if you’re using an object store you might.

> Replicating a real file system is not that hard

What personal experience do you have in this area? In particular, how have you handled greater than single-server scale, storage-level corruption, network partitions, and atomicity under concurrent access?

I use CephFS.

Blob storage is easier than POSIX file systems:

You have server-client state. The concept of opened files, directories, and their states. Locks. The ability for multiple writers to write to the same file while still providing POSIX guarantees.

All of those need to correctly handle failure of both the client and the server.

CephFS implements that with a Metadata server that has lots of logica and needs plenty of RAM.

A distributed file system like CephFS is more convenient than S3 in multiple ways, and I agree it's preferable for most use cases. But it's undoubtedly more complex to build.

It's a legitimate question and I'm glad you asked! (I'm not the author of Garage and have no affiliation).

Filesystems impose a lot of constraints on data-consistency that make things go slow. In particular, when it comes to mutating directory structure. There's also another set of consistency constraints when it comes to dealing with file's contents. Object stores relax or remove these constraints, which allows them to "go faster". You should, however, carefully consider if the constraints are really unnecessary for your case. The typical use-case for object stores is something like storing volume snapshots, VM images, layers of layered filesystems etc. They would perform poorly if you wanted to use them to store the files of your programming project, for example.

> S3 is a horrible interface with a terrible lack of features.

Because turn out that most applications do not require that many features when it comes to persistent storage.

> I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS.

It's mostly just S3, really. You don't see anywhere near as many "clones" of other AWS services like EC2, for instance.

And there's a ton of value on being able to develop against a S3 clone like Garage or Minio and deploy against S3 - or being able to retarget an existing application which expected S3 to one of those clones.

S3 exposes effectively all the metadata that POSIX APIs do, in addition to all the custom metadata headers you can add.

Implementing a filesystem versus an object store involves severe tradeoffs in scalability and complexity that are rarely worth it for users that just want a giant bucket to dump things in.

The API doesn't matter that much, but everything already supports S3, so why not save time on client libraries and implement it? It's not like some alternative PUT/GET/DELETE API will be much simpler-- though naturally LIST could be implemented myriad ways.

You wouldn't want your "interactive" user filesystem on S3, no, but as the storage backend for a server application it makes sense. In those cases you very often are just storing everything in a single flat folder with all the associated metadata in your application's DB instead

By reducing the API surface (to essentially just GET, PUT, DELETE), it increases the flexibility of the backend. It's almost trivial to do a union mount with object storage, where half the files go to one server and half go to another (based on a hash of the name). This can and is done with POSIX filesystems too, but it requires more work to fully satisfy the semantics. One of the biggest complications is having to support file modification and mmap. With S3 you can instead only modify a file by fully replacing it with PUT. Which again might be unacceptable for a desktop OS filesystem, but many server applications already satisfy this constraint by default

> Replicating a real file system is not that hard

Ummmm what? Replicating a file system is insanely hard

it's because many other cloud services offer sending to S3, that's pretty much it

Because at this point it's a well known API. I bet people want to recreate AWS without the Amazon part, and so this is for them.

Which, to your point, makes no sense because as you rightly point out, people use S3 because of the Amazon services and ecosystem it is integrated with - not at all because it is "good tech"

I can only answer for Garage and not others. Garage is the result of the desired organization of the collective behind it: deuxfleurs. The model is that of people willing to establish a horizontal governance, with none being forced to do anything because it all works by consensus. The idea is to have an infrastructure serving the collective, not a self hosted thing that everyone has to maintain, not something in a data center because it has clear ecological impacts, but something in-between. Something that can be hosted on secon-hand machines, at home, but taking the low reliability of machines/electricity/residential internet into account. Some kind of cluster, but not the kind you find in the cloud where machines are supposed to be kind of always on, linked with high-bandwidth, low-latency network: quite the opposite actually.

deuxfleurs thought long and hard about the kind of infra this would translate to. The base came fast enough: some kind of storage, based on a standard (even de-facto only is good because it means it is proven), that would tolerate some nodes go down. The decision of doing a Dynamo-like thing to be accessed through S3 with eventual consistency made sense

So Garage is not "simply" a S3 storage system: it is a system to store blobs in an unreliable but still trusted coonsumer-grade network of passable machines.

Minio assumes each node has identical hardware. Garage is designed for use-cases like self-hosting, where nodes are not expected to have identical hardware.

performance, therefore cheaper

That's a deliberate design decision.

from an operations point of view, I am surprised anyone likes Raft. I have yet to see any application implement Raft in a way that does not spectacularly fail in production and require manual intervention to resolve.

CRDTs do not have the same failure scenarios and favor uptime over consistency.

It’s an S3 api compatible object store that supports distributed storage across different servers.

Object store = store blobs of bytes. Usually by bucket + key accessible over HTTP. No POSIX expectation.

Distributed = works spread across multiple servers in different locations.

Files are normally stored hierarchically (e.g. atomically move directories), and updated in place. Objects are normally considered to exist in a flat namespace and are written/replaced atomically. Object storage requires less expensive (in a distributed system) metadata operations. This means it's both easier and faster to scale out object storage.

There are few.

From the perspective of consistency guarantees, object storage gives fewer of such guarantees (this is seen as allowing implementations to be faster than typical file-systems). For example, since there isn't a concept of directories in object store, the implementation doesn't need to deal with the problems that arise while copying or moving directories with files open in those directories.

There are some non-storage functions that are performed only by filesystems, but not object storage. For example, suid bits.

It's also much more common to use object stores for larger chunks of data s.a. whole disk snapshots, VM images etc. While filesystems aim for the middle-size (small being RDBMs) s.a. text files you'd open in a text editor. Subsequently, they are optimized for these objectives. Filesystems care a lot about what happens when random small incremental and possibly overlapping updates happen to the same file, while object stores care about performance of sequential reads and writes the most.

This excludes the notion of "distributed" as both can be distributed (and in different ways). I suppose you meant to ask about the difference between "distributed object storage" and "distributed filesystem".

I believe OpenStack Swift in particular is known to work well in some large organizations [1], NVIDIA is one of those and also invested in its maintenance [2].

1. https://www.youtube.com/watch?v=H1DunJM1zoc 2. https://platform.swiftstack.com/docs/

Minio is fairly easy to setup locally or in CI.

We use it for CI in ClickHouse, for example: https://github.com/ClickHouse/ClickHouse/blob/master/docker/...

There is formal proof for some parts of garage layout system, see https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/...

AGPL https://git.deuxfleurs.fr/Deuxfleurs/garage

The EU, but yeah. NLNet are the ones that judged the applications and disbursed the funds.

I don't think I have ever personally felt older than having someone describe anything docker related as "old-school"