That's just how Google operates in my experience... Avoid Google libraries unless absolutely necessary, and if you do adopt Google libraries, be prepared to either be forever multiple years out of date or spend significant resources on keeping it up to date.

The API is indeed prone to change without notice, but it isn't anything terribly unmanageable.

> really difficult to keep the version of gVisor I was using up to date

For our project, we update gvisor whenever Tailscale does.

It could be that you happened to find a period of rapid change, but it is also possible that you ran into the issue that raggi mentioned in the sibling comment.

I don't work on gVisor anymore. That said, I think it would be a tough sell. It would be a pretty big breaking change. Also, there is already a problem with people trying to send patches against the go branch and making it the default would make that much worse.

I think the solution is an automatically exported repository at a different path. Kind of (or maybe exactly) like what Tailscale/bradfitz used to maintain.

Speaking of ... https://coder.com/docs -- what's next is empty.

Any tips I'm a beginner

I understand the impulse, but I think it misconstrues the "red tape" this method avoids. It's sidestepping a quirky OS limitation, which dates back to an era of "privileged ports" and multi-user machines. It's not really sidestepping any sort of modern policy boundary. For instance: you could do the exact same thing with WebSockets (and people do).

I think the key insight behind this approach (and I'm biased here having written something similar) is that the difference between QUIC and (wireguard + network stack) is A LOT less than you might think.

I'm confused on why they would need a TUN device for a client or server application, so why they would need this solution in the first place(even with their explanation).

As I understand the only reason you'd use a TUN interface is if you want to send/receive raw IP packets. Their marketing doesn't make it very clear what their product does, but I can't see a reason it would need to send/receive raw IP packets rather than TCP/UDP packets over a specific port...

Agree. Very unclear why they won't simply use a secure socket or why a user space tunnel will be needed.

I surmise that the reason might be that a user space tunnel might be faster (like maybe they can do UDP over TCP or something to gain speed improvements).

Good post nevertheless.

Or TLS. It seems to be a remote cloud desktop type of product, so why not use TLS like every other one?

The quote - is this yet another issue caused by abysmal FFI overhead in Go?

A decent while ago, I was the original author of that performance guide. I tried to lay out the set of performance trade-offs in an objective and realistic way. It is shocking to me that you’re spending so much time commenting on a few figures from there, ostensibly w/o reading it.

System call overhead does matter, but it’s not the ultimate measure of anything. If it were, gVisor with the KVM platform would be faster than native containers (looking at the runsc-kvm data point which you’ve ignored for an unknown reason). But it is obviously more complex than that alone. For example, let’s click down and ask — how is it even possible to be faster? The default docker seccomp profile itself installs an eBPF filter that slows system calls by 20x! (And this path does not apply within the guest context.) On that basis, should you start shouting that everyone should stop using Docker because of the system call overhead? I would hope not, because looking at any one figure in isolation is dumb — consider the overall application and architecture. Containers themselves have a cost (higher context switch time due to cgroup accounting, costs to devirtualize namespaces in many system calls, etc.) but it’s obviously worth it in most cases.

The redis case is called out as a worst case — the application itself does very little beyond dispatching I/O, so almost everything manifests as overhead. But if you’re doing something that has 20% overhead, you need hard security boundaries, and fine-grained multi-tenancy can lower costs by 80% it might make perfect sense. If something doesn’t work for you because your trade-offs are different, just don’t use it!

I don’t believe many PAAS run gVisor; a surprising number just run multitenant docker.

The netstack stuff here has nothing to do with the rest of gVisor.

In the context of coder, the userspace TCP overhead should be negligible. Based on https://gvisor.dev/docs/architecture_guide/performance/ and assuming runc is mostly just using the regular kernel networking stack (I think it does, since it mostly just does syscall filtering?) it should be at most a 30% direct TCP performance hit. But in a real application you typically only spend a negligible amount of total time in the TCP stack - the client code, total e2e latency, and server code corresponding to a particular packet will take much more time.

You'll note their node/ruby benchmarks showed a substantially bigger performance hit. That's because the other gvisor sandboxing functionality (general syscall + file I/O) has more of an impact on performance, but also because these are network-processing bound applications (rare) that were still reaching high QPS in absolute terms for their perspective runtimes (do you know many real-world node apps doing 350qps-800qps per instance?).

Because coder is not likely to be bottlenecked by CPU availability for networking, the resource overhead should be inconsequential, and what's really important is the impact on user latency. But that's something likely on the order of 1ms for a roundtrip that is already spending probably 30-50ms at best in transit between client and server (given that coder's server would be running in a datacenter with clients at home or the office), plus the actual application logic overhead which is at best 10ms. And that's very similar to a lot of gvisor netstack use cases which is why it's not as big of a deal as you think it is.

TLDR: For the stuff you'd actually care about (roundtrip latency) in the coder usecase the perf hit of using gvisor netstack should be like 2% at most, and most likely much less. Either way it's small enough to be imperceivable to the actual human using the client.

Google is my former employer and this statement isn't referring to stuff I heard while employed there.

But after I left, I heard a that alot of the poor performance of Cloud Run is just plain old oversubscribed shared core e2 stuff.

No. The providers that did so soundly used virtualization to accomplish this, and a big part of the appeal of K8s is having a much lightweight unit of scheduling than full virtualization. gVisor is a middle ground between full virtualization and shared-kernel multitenant (which has an abysmal security track record).

OpenVZ, Virtuozzo and friends definitely weren't secure the way gVisor or Firecracker are. You can still do that and some providers do, doesn't make it a good idea.

You can have multiple layers of security boundary on most of the customer-exposed surface area, and avoid more risky surface areas in the kernel.

Portable is a bit of a weird word here because for many of us with gray beards the word means architectures, kernels and systems, but I think in this context it tends to more mean "can run just as easily on my macbook as in a cloud container", but in practice the software isn't that portable, as Go isn't that portable - at least not in the context of vs. a niche C "portable network stack" that would build roughly anywhere that there's a working C toolchain, which is almost everywhere.

Constant security fixes for the kernel are a real pain in deployments unless you follow upstream kernels closely. If your business is in shipping Linux runtimes with a high packing density, you really need to find ways to minimize the exposed Linux surface area, or organize to be able to ship kernel upstream updates at an extremely high frequency (relative to normal infrastructure upgrade rates for kernels / mandatory reboots) (and I would not consider kexec safe in this kind of context, at all).

An alternative approach might be firecracker / microvms and so on, but those have their own tradeoffs too. The core point is that you want more than one layer between the host machines and the user code that wants to interact with Linux features.

Correct (I mean, that's fundamentally what WireGuard is: a UDP TCP/IP tunnel, with strong modern encryption).

Yes; also see: https://github.com/WireGuard/wireguard-go/blob/12269c2761734...

Why do you suspect a user space implementation of TCP would improve those issues beyond existing kernel implementations?