Docker on MacOS is slow and how to fix it

193 points by riccardomc 2 years ago | 203 comments
  • kdrag0n 2 years ago
    Funny that this came up — shameless plug: I've actually been working on a new Linux+Docker+Kubernetes solution for macOS recently! Already added quite a few improvements over existing apps including Docker Desktop, Rancher, Colima, etc:

    - Fast networking: 30 Gbps! vs. 150 Mbps with Docker VPNKit. Full VPN compatibility, IPv6, ping, ICMP and UDP traceroute, and half-open TCP connections.

    - Bidirectional filesystem sharing: fast VirtioFS to access macOS from Linux, but also a mount to access the Linux filesystem from macOS. This setup can help with performance: for example, you could store code in Linux and edit it from macOS with VS Code (which can take the performance hit of sharing), so the container runs with native FS speed.

    - Not limited to Docker or Kubernetes. You can run multiple full Linux distros as system containers (like WSL) so they share resources.

    - Fast x86 emulation with Rosetta

    - Much lower background CPU usage. Only ~0.05% CPU usage and 2-5 idle wakeups per second — less than most apps, while Docker wakes up ~120 times per second. Made possible with low-level kernel optimizations. Also, no Electron!

    - Better solutions to other problems that can occur on macOS: clock drift is corrected monotonically, dynamic disk size, and more I'm working on now. Will look into memory usage too, although I can't guarantee a good fix for that.

    - No root needed.

    Planning to release this as a paid app in January. Not OSS, but I think the value proposition is pretty good and there will be a free trial. Not sure about pricing yet.

    If anyone is interested, drop me an email (see bio) and I'll let you know when this is ready for testing :)

    Also, feel free to ask questions here or let me know if there are other warts you'd like to see fixed.

    • rickette 2 years ago
      Biggest question: is it backwards compatible with Docker? Docker CLI and docker-compose is used in tons of scripts. To have any change of this being adopted in a team setting it needs to be a drop in replacement.
      • kdrag0n 2 years ago
        Yes, Docker CLI will be configured to talk to the VM.
      • maccard 2 years ago
        What are the tradeoffs?
        • kdrag0n 2 years ago
          In general, I don't expect anything to be worse than existing solutions, but not everything will be better.

          Enabling Rosetta can have a minor performance hit on memory-intensive workloads in the VM (not only x86 ones) because of TSO memory ordering, so it'll be optional. Hypervisor.framework doesn't have an API for third-party VMMs to set this and doesn't seem to let the VM modify ACTLR_EL1 either, so unless I can find a private API for it, I'm stuck with Virtualization.framework's limitation of Rosetta being either on or off for the entire VM at boot time.

          Memory usage is probably the biggest uncertainty right now. It should be at least slightly better, but I'm not sure if I can improve it much more due to Virtualization.framework limitations. Still looking into it.

          Networking is implemented with my custom userspace proxy for VPN compatibility. Servers are forwarded to localhost automatically, but you can't connect to the VM by IP because the network doesn't exist from the host's perspective. I've ran into too many issues with Apple's NAT setup and host-only networking is a private API, so this is postponed for now. Should be able to do better with root.

          Graphics won't be supported at launch, but I could look into it later if there's interest. Not sure how feasible acceleration will be if I can't find a way around having to use Virtualization.framework.

          Let me know if there's anything specific that I missed!

          • jitl 2 years ago
            It cost money sounds like the big one
            • i386 2 years ago
              Even after building and selling developer tools for a decade, it always surprises and enrages me to see how miserly developers are.
              • kdrag0n 2 years ago
                That's fair, but I think the value proposition is there for some :)

                I'm honestly not sure how pricing and licensing will work yet, but there will be some way to try it for free. Maybe something like Docker Desktop: free for personal use, license required for companies? That seems like a risky bet as an indie dev.

                There's also the whole question of one-time purchases vs. subscriptions. Subscriptions seem like the optimal model for this, so I'm not sure how to accommodate people who just don't like them.

                Would love to hear if you have any thoughts on how it could be done to reach as many users as possible.

                • brailsafe 2 years ago
                  Also seems like they put in a shit ton of work
              • viraptor 2 years ago
                Docker VPNKit, fast VirtioFS, Much lower background CPU usage

                Are there some new Linux drivers involved, or is this "just" a better tuned VM?

                • kdrag0n 2 years ago
                  No new drivers, but I did make some changes to the Linux kernel. It's mostly a better tuned VM and services on both sides, e.g. a custom fast networking stack in place of Docker's VPNKit.

                  (Also, by "fast VirtioFS", I meant the same VirtioFS implementation tested in the article because it's faster than other solutions — sorry if it wasn't clear.)

                • paolomainardi 2 years ago
                  Blog post author here, cannot wait to see the solution out in the wild, it will be open-source ? Can you let me try it ?
                  • kdrag0n 2 years ago
                    It won't be open-source, sorry.

                    > Planning to release this as a paid app in January. Not OSS, but I think the value proposition is pretty good and there will be a free trial. Not sure about pricing yet.

                    > If anyone is interested, drop me an email (see bio) and I'll let you know when this is ready for testing :)

                  • llIIllIIllIIl 2 years ago
                    Is there any way to join the waitlist for the software?
                    • kdrag0n 2 years ago
                      No formal waitlist yet, but drop me an email (see bio) and I'll let you know when there is!
                  • andix 2 years ago
                    Often overlooked: there is also podman machine and podman desktop (for windows and macOS). It is not as fancy as docker, but fully free and open source.

                    It provides docker compatibility to some extent, you don’t need a license and it’s much less heavy than docker desktop. If you need kubernetes, there is also minikube, which provides a lot of options.

                    Most of the things discussed in this article still apply for podman machine and minikube.

                    • dangoor 2 years ago
                      There's also Rancher Desktop. I don't know if it's less heavy, but it doesn't require a license and also includes k8s.

                      https://rancherdesktop.io/

                      • andix 2 years ago
                        I didn’t try it out yet. But usually thing from rancher are awesome.
                        • da768 2 years ago
                          Last time I tried it, it was using sshfs mounts. Way slower than whatever Docker Desktop is using. Looks like sshfs is now unmaintained, I can't find what they're using now.
                      • Arbortheus 2 years ago
                        Podman has been a massive disappointment for me. I stupidly gave up my docker desktop because the company was trying to make cost savings and it was supposedly a drop-in replacement.

                        Some short-lived containers like our repo’s linter takes easily 4x as long to run in podman as it did with Docker. Immediately I have lost productivity.

                        It’s incredibly unreliable, every time I start my computer I have to podman machine stop then podman machine start because there’s something broken about how it gets initialised at startup. I’ve spent ages debugging random broken functionality.

                        It doesn’t support docker-compose. There’s a community project called podman-compose, but it’s not great because it won’t do stuff like build containers concurrently, and it has random weird quirks about volumes already existing when you do podman-compose up —-build whereas docker doesn’t complain for the same compose file.

                        Overall podman has been a massive regret for me, and I wish I hadn’t given up my docker desktop just to save a minuscule amount of money.

                        • sitta 2 years ago
                          When did you last use it? It _does_ support docker compose. Has for a while now actually.

                          I love it on Linux. The Mac version is not as smooth yet, but, for my use case, still works a hell of a lot better than docker desktop. There is something deeply wrong with docker desktop's networking, and I literally have to restart it almost every time I make a change to one of our services. Not an issue with podman.

                          • Arbortheus 2 years ago
                            Huh this was literally 2 weeks ago I looked into this and ended up using a third-party community podman docker compose substitute. I will have another look.
                            • andix 2 years ago
                              It does, but last time I tried „docker-compose build“ on podman it just failed. So it doesn’t seem to just work as a drop-in replacement
                            • andix 2 years ago
                              I haven’t used it much I have to admit. But I also experienced the issues you are talking about.

                              I thought it was because I also have docker desktop on the same machine, but probably that’s not an issue at all and it’s podman remote that is unreliable.

                            • null0ranje 2 years ago
                              I want podman to be a drop-in replacement, but it just isn’t there. I’ve wasted more time running down random errors I podman, so much so that I’ve switched back. Maybe in another year or two…
                              • laserlight 2 years ago
                                I've recently tried podman on a Mac and `podman machine start` failed with an “unknown error”. It was a disappointment.
                                • depereo 2 years ago
                                  I had similar issues when podman desktop first came out. Working much better on a re-try this week.
                              • jscheel 2 years ago
                                Docker is cripplingly slow on MacOS. I have a maxed out 16" mbp... starting rspec on our app takes 55-60 seconds. Compare that to my coworkers on Linux and Windows, who both see sub 10 second boots, and it's absolutely impossible to be ok with those numbers.
                                • rubyist5eva 2 years ago
                                  I have a nearly maxed out Mac Studio and my experience was the same, until I got my work to fork out a license for parallels and I just installed docker in an ubuntu VM and configured my DOCKER_HOST on the host to talk to the vm. Now it's crazy fast.
                                  • jscheel 2 years ago
                                    You are maintaining all your files in the VM though, right? I'm playing with setting up a sync from vm to host so I can do that but still use my local tools.
                                    • rubyist5eva 2 years ago
                                      IntelliJ autosyncs all my files to the VM seamlessly using rsync, yes. I've considered switching to use the remote development features in intellij and just managing everything in the VM and not having anything on the host but I haven't had any trouble with my current setup so I haven't bothered tbh.
                                    • chrisbolt 2 years ago
                                      How does file sharing performance compare?
                                      • rubyist5eva 2 years ago
                                        I automatically sync my workspace to the VM using IntelliJ and rsync so I don't mount any host filesystems in the VM for docker. I just use linux native volumes inside the vm itself and it works fine for me so far. It takes a bit of configuration from the IDE side but it works pretty seamless once it's setup.
                                    • navels 2 years ago
                                      Might be worth digging into why your rspec startup times are so slow. I recently dockerized our dev setup with separate containers for our rails backend, mysql, localstack, and rabbitmq. With Docker Desktop configured to use VirtioFS and the native virtualization framework (which is now I think the default), speeds are great. I've left my non-M1 coworkers in the dust.
                                      • jscheel 2 years ago
                                        It's not just rspec, any rails boot is slow. We have a pretty large project with a significant number of files and a significant number of gems, so the initial boot that loads all the constants etc is what does it.
                                      • kevinhq 2 years ago
                                        this is very true. not to mention it's very slow on its networking... i've given up docker on macOS long ago. If I was to use docker, I'll switch to my debian laptop. way much much faster.
                                        • tut-urut-utut 2 years ago
                                          Happy to give up Docker, but still using MacOS as a daily driver?
                                          • josephg 2 years ago
                                            Easy.

                                            Just do whatever you're doing in docker natively in macos. Python, Nodejs, Ruby, Rust, Postgres, etc can all run as native macos processes.

                                            The big advantage (aside from performance) is that you gain access to all the OS-native debugging capacities. You can just look at files, open multiple terminal sessions in the same folder, use Profiler, click the debug button without special configuration in your IDE, and so on. All without needing to think about VM images, docker containers, networking and all that rubbish.

                                            The downside is you need to set up a second build environment (which might not match your deployment environment). Unless you're doing something truly special, setting up a macos-native build environment is usually pretty easy. Its normally just a few "brew install" / "npm" / "gem install" / "cargo build" etc commands away from working.

                                            • kevinhq 2 years ago
                                              yes. it's not that i like macOS, but I need its xcode.
                                          • zenosmosis 2 years ago
                                            I use Docker inside of Parallels running Linux and get far better performance than Docker Desktop.
                                          • EdwardDiego 2 years ago
                                            My preferred fix: don't develop in Docker.
                                            • thunky 2 years ago
                                              My preferred fix: don't develop in Mac.
                                              • vforvendettador 2 years ago
                                                For many Mac is the only sensible option. There's one developer in our company that uses Linux and it's a lot of pain to setup.

                                                Mac has the best balance between coding, utility tools and "other work stuff".

                                                Windows probably on par if not more for "work stuff" but falls badly in the coding & tooling department.

                                                Linux is OK ish for coding and utility but falls behind for "other work stuff" and certainly a pain to just keep it updated.

                                                So in our company, everyone in the development & support team uses Mac (except this one guy who insisted in Linux), most in the sales & marketing team use Windows.

                                                • zamalek 2 years ago
                                                  > everyone in the development & support team uses Mac (except this one guy who insisted in Linux),

                                                  > certainly a pain to just keep it updated.

                                                  > Linux and it's a lot of pain to setup.

                                                  If you're not using Linux, how are you justifying these claims?

                                                  > certainly a pain to just keep it updated.

                                                  Excluding the boot time for both, MacOS takes between 15 and 45min to update. Linux is a few seconds. I suspect that the only OS that has a more ridiculous update process than MacOS is Gentoo.

                                                  • jeroenhd 2 years ago
                                                    Everyone in my company just uses Windows if they don't want to bother with Ubuntu. Excellent Linux tool chain support, excellent driver support, it just works.

                                                    WSL 2 is a game changer because it makes all the Linux centric dev tools available to Windows without setting up virtual machines or other such nonsense, even running graphical applications these days. The only major pain point I've run into (that isn't "I prefer Linux") is the lack of IPv6 support within WSL 2.

                                                    If you avoid buying Nvidia hardware, Linux generally "just works", unless you use Windows-only software (which macOS also suffers from) or choose to make your life harder by installing Arch or Gentoo. Ubuntu's snap is a pain for power users who want to hack on their Linux system but if all you want to do is develop or do work stuff, it just works out of the box.

                                                    • dgan 2 years ago
                                                      > There's one developer in our company that uses Linux and it's a lot of pain to setup.

                                                      There was one developer in our team using Mac. Everyone (>20 people) else were using Linux. Mac was a lot of pain to setup.

                                                      Linux is the best balance between coding, utility tools and "other work stuff".

                                                      Windows is not even considered for development, it falls badly in the coding & tooling department.

                                                      So in our company almost every dev uses Linux (except this one hipster using Mac) & marketing team use Windows

                                                      • 29athrowaway 2 years ago
                                                        And your production environment is Linux, so you will be using Linux anyways and all the testing you do on Mac has differences with production unless you do it in a VM where everything is slower.

                                                        Linux is not a pain to setup.

                                                        • rvdginste 2 years ago
                                                          Seriously, of all the things you can say about linux, you need to mention that it's a pain to just keep it updated?

                                                          I am a .net developer and run Debian linux on my work laptop since ages. Keeping the OS and most of the software up-to-date is just "apt-get update; apt-get dist-upgrade". Microsoft has Debian packages for teams, skype, powershell-core, .net (core), azure-cli. Google has Debian packages for chrome. I use a lot of JetBrains' tools and keep those up-to-date using the JetBrains ToolBox. Where I work, we use Google workspace and Slack and I use those through chrome.

                                                          Just to be clear, most of my development is currently done on Linux using Rider, but I do have a Windows VM (on KVM) for older projects that run on .NET full.

                                                          The issues I come across, are related to our customers. For example when I am on location and need to connect to external hardware. One example is to connect wirelessly to a WiFi Direct display: this does not work for me and I did not investigate if there are drivers available or not. Another example is DisplayLink to use an external display through a dock: this I checked and there are drivers available and I did have it working at some point, but it broke after a kernel upgrade and it's too much bother to fix it again. Also for some customers we can connect remotely to their systems over VPN, but not all VPN solutions are available (or work out of the box) on Linux.

                                                          In any case, for my day-to-day work, I don't have any issues at all on Linux and I believe it's very very capable for coding, tooling and other work stuff. I definitely prefer it above Windows and Mac.

                                                          • bmitc 2 years ago
                                                            > Mac has the best balance between coding, utility tools and "other work stuff".

                                                            That maybe used to be true. But today, Windows is that OS. In one OS, I can freely develop in Windows, Docker, WSL2, including near seamless integration of apps, browsers, and even GUIs. And with VS Code, I basically have any OS except macOS (but who cares?) at my fingertips in a single interface. The dev experience is by far the lowest friction between Windows and macOS.

                                                            And Windows has superior support for external hardware. macOS refuses to work well with anything that doesn't have Apple on the box.

                                                            • pjmlp 2 years ago
                                                              > Windows probably on par if not more for "work stuff" but falls badly in the coding & tooling department.

                                                              Given the option, I rather spend my day on VS than XCode, and C#/F#/C++ than Swift/Objective-C, but to each their own I guess.

                                                              And then there is the whole thing about where macOS Server end up.

                                                              • jiggawatts 2 years ago
                                                                > but falls badly in the coding & tooling department.

                                                                That's more perception than reality, often from people who simply don't know how to use Windows.

                                                                Visual Studio, Visual Studio Code, and IntelliJ IDEA blow any Linux text editor out of the water for developer productivity.

                                                                For Linux workloads there is the Windows Subsystem for Linux (WSL 2), which now even supports GUIs with GPU acceleration!

                                                                Visual Studio Code can even operate in "remote" mode where it tunnels into a Docker container or Linux server and acts as-if the remote target was the local machine.

                                                                On Windows, x86 and x64 Linux Docker containers run in process isolation at full speed, unlike on Macs where there is CPU emulation required.

                                                                • vips7L 2 years ago
                                                                  > Windows probably on par if not more for "work stuff" but falls badly in the coding & tooling department.

                                                                  This is absolutely false. Windows is a perfectly fine development environment and has perfectly fine tooling. You just need to embrace powershell, windows tooling, and use cross platform tools. Too many devs put themselves in a corner by relying on posix shell or posix only tooling.

                                                                  If you do user space programming your host OS should never matter. In my 10 years of programming the only time the host OS mattered was when I was writing Linux drivers.

                                                                  • paulmd 2 years ago
                                                                    My previous 2 companies have both switched to OSX for everyone. There are some teething problems - OSX is not really meant for a domain environment and JAMF Connect is necessary glue to work properly with Active Directory sorts of stuff, and it's still not quite perfect.

                                                                    But overall it's actually worked out surprisingly well because there's something for everyone - developers get *Nix On The Desktop but with an actual support story, and the non-technical users get a happy bubble OS that holds their hand.

                                                                    Linux code churn and distro fragmentation makes it fundamentally unsupportable in the vast majority of workplaces (outside very controlled server environments/etc - talking desktop use here) and for the vast majority of users. The code churn makes the support story (polish and documentation) impossible and the distro fragmentation means that there's 50 different solutions to the same problem. The Bazaar and the Cathedral doesn't mean the bazaar is better in all situations, a random non-technical business analyst is never going to learn how to build Arch or install Gentoo and a really good streamlined, polished Cathedral Experience is much more suitable to the business environment. That's the fundamental lesson from Linux and Windows and OSX now takes its place in that too. You can keep the good things about Unix-y environments and opt out of the terrible parts of the Linux ecosystem.

                                                                    Unfortunately, like BSDs, that's not what Docker is built around. Docker assumes a Linux kernel, and Linux kernel ABI is not the same as Unix kernel ABI. That's the biggest problem. Same as FreeBSD Jails or Solaris Zones... they're a decade ahead of docker in terms of capability, security, performance, and polish, but Docker is where the mindshare is. I can't install a jail from a registry with a single command and that's not where the support/development time is going even for the people who have engineered those alternative docker-registry solutions for jails.

                                                                    The only "fast" option for non-linux kernels besides full virtualization is to thunk the calls to your own kernel to patch around the differences. Obviously that didn't work out with the Windows kernel, it's just too different, but FreeBSD/Solaris have implemented this functionality for a long time as part of "Branded Zones". But everyone is enthusiastically rebuilding the wheel around ubuntu (specifically - not even linux generally) so that's not going to happen.

                                                                    https://wiki.freebsd.org/LinuxJails

                                                                    https://docs.freebsd.org/en/books/handbook/jails/

                                                                    (the freebsd handbook is a great example of the kinds of documentation that rarely gets written for linux distros - other than commercial ones - because of the overwhelming code churn and the inevitable bit-rot that entails in the rest of the user experience. It's way more fun to write a new audio pipeline or init system than to document it fully, everyone knows it.)

                                                                    https://www.oracle.com/technical-resources/articles/it-infra...

                                                                    https://docs.oracle.com/cd/E19455-01/817-1592/gchhy/index.ht...

                                                                    https://en.wikipedia.org/wiki/Solaris_Containers#Branded_zon...

                                                                    (and note the Solaris stuff almost entirely applies to OpenSolaris/Illumos as well, you don't have to use commercial solaris to get Branded Zones.)

                                                                    Anyway, apropos of nothing, but with the newfound attention on OS X from developers and power-users, it'd be really nice if Apple released a M1/M2-based "toughbook". Completely against their design aesthetic but I think a lot of people don't really like the idea of wafer-thin apple laptops and would like something that can take some bumps without shattering. Power users are becoming a more core demographic for macbooks and it'd be nice to see them cater a little more.

                                                                    • kevinhq 2 years ago
                                                                      very true. i saw this as well when i was still working in the office ten years ago. in Linux, what you can do is mostly coding and browsing.
                                                                    • princevegeta89 2 years ago
                                                                      Unfortunately Mac is the best in terms of hardware and portability and for compliance stuff
                                                                      • jrib 2 years ago
                                                                        yeah, I really wanted to buy a Linux laptop. I highly value battery life though and the m1 chip doesn't really have a competitor in that department.
                                                                      • EdwardDiego 2 years ago
                                                                        I agree, but my company is in a very regulated area, so they standarised on Macs because of the very strong controls built into them at a hardware level.

                                                                        I do miss my thinkpad and Fedora.

                                                                        • kevinhq 2 years ago
                                                                          this is hard. what if you're on project with

                                                                          - backend (docker) - needs linux based machine - client app (iOS) - needs xcode on macOS

                                                                          both are in one repository.

                                                                        • sieabahlpark 2 years ago
                                                                          Have fun managing osx and Linux dependencies then when you could just maintain one.
                                                                          • acdha 2 years ago
                                                                            That really hasn’t been an issue for me since Homebrew came out a decade ago - using Python, Rust, Node, Java, etc., all of which have mature stories for cross-platform development. The main area I’ve run into problems are legacy projects where it’s not “Linux” but more like “one old Linux distribution with a 32-but binary nobody can reproduce” and Docker is really the least troublesome part of those projects.
                                                                            • Spivak 2 years ago
                                                                              If you're not IT, god speed y'all, then it's not all that bad honestly. I set up my stuff with Ansible and 90% of the "porting" work was the mapping between rpms and brew packages.

                                                                              The only headache I get sometimes is because I have the GNU utils first in the path which makes compilation scripts mad sometimes.

                                                                            • pjmlp 2 years ago
                                                                              Indeed, one can maintain only a macOS application.
                                                                              • princevegeta89 2 years ago
                                                                                And also there is some additional fun dealing with broken or incompatible packages whenever you upgrade OSX
                                                                                • nicoburns 2 years ago
                                                                                  In the languages I work with (primarily JavaScript and Rust), cross-platform compatibility tends to be pretty much a non-issue. I develop on macOS, deploy on Linux, and it Just Works. No extra work required.
                                                                                  • nsonha 2 years ago
                                                                                    You forget to mention all backend services (db, redis etc are managed and you need internet connection and cloud credentials for even a development workflow.
                                                                                    • sieabahlpark 2 years ago
                                                                                      [dead]
                                                                                    • notwokeno 2 years ago
                                                                                      When I used to use a mac I just put all my "Linux" (GNU really) dependencies in a prefix and that worked pretty well. Docker is kind of overkill for what people use it for IMO.
                                                                                      • dhritzkiv 2 years ago
                                                                                        I used to be on the side of relying on native tools/libs, and managing them in a similar way to what you to describe, but it all became too much to handle, with dependencies across projects breaking with regularity.

                                                                                        Maybe I wasn't doing it right, but switching to Docker to sequester my projects and their dependencies has saved me so much time and hassle, especially with the amount of repos I work on throughout the year.

                                                                                        My biggest weakness today is that I still don't reach for Docker right away when starting work on a new project or when evaluating a new tool. Old habits…

                                                                                        • 2 years ago
                                                                                      • RockRobotRock 2 years ago
                                                                                        This is an unhelpful and useless comment
                                                                                        • mdavidn 2 years ago
                                                                                          And so is yours?

                                                                                          I do often find myself wondering whether Docker saved developers or system administrators any time. Is Docker really better than building an AMI and provisioning EC2 instances on-demand?

                                                                                          • jeppester 2 years ago
                                                                                            As a developer I can say that docker is saving me a lot of time that I would have otherwise spent on setting up different versions of postgres, redis, elastic search, etc. For the variety of client solutions we are building.

                                                                                            With a docker compose file in place, all I need to do is run "docker-compose" and everything is up and running.

                                                                                            It's such an upgrade over what we had before.

                                                                                            • RockRobotRock 2 years ago
                                                                                              Yes, I'm sure my wallet would be very happy with me running 20 different EC2 instances for small apps and databases.
                                                                                              • nsonha 2 years ago
                                                                                                You provide multiple cloud environments for every dev?
                                                                                          • TimTheTinker 2 years ago
                                                                                            My preferred fix: pay for a Parallels Pro license and run Ubuntu on a VM, then run docker there. The VM is configured to start on login and run in the background.

                                                                                            I have the Remote SSH plugin set up in VSCode, a `vmlogin` alias set up in bash, and all container ports forwarded in the VM's config.

                                                                                            • rubyist5eva 2 years ago
                                                                                              Can vouch for this approach, been using it for rails development for the past 6 months after my work swapped my macbook pro for an M1 Max Mac Studio and it's been solid.
                                                                                            • mccolin 2 years ago
                                                                                              With VirtioFS on the scene I just don’t have this experience anymore. Docker for Mac is significantly faster than it used to be, particularly when using named volumes.

                                                                                              Mutagen also improved the experience but I prefer VirtioFS as it’s “built-in”

                                                                                              • doctoboggan 2 years ago
                                                                                                What sort of workloads are people doing where the filesystem access is limiting them? I develop python web apps on a mac and use dockerized postgres and a dockerized flask app. I don't seem to experience any noticeable issues. When I am developing I mount the source code directory as a volume so code edits are synced live into the running docker container.

                                                                                                I also develop frontends using vue, managed by npm. In my experience this doesnt need to be dockerized since npm installs everything in a subdirectory per project. Is there a benefit to running this as a dockerized app?

                                                                                                • zeta0134 2 years ago
                                                                                                  One issue I ran into at my previous employer was pylint on a large Python codebase. Pylint is slow on the best of days, but the difference on an M1 Mac running under docker (to standardize the version and settings across the team) was something like 10x as slow; several *minutes* to lint the codebase, which we absolutely required before code could be checked in. It finally got a lot better when VirtioFS came out, which, when enabled on an arm64 image, sped up filesystem access dramatically; suddenly my lints were taking seconds again.
                                                                                                  • nyrikki 2 years ago
                                                                                                    Are your Docker images x86_64? On an M1 Mac x86_64 images run under qemu which is very slow, If an ARM64 image is available it should run a bit faster.
                                                                                                    • nijave 2 years ago
                                                                                                      Oh man x86 is so slow it's insane. If you're using a non-ARM base image Docker will happily run the x86 variant for you automatically. Unsurprisingly, running an x86 VM inside an ARM VM on a laptop is very very slow
                                                                                                    • nikanj 2 years ago
                                                                                                      Are you linting the whole codebase, or just files changed in the commit?
                                                                                                    • a_t48 2 years ago
                                                                                                      I work on a _large_ C++ codebase on Linux. If I'm on my main (Linux) machine, things are fine, bind mounts are okay. If I'm stuck on my MacBook then compilation performance is...bad. I suspect it's due to heavy filesystem access from the compiler (reading source, writing object files, etc). At some point I need to confirm this by copying in my source directory.
                                                                                                      • pmontra 2 years ago
                                                                                                        A customer's Mac with a M1 is only 50% faster than my Intel laptop from 2014 at running Rails tests, because they run in a docker container: 50s vs 75s. The difference between the two machines should be much more than that (CPU, RAM, data bus, etc.)
                                                                                                        • brodock 2 years ago
                                                                                                          You should try running your dependencies on docker but ruby on the host machine.
                                                                                                          • pmontra 2 years ago
                                                                                                            I'd be fine to run Ruby with asdf or rvm on my Linux laptop. I'm also fine to run it in a docker container. Performance is basically the same for me. The choice was made by my customer and it's them using Macs. They deploy in Linux containers though so that's probably why they accept not using all the performance of their hardware: same environment for production and development, no surprises.
                                                                                                        • dgunay 2 years ago
                                                                                                          I tried running our Golang mock generator through Docker on an M1 Mac and it was much slower than running it directly on the host. Probably since it reads every file in the codebase and writes out a file for every interface it finds.
                                                                                                          • latchkey 2 years ago
                                                                                                            mockery? That thing runs super slow just on its own.
                                                                                                          • xtracto 2 years ago
                                                                                                            Magento in docker on Mac is horrible for this reason.
                                                                                                            • mianos 2 years ago
                                                                                                              Mentioned in the article, node packages that create tens of thousands, sometimes hundreds of thousands of files.
                                                                                                              • Arbortheus 2 years ago
                                                                                                                My company has a large python repo. Anyone that develops in this repo on a Mac is considered a sadist because…

                                                                                                                Running Pylint on a Linux machine in docker: 3 hours from no cache

                                                                                                                Running Pylint on a Mac in docker: 9+ hours from no cache unless VirtioFS is used, which makes it closer to 4 hours.

                                                                                                                • jeffrom 2 years ago
                                                                                                                  Workloads with a lot of files, for example a large elixir web app with hot reload / fswatch enabled, have upwards of 20 second page load time. More than enough to mess up my flow.
                                                                                                                  • 2 years ago
                                                                                                                    • taf2 2 years ago
                                                                                                                      nginx is pretty unusable via docker - I guess file system cache is the issue…
                                                                                                                      • osrec 2 years ago
                                                                                                                        You can turn caching off, I believe.
                                                                                                                      • _-____-_ 2 years ago
                                                                                                                        [dead]
                                                                                                                      • tropshop 2 years ago
                                                                                                                        I have Apple Silicon but develop on docker x86. The game changer for me was macOS Ventura with rosetta support for linux vms.

                                                                                                                        I use UTM to run Debian 11 ARM. The update-binfmts command is absolutely magical, docker images will happily run both arm and x86 binaries.

                                                                                                                        Battery lasts all day and the machine stays ice-cold.

                                                                                                                        https://docs.getutm.app/advanced/rosetta/

                                                                                                                        • SirensOfTitan 2 years ago
                                                                                                                          The most recent version of Colima supports both virtiofs and mac's native virtualization framework (macOS 12.5+). I get totally acceptable performance using it.
                                                                                                                          • therealmarv 2 years ago
                                                                                                                            Is not this the main way to speed up Docker on Mac: use a beefed up Linux Virtual Machine (VirtualBox, UTM, tart) and run Docker inside this Virtual Machine?
                                                                                                                            • dijit 2 years ago
                                                                                                                              the 'DOCKER_HOST' variable (and the fact that all SDK's seem to support it) is honestly the greatest bloody thing in the entire ecosystem.

                                                                                                                              My workflow for the past 3 years with Docker has been: set up some desktop machine somewhere, configure docker, configure ssh like normal: set DOCKER_HOST=ssh://<tailscale_ip> on my laptop.

                                                                                                                              Docker responds as if it's local, but I get absurd build/fetch speedup (since the wired connection is faster than Wifi) and it's not running inside a slow VM.

                                                                                                                              Recently I've been using colima on my Mac natively, but I keep reaching for the DOCKER_HOST option.

                                                                                                                              • silverwind 2 years ago
                                                                                                                                I assume port forwarding would be a pain in such a setup, right?
                                                                                                                                • andix 2 years ago
                                                                                                                                  You just access everything on the docker machines IP instead of localhost. Dev servers may need an additional parameter for that, it i don’t see any big problems.
                                                                                                                                  • trillic 2 years ago
                                                                                                                                    The above commenter claims they're using Tailscale, a zero-conf VPN, which negates any port forwarding issues.
                                                                                                                                • acchow 2 years ago
                                                                                                                                  This assumes you want a distinct storage drive within your VM.

                                                                                                                                  Many developers prefer to code in their host OS but run the image via Docker for Mac. They also want instant real-time code changes to appear inside the running Docker image. I suppose you could have some of the disk live within the VM and the code portions be memory mapped or Rsynced. I haven’t thought through the downsides.

                                                                                                                                  • charrondev 2 years ago
                                                                                                                                    There was a project for that called docker sync (it could use of 3 mechanisms (one being rsync) to continually keep the files up to date in the container. The problem I found was it would just randomly stop syncing with no warnings or errors of any kind. It was very flaky.

                                                                                                                                    Nowadays make sure you use their new virtual machine thing in docker for Mac and add :cached in your compose file of any mounted volumes and I found that alleviated my issues. It used to be really bad though.

                                                                                                                                  • KyeRussell 2 years ago
                                                                                                                                    This doesn’t address the root problem. If you want the FS features that the “””native””” Docker Desktop provides, you end up with the same drawbacks re performance.
                                                                                                                                  • rcarmo 2 years ago
                                                                                                                                    I stopped using Docker desktop and just forwarded my Docker CLI to a VM’s TCP port the instant I found how it exported my Mac file system wholesale to the Docker VM. Never looked back, and these days I just use sshfs or VSCode to develop remotely (which works everywhere).
                                                                                                                                    • wood-porch 2 years ago
                                                                                                                                      A project I made solved this by running docker on AWS and doing two-way file sync on changes. Runs quite nicely and transparently.

                                                                                                                                      https://github.com/lime-green/remote-docker-aws

                                                                                                                                      Lots of benefits: speed, battery, fan noise

                                                                                                                                      • pak9rabid 2 years ago
                                                                                                                                        And money
                                                                                                                                        • trillic 2 years ago
                                                                                                                                          I use a shitbox of a thinkpad with my dev environment in the cloud, but they run Linux so I can just as easily run it locally if I have no connection. I buy them for $300-500 on eBay, upgrade them to at least 16gb of RAM and a 1TB Samsung SATA SSD. I have 2 cold spares ready to go, credentials loaded, just need to update and sync a few git repos. 14 inch 1080p IPS displays, i5 processor, 16GB of RAM.

                                                                                                                                          The money I save not paying for Apple laptops could pay for a crazy overpowered dev VM until the end of time.

                                                                                                                                          I used Apple laptops for about 10 years until about 5 keyboard replacements with the butterfly switch debacle.

                                                                                                                                          • speedgoose 2 years ago
                                                                                                                                            A MacBook is not that expensive and much faster than AWS VMs in the same price range (assuming a few years of usage) in my experience. The network storage of cloud VMs is particularly frustrating for development, unless you use ephemeral storage perhaps.
                                                                                                                                      • hakanderyal 2 years ago
                                                                                                                                        I finally gave up the effects of Docker on performance and battery life and switched to Windows. I still don't have a long lasting battery, but at least performance is better.
                                                                                                                                        • rad_gruchalski 2 years ago
                                                                                                                                          I have been working in a Citrix workspace Windows machine for the past 6 months with a Ubuntu 20 wsl2. I was very much anxious of the experience before onboarding but I have to say, I barely deal with the Windows side of things. Having said that, if I was to move away from mac, I‘d just go directly to Linux.
                                                                                                                                          • hakanderyal 2 years ago
                                                                                                                                            Gaming is my guilty pleasure. While the gaming ecosystem on Linux is getting better everyday (thanks, Steam!), the limited time I spend on gaming doesn't allow me to tinker, I just want everything to run without limitations.
                                                                                                                                        • SanjayMehta 2 years ago
                                                                                                                                          We gave up on Docker on macOS long ago. Wherever we absolutely need Docker, we just throw a Linux machine at it. Developer time is costlier than hardware.
                                                                                                                                          • xenoscopic 2 years ago
                                                                                                                                            If anyone has any questions about Mutagen (or integrating it into their Docker-based workflows), I'm happy to help.

                                                                                                                                            Just one clarification on the article: Mutagen offers Docker Compose integration, not Composer integration (Composer is a PHP package manager). However, as mentioned, DDEV is a great option if you're looking to do containerized PHP development while using Mutagen for performance improvements.

                                                                                                                                            • aliljet 2 years ago
                                                                                                                                              Have any of you tried to chase the microvm train instead of docker on MacOS? Thus far, it feels like this is a deeply lost cause with a passing hope that somehow you can hack a solution to nested virtualization and drop into a KVM-style experience on a guest vm and then go a layer deeper with microvms on top of that guest. Oofff. What an absolute horror show.
                                                                                                                                              • kevinhq 2 years ago
                                                                                                                                                > We had to abandon Docker because we had folks with macOS on the team. But, other tasks (email, conference calling, scanning, word, upgrading without breakage) came with more friction, and those tend to fill up ever larger shares of my day.

                                                                                                                                                source: https://kvz.io/macos-install.html

                                                                                                                                                • drcongo 2 years ago
                                                                                                                                                  I have a large, fairly complex Django application that I run in containers on a MBP with an M1 Pro, and it runs twice as fast as the AMD 16 core production server. It was a bit of a dog on an Intel Mac, but everything I use Docker for is blazing fast on Apple Silicon. Is there something about Python apps that makes this different?
                                                                                                                                                  • xwowsersx 2 years ago
                                                                                                                                                    I have "use virtualization" turned on in Docker Desktop for Mac, but I don't see different options for the file sharing implementation. I'm currently on version 4.14.1 (91661).

                                                                                                                                                    Separately, with "use virtualization" turned on, should I also enable "VirtioFS accelerated directory sharing"?

                                                                                                                                                    • ehutch79 2 years ago
                                                                                                                                                      It’s under experimental features in 4.14. It’s the default where available in 4.15.

                                                                                                                                                      You might need to upgrade both docker desktop and macOS.

                                                                                                                                                    • benfrancom 2 years ago
                                                                                                                                                      I’m interested to see how project finch works out compared to Docker on a Mac. It’s an open source client for container development.

                                                                                                                                                      https://github.com/runfinch

                                                                                                                                                      When I tried Rancher Desktop it didn’t work so well.

                                                                                                                                                      • totalhack 2 years ago
                                                                                                                                                        I fixed this problem by not getting a MacBook Pro this year when my last one from 2018 had no battery and two letters falling off the keyboard.

                                                                                                                                                        Bonus: paid half what a similarly spec'd M2 Air would cost.

                                                                                                                                                        • ffactory 2 years ago
                                                                                                                                                          Cool blog theme, feel like I've seen it before ;) https://filippo-orru.com/
                                                                                                                                                          • rootusrootus 2 years ago
                                                                                                                                                            Funny thing is that I switched from virtualbox for Vagrant on OSX over to Docker because vbox file operations were so incredibly slow.
                                                                                                                                                            • nsonha 2 years ago
                                                                                                                                                              Why would you want to mount node_modules? it's useless on the host because the binaries are for a different arch
                                                                                                                                                              • westoque 2 years ago
                                                                                                                                                                this is one of the reasons i changed my OS to linux since docker there works natively. i believe there will always be a bottleneck for peak performance on macOS until we are able to run natively, not virtualized.
                                                                                                                                                                • black_13 2 years ago
                                                                                                                                                                  [dead]
                                                                                                                                                                  • KyeRussell 2 years ago
                                                                                                                                                                    It’s no surprise that these replies have seen the usual jarring unfriendly personalities come out of the woodwork to advocate for the absolute use of desktop Linux.