$ for i in $(seq 3 16); do
        curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz
        tar xzf go1.$i.tgz go/bin/gofmt
        size=$(ls -l go/bin/gofmt | awk '{print $5}')
        strip go/bin/gofmt
        size2=$(ls -l go/bin/gofmt | awk '{print $5}')
        echo go1.$i $size $size2
    done
    
    go1.3  3496520 2528664
    go1.4² 14398336 13139184
    go1.5  3937888 2765696
    go1.6  3894568 2725376
    go1.7  3036195 1913704
    go1.8  3481554 2326760
    go1.9  3257829 2190792
    go1.10 3477807 2166536
    go1.11 3369391 2441288
    go1.12 3513529 2506632
    go1.13 3543823 2552632
    go1.14 3587746 2561208
    go1.15 3501176 2432248
    go1.16 3448663 2443736
    $ 

At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster).

¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends.

² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.

> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies.

That 70% includes the ELF/DWARF metadata that is easily removed from the binary using strip. It's true that the DWARF info in particular has gotten larger in recent releases, because we've included more information to make debuggers work better. I don't think it has grown quite as rapidly as the table indicates - I think some of the rows already have some of the debug metadata removed in "Raw size".

Regardless, I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all.

That is, the right comparison is to the "Stripped" column in the big table. If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead (debatable but not today), then the situation has actually improved markedly since Go 1.12, which would have been current in April 2019 when the first post was written.

Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%.

This is a marked-up copy of the table from the dr-knz.net revised post that at time of writing has not yet made it to cockroachlabs.com: https://swtch.com/tmp/cockroach-blog.png

I think the numbers in the table may be suspect in other ways, so I am NOT claiming that from Go 1.12 to Go 1.16 there has actually been a 15% reduction in "Go metadata overhead". I honestly don't know one way or the other without spending a lot more time looking into this.

But supposing we accept for sake of argument that the byte counts in the table are valid, they do not support the text or the title of the post. In fact, they tell the opposite story: the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes.

Over on Lobsters, zeebo took the time (thanks!) to build the same version of CockroachDB with various historical Go toolchains, varying only the toolchain, and found that if anything the Go executables are getting smaller over time, in some cases significantly so.

            v1.0          v20.2.0
    1.8     58,099,688    n/a
    1.9     57,897,616    314,191,032
    1.10    57,722,520    313,669,616
    1.11    48,961,712    233,170,304
    1.12    52,440,168    236,192,600
    1.13    50,844,048    214,373,144
    1.14    50,527,320    212,699,656
    1.15    47,910,360    201,391,416

The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time.

You do NOT write an article talking about "dark bytes" and a 70% "not useful" piece of binary for an Open Source project.

Specially one such as Golang where you can easily reach out to the creators at https://groups.google.com/g/golang-dev to ask for help understanding what your experiment tools can't explain...

Eg. """ Hey guys I am doing some digging with "such and such" tools and found that 70% of my binary size has no explanation. BTW this binary is Cockroach DB, an well known program in the Go ecosystem. How can I find out what are those bytes for? Is there any documentation about it, other than the source code itself? Or maybe some tooling that explains it? Or some other set of tools I should be using instead or on top of my current setup? """

I mean... there is no defense possible after writing such an article IMHO. Why didn't you ask first?

Having said that, when you deviate from the ubiquitous C standards for calling conventions and binary format layout, that comes with a price. I totally get that Go had to take such path, because Go binaries embed a non trivial runtime dealing, at the very least, with goroutines & GC support. But the price is there nonetheless: As you are not using a standard, you cannot rely on pre-existing tooling or documentation. You have to write your own.

It is understandable that writing such binary dissection documentation or tooling (or both) might not be a top priority compared to other more pressing matters for the Go Language. But this kind of thing becomes technical debt and as such, the longer you take to pay it, the more expensive it becomes. For instance: - This happened, and might have damaged the Go Language reputation, at least temporarily for some people. - I am surprised how could the Go team (or community) achieve improvements at all in the binary size work without tools to properly dissect binaries or detailed documentation. I am guessing it relied on the original programmers of those bits to have a very clear idea of the binary layout uploaded in their brains, so they did not need to re-grok all the sources again... with time that becomes more and more difficult, people forget stuff, move to other projects, etc.

I think it would be wise to prioritize this somehow. If the Go team cannot take it for now, maybe they can promote it as a good Go community project to be done ASAP, and promise to help the takers with all info and reviews needed.

Or maybe the article writers should take up the challenge, to fully redeem themselves ;-p

yes this is acknowledged in the OP

> We (the Go team) did not “recompress” pclntab in Go 1.15.

There's now less redundancy in the funcdata, so in my book less redundancy = more compression.

> We did not remove pclntab in Go 1.16.

Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more.

Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon)

> we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote.

Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph.

> If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text.

2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes.

I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability.

> The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version.

Correct. Added a note to emphasize this fact.

> Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful.

Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon":

https://ieeexplore.ieee.org/abstract/document/6307773

We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function.

I'm all for trade-offs, but IMHO they should be transparent.

Might it be possible to stream binaries or to detect junks which could be unloaded like an json parser which is only needed when reading json

The DWARF data is currently so represented, and so was pclntab prior to 1.16.

Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown.

> If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

Probably not if all of it was removed at once.

It could be that just 5% of it is useful and removing all of it would produce a non-working binary. What does the experiment prove exactly?

Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.

As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.

I think it is exactly this mindset that caused the current situation (that the 2/3 of the compiled binaries are useless to the users).

I guess it is just a tradition with Go, rediscovering history.

(This is derived from Russ' discussion above.)

CPython doesn't generate anything, and the binary installers (including docs and all) are about 30MB. Hell, the 64b embeddable package for windows (https://docs.python.org/3/using/windows.html#the-embeddable-...) is 16MB uncompressed.

I know for a good long time Go wasn't even stripping out unused functions, e.g., if you used any of the "strings" standard module you got the whole thing compiled in. I don't know if that's still the case, but it would be another source of size problems.

I'm also not sure why you're talking about init time; binary size doesn't necessarily factor in to that very much. The things that make the Go binaries large also tend to lean in favor of faster initialization anyhow. A lot of these binaries won't even necessarily be read in to memory, or have to stay there in the event of swap pressure, since they're already disk-backed memory pages that can be never loaded in the first place, or cheaply evicted by the virtual memory system if space is needed.

However, once you exceed the 10-20Mb regime, I seem to find more 20-200Mb Go bins in the wild. This can be misleading, since the Go world is a fan of packing assets into binaries, eg for web GUIs.

What dings Go bins on init time is the Go runtime. Still, it's rarely noticeable unless you are firing thousands of times a second.

Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. That said, the bottleneck to fast deployments isn't the binary starting up, but the machine pulling the executable artifact, so the binary sizes do matter to me. That said, very often these executable artifacts are a Docker image, which tend to come with a lot more bloat than one will find in a Go binary, so step 1 is getting your Go binary on a scratch image.

Go is cheating a bit on this one by heavily caching everything. Building in fresh container is quite slow (ok, maybe not c++ slow but still much slower then C).

Always having to build anything due to static linking does not help either.

I don't think there's an easy either-or answer to questions like this, but broadly language design seems to be about finding a sweet spot while balancing lots of trade offs, so to get fast compiles you're going to make trade-offs in other areas of the language and implementation. I imagine if the compilation time budget was higher there'd be some more space for binary pruning passes.

It mirrors their GC philosophy.

[1]: https://github.com/golang/go/issues/6853

- Go's design, based on both C compilers and Inferno's limbo

- /proc

- utf-8

- 9p