Makefiles for Go Programs

[jlfo...@berkeley.edu]

I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the overhead of just always compiling and linking everything?

One piece of evidence for this is that the Go compiler leaves no artifacts, like object files, so as is make wouldn't fit into the current build method.

Cordially,

Jon Forrest

[Kurtis Rader]

On Sun, Aug 22, 2021 at 8:11 PM jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote:

I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the overhead of just always compiling and linking everything?

One piece of evidence for this is that the Go compiler leaves no artifacts, like object files, so as is make wouldn't fit into the current build method.

I have no idea how many projects using Go also use makefiles. The sole Go project I contribute changes to does have a makefile. See the Elvish shell project: https://github.com/elves/elvish/. It does so for a few reasons:

1) There should be a simple mechanism to executed several tasks that don't involve the "go" command.

2) The flags used for unit tests (e.g., whether to enable race detection) varies by platform.

3) Building the binary for installation involves passing "-ldflags" arguments built from running shell commands.

It's not clear if your use of "program" was meant to be read as "project". If you did mean something like "individual program" then, yes, I have no doubt that scenario is probably unlikely to have a makefile.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[David Riley]

On Aug 22, 2021, at 23:11, jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote:
>
> 

> I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the overhead of just always compiling and linking everything?
> One piece of evidence for this is that the Go compiler leaves no artifacts, like object files, so as is make wouldn't fit into the current build method.

I use Makefiles in our own internal projects because it allows me to establish a lot of good tooling around them, especially for repos which build a lot of binaries, Dockerize them, etc. It’s quite useful, despite the lack of intermediate files which Make tends to rely on to determine whether to rebuild things. Among other things, Make deals a lot better with lists of files and the concept of prerequisite steps than e.g. Bourne shell, and the Make package is actually considerably smaller than Bash in most tiny distros like Alpine, which is a consideration for Dockerfiles.

Unfortunately, to make Make worth the effort in Go projects, you need to understand it, otherwise you’ll just wind up with a lot of independent targets with a lot of repetitive code. Many developers these days (especially newer ones) have never been exposed to Make other than just running it and aren’t aware of what it can do, and so tend not to understand why you might not use it instead of e.g. Bash (they also tend not to understand why it might not be a good idea to use Bashisms instead of vanilla Bourne shell, but that’s a conversation for another time).

Basically, Make can make your Go projects a lot more intuitive to build, especially when it comes to things like default flags and consistency with CI, but you do have to put some effort into it, and you have to get people over the “Make is hard” mindset or they will “fix” your project by getting rid of it, IME.

FWIW, our default “make test” target generally just does “go test -race ./…”, with some optional coverage arguments spliced in when you “make cover”. “make cover-report” is where Make starts to shine; there’s a file prerequisite rule defined for the coverage file as well as for the HTML coverage report, and “make cover-report” just says to build the report file and Make’s dependency resolution magically takes care of the rest. It’s actually a pretty good basic demonstration of capabilities that used to be considered standard knowledge.


- Dave

[jake...@gmail.com]

On Sunday, August 22, 2021 at 11:11:23 PM UTC-4 jlfo...@berkeley.edu wrote:

I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the overhead of just always compiling and linking everything?

 

Go had built in build caching. So it will not have to "always compile and link everything." In addition, Go builds tend to be much, much faster than most other compiled languages. As a result, Makefiles are only useful if you have other operations to perform that are not part of the normal go tooling.

[Roland Müller]

What are the alternatives to Makefile that are used by Go developers? Please comment :-)

I know Make from old C/C++ times. Therefore, my picture is that it is not very portable and requires for quite many operations the usage of external tools that again differ between the platforms. Basically Makefiles are somehow enhanced shell scripts (Linux/Unix) or batch files (Windows).

Currently, at work I deal a lot with Maven, that is a bit too Java -oriented in spite of being capable in principle to build and compile other things too. Another, issue is the XML syntax that's makes editing without tool support very hard.

Gradle would be another candidate. I am just began to explore it. It's a bit like Maven with human syntax, but lacks again on lifecycle support that I like with Maven.

BR,

Roland

 

Cordially,

Jon Forrest

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/449781bc-1605-4acf-b03c-dab98cf710efn%40googlegroups.com.

[Wojciech S. Czarnecki]

Dnia 2021-08-23, o godz. 19:47:47
Roland Müller <rol...@gmail.com> napisał(a):

> What are the alternatives to Makefile that are used by Go developers?

https://github.com/magefile/mage

yw.

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Sergey Matveev]

*** Roland Müller [2021-08-23 19:47]:

>What are the alternatives to Makefile that are used by Go developers?
>Please comment :-)

The best thing I saw, that literally completely changed my life is DJB's
redo build system. I replaced everything related to Makefile in all my
projects, gaining simplicity, reliable builds, huge convenience,
portability, good parallelizability and so on, so on. There are various
redo's implementation and redo-inspired creations, but I have created my
own, written on Go, the best one I saw from features and performance
point of view: http://www.goredo.cypherpunks.ru/
https://fzakaria.com/2020/06/08/my-love-letter-to-redo.html
https://apenwarr.ca/log/20101214
https://redo.readthedocs.io/en/latest/
http://jdebp.uk/FGA/introduction-to-redo.html
https://habr.com/ru/post/517490/ (my post on russian)
Nothing compares to its simplicity and still covering *all* Makefile's
use-cases and many more. However Go takes over much work out-of-box, so
there is not need to make rules for .o/.h/CFLAGS/whatever targets.

--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF

[David Riley]

On Aug 23, 2021, at 12:48, Roland Müller <rol...@gmail.com> wrote:

What are the alternatives to Makefile that are used by Go developers? Please comment :-)

Well, there’s mage, which aims to more or less replace the functionality of Make for Go. I’m not really sold on *needing* a replacement for Make, and if you’re doing CI builds, it still adds an external dependency, but it is an interesting project: https://magefile.org/

I know Make from old C/C++ times. Therefore, my picture is that it is not very portable and requires for quite many operations the usage of external tools that again differ between the platforms. Basically Makefiles are somehow enhanced shell scripts (Linux/Unix) or batch files (Windows).

Makefiles are quite portable, at least assuming you’re using GNU Make (which is at least available nearly everywhere). It may not be the most ideal option with Windows, but nearly everywhere else it’s pretty solid.

If you have problems with external tools behaving differently across platforms (the behavior of “which” on Solaris vs. Linux or BSD being a particular sticking point I’ve run across in scripts), I would argue that there’s not much out there that’s going to solve that problem.

Currently, at work I deal a lot with Maven, that is a bit too Java -oriented in spite of being capable in principle to build and compile other things too. Another, issue is the XML syntax that's makes editing without tool support very hard.

Most of the newer build tools like Maven, Gradle and Bazel seem to be more oriented towards either IDEs or large-scale projects. Make scales quite nicely to small, and moderately well to large. Recursive builds tend to be a problem, but fortunately with Go, you don’t tend to need those.

I would say Go tooling goes along rather well with Make if you’re following the semi-canonical repo structure, because you can tell Go to just build a list of executables from the ./cmd directory and the build tool takes care of caching, figuring out dependencies, etc. Not much in the way of portability issues there.

Gradle would be another candidate. I am just began to explore it. It's a bit like Maven with human syntax, but lacks again on lifecycle support that I like with Maven.

I feel like Gradle is another very Java-oriented tool, and as a consequence seems to have inherited the very Byzantine nature of nearly every other Java ecosystem tool. I haven’t tried to use it for non-Java stuff, but I wouldn’t, personally. Not least because in a CI environment, I tend to try to stick to things either native to the language I’m using (so, the native Go build tools, *maybe* mage), or things present or easily installed in the host Docker image (both Bourne shell and Make fit this bill nicely).

The other benefit here is that in the projects I work on for work, not everyone wants to use Make (some folks have a pathological aversion to it), but it’s easy for us to make sure that Make is only ever a convenience method for things that can otherwise be easily done from the command-line (e.g. “go test ./…”). Make then becomes a) a convenience for running basic things (e.g. make test, make cover, make docker-test-race, that sort of thing) and b) a method for making sure our developers are running the same commands locally that the CI process does (don’t underestimate the importance of that for avoiding difficult-to-diagnose problems).

It’s also nothing you can’t do with plain shell scripts, but you’ll find yourself reinventing a lot of things that Make does for you quite nicely out of the box, like default parameters, list handling, target dependencies, etc.

- Dave

[Amnon]

I think the basic idea is that Go projects do not have makefiles because they do not need makefiles.
Ideally the Go command does the right thing, including fetching and building dependencies, and building 
entire trees of projects. Go is opinionated, and dictates where each package can be found on the filesystem.
Having a fixed convention means that configuration is not necessary. And the Go command usually does this 
better than hacked together make-files. The dependency analysis and object caching just work, so we don't 
spend our lives running `make clean; make distclean; make cleanall; make archconfig; make` 
each time our build fails to pick up the latest changes.

In the real world we do need to build docker images, and non-Go artifacts, so we do often have to fall back on some
sort of Makefile. But it is refreshing not to having each time to re-invent how to build a project spanning multiple directories.

[Iago Rubio]

> On 23 Aug 2021, at 05:11, jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote:
>
> 
> I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the of just always compiling and linking everything?
Most likely, it’s because go comes with it’s own build system built-in.

Just using “go build whatever.go” will do the job.

Even the cross-compilation is built -in.

[Connor Kuehl]

> On Aug 22, 2021, at 10:11 PM, jlfo...@berkeley.edu <jlfo...@berkeley.edu> wrote:
>
>
> I've noticed that few, if any, Go programs use Makefiles. Is that because the overhead of using make is greater than the overhead of just always compiling and linking everything?
> One piece of evidence for this is that the Go compiler leaves no artifacts, like object files, so as is make wouldn't fit into the current build method.
>

I started using a Makefile for my Go project so that I could inject a version string at build time without having to remember what to pass to go binaries.

I made a Discord bot to use in a server with my friends, and since I have some automation in place to automatically deploy the tip of my main branch, I thought it’d be convenient to be able to ask the bot what build it’s running so we can see which commits are “live."

So while it’s not a “traditional” use of make, it certainly is convenient.

Connor

P.S., here’s the Makefile. The important bits are the lines that mention “LD_FLAGS"

VERSION := v2.2.0+dev
BUILD := $(shell git describe --tags 2>/dev/null || echo "$(VERSION)")

LD_FLAGS := "-X 'main.Version=$(BUILD)'"

SOURCES := $(shell find . -type f -name '*.go')
SOURCES += go.mod go.sum

.PHONY: build clean test

all: popple

popple: build

build: $(SOURCES)
@go build -v -ldflags=$(LD_FLAGS) ./...

test: popple
@go test -v -ldflags=$(LD_FLAGS) ./...

clean:
@rm -rf popple

[Michael Ellis]

Three cheers for mage.  It's more verbose than make but it's pure Go. I use it to build and test projects that include generated html/css/js supported by a Web Assembly clients communicating with a server.

[Reto]

On Mon, Aug 23, 2021 at 06:12:43PM -0700, Michael Ellis wrote:
> Three cheers for mage <https://magefile.org/>. It's more verbose than make

> but it's pure Go. I use it to build and test projects that include
> generated html/css/js supported by a Web Assembly clients communicating
> with a server.

It may be nicer, however the beauty of make is that it is ubiquitous,
everybody already has it.

With mage, your installation instructions now need to contain how /
where to get mage in the first place, leaving the user with a random
binary somewhere which they probably never ever need nor update again.

[Wojciech S. Czarnecki]

Dnia 2021-08-24, o godz. 07:54:35
Reto <re...@labrat.space> napisał(a):

> It may be nicer, however the beauty of make is that it is ubiquitous,
> everybody already has it.

No, not everybody has it. Esp. on Windows host.

> With mage, your installation instructions now need to contain how /
> where to get mage in the first place, leaving the user with a random
> binary somewhere which they probably never ever need nor update again.

Nope. If user is going to compile from sources she also needs to install Go compiler and tools. Once she has it the mage is a `go install` command away. Shorter than typical line of "prerequisites" needed by non-developer user on most of linux distros. (If she is about to install binary she does not need mage at all - 99% deployments of Go based apps are just the single executable.)

> leaving the user with ... they probably never ever need nor update again.

Make does not come alone for end-users. Usually it comes bundled with several hundreds of megabytes of the devel packages
user gets after issuing eg. `sudo apt-get install build-essential`. Then she's left with a bit more more unused stuff than a single
mage binary.

TC,

[Stephen Illingworth]

I use Makefiles with Go projects. For many of the reasons already given but it's especially useful for me for cross compilation to the target platforms at release time. So, when running "make release" I have "test" and "generate" targets as dependencies to make sure I've not missed anything.

I don't need a build system to do that but it's good to have a script for that kind of thing, IMO.

I also use it in the git pre-commit hook. For example, I have "make lint" in that script which runs whatever linter I'm using at the moment and I can be assured that it's the same linter I'm using at the command line.

As for why not another build system, for the same reason as other respondents have said. Make has been around forever and I don't need anything more sophisticated.

> Make does not come alone for end-users. Usually it comes bundled with several hundreds of megabytes of > the devel packages  user gets after issuing eg. `sudo apt-get install build-essential`.

Surely, you would just do: `sudo apt-get install make`

[Roland Müller]

Hello,

A big thank you to everybody who contributed to this thread! I guess it will take some time to process the things learned from this discussion.

BR,

Roland

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/039b0cac-a81c-40ae-bb46-f12d11261707n%40googlegroups.com.