Self-Documenting Makefiles

23 Jan, 2024

This post is for the real ones: Anyone still using make in 2024.

What kind of nerd still uses Makefiles?

If anything, I'm using make more in the past couple of years than I ever have before. A Makefile is one of the first things I create when I'm starting a new project, and I'm using them for basically every single codebase or technical task I'm working on.

For example:

• Managing and orchestrating Python projects.
• Building, publishing, and adding new content to this blog.
• As one layer of the ops tooling stack at my day job.

The obvious advantage of using a build automation tool (such as make/rake/invoke) is that it automates things you would normally have to do manually, and it allows you to decompose the work you need to do into reusable components that can depend on each other.

One not-so-obvious advantage is that the tool itself serves as documentation of the things you might need to do more than once. For example, after a long time away from this blog, I often forget the exact commands to run a local hugo server, add new content, or (especially) to push the content to S3 and issue an AWS CloudFront invalidation. No worries, though: I've added Makefile rules to handle each one of those things.

But what is even in this thing?

After awhile, though, you've got a Makefile full of rules, and you're not sure you even remember all of the things your Makefile can do.

You could document everything and put it in a make help rule, but now you've got to remember to update the text you've manually written in the help rule whenever you add/remove/change something in one of the other rules.

Unless...

Solution: A self-documenting help rule

There is a better way: A one-liner that you can stick into a help rule that will automatically pull out all rules in your Makefile, including the text of a special comment that you've added to the rule itself.

Here's how I start every single Makefile I write:

.PHONY: help

help: ## Display this help screen
	@echo
	@echo "Usage:"
	@echo
	@sed -n 's/^\([A-Za-z0-9_.-]*\):.*## \(.*\)$$/\t\1: \2/p' Makefile | sort | column -t -s ':'
	@echo

This help rule will automatically list all of the rules in your Makefile, in alphabetical order, and it will include any comments after the Makefile rule that start with ## as the help text for that rule in the make help output.

The result is nicely-aligned output, such as this output from the Makefile I use to manage this blog:

$ make help

Usage:
	clean help new publish run
	Clean up resources created by Hugo
	Display this help screen
	Create a new post
	Upload files to S3 and issue a CloudFront invalidation Run a local development server, showing all draft content

Explanation

We're about to get down in the weeds to understand what's going on, so if you're just here for that sweet, sweet copy-paste action, feel free to stop reading.

The two main things this rule does is (1) match and transform text with sed, and (2) sort and table-align the resulting output with sort and column.

Using sed to capture and reformat the text we care about

I'm not affiliated, but the best tool for learning, experimenting with, and explaining regexes that I've ever seen is RegExr.com. Check out the interactive example on RegExr.com for the regex I used above. The Explain tab on the bottom will tell you the meaning of each part of the regex, and the Details tab will show you the contents matched by capture groups 1 and 2.

A few ways the regex in my snippet above is different from what's in the RegExr example:

• We're using sed, so the regex engine is slightly different: We need to escape all ( as \(.
• We're in a Makefile, where $ has a special meaning. We need to escape the dollar sign as $$ to pass it through to the regex.

Other notes:

• \1 and \2 in the sed replacement string are references to the text matched/captured by the capture groups. Check out this page if you're not familiar with capture groups.
• sed will default to printing all lines from the input file, but we only want the transformed output of lines that matched our regex. Passing -n to sed will disable printing of every single line, and then the trailing /p in our match/replace string will tell sed to print the output that was matched and transformed.

Using sort and column to give us nicely-formatted output

sort will put the resulting lines in alphabetical order, then column will give us the nice formatting in the screenshot above. The two key arguments to column are:

• -t: Use table alignment
• -s ':': Delineate table columns by the colon character

FAQ

Why didn't you use -r to make sed use extended syntax?

I know, extended syntax means that you don't need to do things like escape parentheses. But extended syntax is off by default, and the flag to turn it on is not consistent across platforms (it's -r on Linux, but -E on MacOS and OpenBSD). Because I use my Makefiles to manage cross-OS compiles of binary Python extensions on Linux, MacOS, and (yes!) Windows, I try to minimize the use of anything platform-specific.

In your regex, you use *, but shouldn't you really use +?

Probably, but + is part of the sed extended regex syntax, which I intentionally avoid (see the above answer).

Why are you setting the default rule to help instead of having it compile source code, you heretic?

In my world, all of my use cases for make center around orchestrating and automating a series of operational activities, and then exposing them in a way that is easy for a human or a build system to execute. Personally, I also like the pattern of a command with no arguments giving you a help message for how to use that command.

If you're delivering open source software and your Makefile is provided to end users so they can compile and install your software, then please do stick the existing familiar patterns.

Why are you still using make, gramps? Haven't you heard of rake/invoke/task/ninja/just/ whatever?

Maybe someday I'll expand this answer into its own post, but for now the main reasons are:

• It's ubiquitous: It's already present on almost every environment I work in (even Windows-based GitHub Action runners!), so I don't need to install it.
• It sits above the rest of the dev tooling: Many of my use cases involve bootstrapping a build or development environment, and I don't want to have orchestration steps that a human has to remember before they can start using the orchestration tooling (such as pip install invoke).
• It's stable, and it's stood the test of time. I don't need to worry about the tool being abandoned, or about breaking changes happening due to rapid development. I don't have to worry about keeping up with the latest fads in build orchestration tooling because make is like a cockroach in a nuclear apocalypse: When everything else has faded away, make is still there (no matter how much you hate it).

#software