I recently set up a Raspberry Pi K3s cluster at home, to escape having to pay cloud services for hosting my myriad of half-finished side projects. Something I didn't consider, perhaps naively, was the differing CPU architecture of a RasPi to my usual development environment. Enter a Sunday afternoon of learning all about using
docker buildx to compile multiarch Golang binaries and Docker images!
The project I was trying to deploy was a wee Slackbot, written in Go on my laptop with its Intel processor - not an ARM chip like the Raspberry Pis.
Go itself has quite a nice cross-compilation story: passing environment variables to the compiler such as
GOOS works seamlessly. For example, I was able to generate a multitude of working binaries pretty easily:
$ CGO_ENABLED=0 GOARCH=amd64 go build -o victim_amd64 $ CGO_ENABLED=0 GOARCH=arm GOARM=7 go build -o victim_armv7 $ CGO_ENABLED=0 GOARCH=arm64 go build -o victim_arm64 $ CGO_ENABLED=0 GOOS=windows go build -o victim.exe
My Sunday would have been far easier if I coud have left it there: throw together a binary with the required flags,
COPY it to a Docker image built
FROM scratch so that it should run on any architecture, and go drink a margarita or six.
But, frankly, that's not good enough, for two reasons.
First, and most importantly, I am fed up of trying to remember how I deployed a thing months or years later - that's bitten me a hundred times, easy. I wanted the whole process automated. Secondly, I wanted the Docker build step to also compile the Go application - both to be more sure that builds are reproducible regardless of the Docker host machine, as well as, again, to eliminate as many manual steps from the process as possible.
go build with various flags isn't tricky by any means, but it's an extra step on top of
docker build, and besides, I can't remember the flags I need to pass to enable cross-compilation to ARM on a good day, let alone in six months' time.
So first, let's get a working, cross-compile-able Docker image sorted out.
Knowing I wanted a small image, and knowing that images built
FROM scratch tend to behave pretty much anywhere, I wanted to use a multi-stage build right out of the gate. I started with something like the following:
FROM golang:1.16-alpine AS builder COPY . . RUN go mod download RUN CGO_ENABLED=0 GOARCH=armv7 go build -a -o /victim # --------- FROM scratch COPY --from=builder /victim /victim CMD [ "/victim" ]
While iterating on the above did produce results, and the resulting Docker image was around 5MB in size, I wasn't super happy with having the architecture hard coded in there. This meant I'd need to run several builds to account for different architectures, and then do a bunch of wrangling to get all those images into the same manifest.
Turns out Docker themselves weren't happy with that state of affairs either, so they created
buildx, which skips all the tedious manual steps and creates a single multiarch image for you. And they even pass build arguments into your image, which I knew I'd be able to wrangle into the golang environment variables I needed:
A list of build arguments likeDocker - Working with Buildx
TARGETPLATFORMis available automatically inside your Dockerfile and can be leveraged by the processes running as part of your build:
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" > /log
Luckily, there's no need to go through the tedious process of manipulating these into the
GO* compiler flags - some kind soul has done the work for us in this project. So our final Dockerfile for dynamic cross-compilation with buildx becomes:
FROM --platform=$BUILDPLATFORM golang:1.16-alpine AS builder # Parses TARGETPLATFORM and converts it to GOOS, GOARCH, and GOARM. COPY --from=tonistiigi/xx:golang / / ARG TARGETPLATFORM COPY . . RUN go mod download # Build using GOOS, GOARCH, and GOARM RUN CGO_ENABLED=0 go build -a -o /victim # --------- FROM scratch COPY --from=builder /etc/passwd /etc/passwd COPY --from=builder /etc/group /etc/group COPY --from=builder /victim /victim EXPOSE 8080 USER guest CMD [ "/victim" ]
And can simply be built with
But that's still not good enough! I want zero commands to remember, and as much automation as possible. Time to configure Github Actions to do all this for me.
I've very little experience with Github Actions still - we use Gitlab at work, and its CI suite works quite differently - but was quickly able to find a buildx action that looks like the following:
- name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Build and push uses: docker/build-push-action@v2 with: context: . platforms: linux/arm/v7,linux/arm64/v8,linux/amd64 push: true tags: kieranajp/victim:latest
This will, in one fell swoop, cross-compile our multiarch docker image, set up a manifest, tag everything accordingly, and push our resulting image up to Docker Hub. Sweet!
A sprinkling of Helm chart later, and we have our running application on the Raspi cluster!
$ kubectl get pod NAME READY STATUS RESTARTS AGE victim-55cdf87c48-9qzl9 1/1 Running 0 10m
If you want to see the whole application with its multi-arch build and deploy setup, it's open source at kieranajp/victim. The app itself is just a simple Slack chatbot that solves a problem my partner Edele was facing at her work - but I learnt a lot while trying to get it deployed, and I'll be using this method to deploy a bunch of other stuff soon!