Site logo Kieran Patel

Adventures in cross-compilation

Using Github Actions to cross-compile and containerise a Go app, so it runs in K3s on a Raspberry Pi.

I recently set up a Raspberry Pi K3s cluster at home, to escape having to pay cloud services for hosting my myriad of half-finished side projects. Something I didn't consider, perhaps naively, was the differing CPU architecture of a RasPi to my usual development environment. Enter a Sunday afternoon of learning all about using docker buildx to compile multiarch Golang binaries and Docker images!

The project I was trying to deploy was a wee Slackbot, written in Go on my laptop with its Intel processor - not an ARM chip like the Raspberry Pis.

Go itself has quite a nice cross-compilation story: passing environment variables to the compiler such as GOARCH and GOOS works seamlessly. For example, I was able to generate a multitude of working binaries pretty easily:

$ CGO_ENABLED=0 GOARCH=amd64 go build -o victim_amd64
$ CGO_ENABLED=0 GOARCH=arm GOARM=7 go build -o victim_armv7
$ CGO_ENABLED=0 GOARCH=arm64 go build -o victim_arm64
$ CGO_ENABLED=0 GOOS=windows go build -o victim.exe

My Sunday would have been far easier if I coud have left it there: throw together a binary with the required flags, COPY it to a Docker image built FROM scratch so that it should run on any architecture, and go drink a margarita or six.

But, frankly, that's not good enough, for two reasons.

First, and most importantly, I am fed up of trying to remember how I deployed a thing months or years later - that's bitten me a hundred times, easy. I wanted the whole process automated. Secondly, I wanted the Docker build step to also compile the Go application - both to be more sure that builds are reproducible regardless of the Docker host machine, as well as, again, to eliminate as many manual steps from the process as possible.

Sure, running go build with various flags isn't tricky by any means, but it's an extra step on top of docker build, and besides, I can't remember the flags I need to pass to enable cross-compilation to ARM on a good day, let alone in six months' time.

So first, let's get a working, cross-compile-able Docker image sorted out.

Build... X?

Knowing I wanted a small image, and knowing that images built FROM scratch tend to behave pretty much anywhere, I wanted to use a multi-stage build right out of the gate. I started with something like the following:

FROM golang:1.16-alpine AS builder

COPY . . 
RUN go mod download

RUN CGO_ENABLED=0 GOARCH=armv7 go build -a -o /victim

# ---------

FROM scratch

COPY --from=builder /victim /victim

CMD [ "/victim" ]

While iterating on the above did produce results, and the resulting Docker image was around 5MB in size, I wasn't super happy with having the architecture hard coded in there. This meant I'd need to run several builds to account for different architectures, and then do a bunch of wrangling to get all those images into the same manifest.

Turns out Docker themselves weren't happy with that state of affairs either, so they created buildx, which skips all the tedious manual steps and creates a single multiarch image for you. And they even pass build arguments into your image, which I knew I'd be able to wrangle into the golang environment variables I needed:

A list of build arguments like BUILDPLATFORM and TARGETPLATFORM is available automatically inside your Dockerfile and can be leveraged by the processes running as part of your build: RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" > /log

Docker - Working with Buildx

Luckily, there's no need to go through the tedious process of manipulating these into the GO* compiler flags - some kind soul has done the work for us in this project. So our final Dockerfile for dynamic cross-compilation with buildx becomes:

FROM --platform=$BUILDPLATFORM  golang:1.16-alpine AS builder

# Parses TARGETPLATFORM and converts it to GOOS, GOARCH, and GOARM.
COPY --from=tonistiigi/xx:golang / /
ARG TARGETPLATFORM

COPY . . 
RUN go mod download

# Build using GOOS, GOARCH, and GOARM
RUN CGO_ENABLED=0 go build -a -o /victim

# ---------

FROM scratch

COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group
COPY --from=builder /victim /victim

EXPOSE 8080
USER guest
CMD [ "/victim" ]

And can simply be built with docker buildx.

But that's still not good enough! I want zero commands to remember, and as much automation as possible. Time to configure Github Actions to do all this for me.

I've very little experience with Github Actions still - we use Gitlab at work, and its CI suite works quite differently - but was quickly able to find a buildx action that looks like the following:

-
    name: Set up QEMU
    uses: docker/setup-qemu-action@v1
-
    name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v1
-
    name: Build and push
    uses: docker/build-push-action@v2
    with:
       context: .
       platforms: linux/arm/v7,linux/arm64/v8,linux/amd64
       push: true
       tags: kieranajp/victim:latest

This will, in one fell swoop, cross-compile our multiarch docker image, set up a manifest, tag everything accordingly, and push our resulting image up to Docker Hub. Sweet!

A sprinkling of Helm chart later, and we have our running application on the Raspi cluster!

$ kubectl get pod
NAME                      READY   STATUS    RESTARTS   AGE
victim-55cdf87c48-9qzl9   1/1     Running   0          10m

If you want to see the whole application with its multi-arch build and deploy setup, it's open source at kieranajp/victim. The app itself is just a simple Slack chatbot that solves a problem my partner Edele was facing at her work - but I learnt a lot while trying to get it deployed, and I'll be using this method to deploy a bunch of other stuff soon!

Published on ☜ Back Home