Sparrow Compute Blog - Skaffold + Go for a delightful kubernetes-based development experience

Creating a satisfying development experience is essential for fostering a productive and motivated engineering team. A poor development environment can quickly lead to frustration and inefficiency. To address this, I’ve been exploring the use of Skaffold to streamline the development process, enabling rapid and consistent iteration across various services. In this example, I will demonstrate how Skaffold can be utilized to improve the workflow for Go (Golang) applications. The key objectives are:

Simplifying and accelerating the most common code changes.
Ensuring less frequent code changes are also easy to implement without compromising the efficiency of common tasks.
Maintaining compatibility with existing debugging tools and workflows.

All files mentioned in this post can be found in this repo

Outline

Our approach involves using Skaffold to build our Docker image. This image will include all the necessary tools for building and running the application, as well as any additional utilities required for tasks like debugging (e.g., standard CLI tools such as ps, less, etc.). Since this image is not intended for production deployment, security is not a primary concern so this does not need to be a locked down image.

When a code change is made, Skaffold will detect the change and restart the Go application without restarting the container, thus minimizing iteration overhead. In subsequent posts, I will delve into further optimizations to enhance our iteration speed even more.

Init

Before we begin, let’s talk about init processes. The popular, lightweight tini is great, but is missing some functionality that we’ll need. Primarily, as mentioned above, we want to restart our application without the container restarting. We have a few options: We could use something like Air or gowatch, but those tools will sit and watch for file changes. That’s redundant since skaffold is already watching for file changes. So what we really need is for skaffold to tell our application to restart. We could build that functionality in our app, but that requires a code change of our production grade app that will only be used for development, so again, isn’t the best idea (not to mention if you have dozens or hundreds of services, changing all of them will not be a fun experience).

So instead, let’s build something ourselves to handle our use case. I’ve built a very simple shell script called devinit.

The script works as following:

Takes a command line argument of an application to start (eg ./app or go run .)
Starts the application
When it receives a sig USR1 (eg via kill -USR1 <PID>) it restarts the application
When it receives a sig TERM (eg via kill -TERM <PID>) it stops the application and shuts down

Straightforward enough. Pull the repo and try it out locally. Let’s make a simple go application:

terminal

go mod init github.com/sparrowengine/blog-skaffold-go

output

go: creating new go.mod: module github.com/sparrowengine/blog-skaffold-go

Add a main.go

app/main.go

package main

import (
    "fmt"
    "time"
)

func main() {
    for {
        fmt.Println("Hello, World!")
        time.Sleep(1 * time.Second)
    }
}

Run with devinit

terminal

./devinit go run .

output

Hello, World!
Hello, World!
...

Now keep the application running and make a change to main.go (I changed it to say Goodbye, World) and run the following in a different terminal:

terminal

pgrep -f devinit | xargs kill -USR1

And the results:

output

...
Hello, World!
Hello, World!
Restarting app...
Goodbye, World!
Goodbye, World!
...

Now, when we build our Dockerfile, we’ll just need to start using the devinit process. Then we’ll tell skaffold to signal the script to restart when we’ve made changes.

Docker

Let’s create our base dockerfile. This will be changed in future blog posts, but should be a good start.

app/Dockerfile

ARG GO_VERSION=1.22
FROM golang:${GO_VERSION}
ARG GO_VERSION

# init process that enables us to tell go to rebuild/rerun when files have been synced
RUN curl https://raw.githubusercontent.com/sparrowengine/devinit/main/devinit -o /usr/local/bin/devinit && \
  chmod 755 /usr/local/bin/devinit

# You can add any other tools you need here (remember, this is only used for dev so image size/security/etc shouldn't be a concern)
# RUN apt update && apt install -y curl vim

# Set up where our current working dir, where our app code will be stored
ARG WORKDIR=/workspace
WORKDIR ${WORKDIR}

# Copy all files (Make sure you have a good .dockerignore to avoid unnecessary rebuilds!)
COPY * ${WORKDIR}/

# Use signaler to call go run. We could have also built the app and run the binary, but this is simpler
ENTRYPOINT ["sh", "-c", "devinit /usr/local/go/bin/go run ."]

The dockerfile is commented and should be fairly straightforward.

It’s important to make sure your .dockerignore is very restrictive. You don’t want your app to restart or rebuilds to happen when you change README files or files that have no impact on your go app (alternatively, you can be more granular in your Dockerfile COPY - But if you find you’re modifying the Dockerfile frequently, you might be doing something wrong).

Here’s mine:

app/.dockerignore

go.work
go.work.sum
README.md
LICENSE
Skaffold.yaml
Dockerfile
.dockerignore
.git

Kubernetes

We’ll start simple here. Create a directory called k8s and add an app.yaml file. We’ll be deploying our app as a pod rather than a deployment or something that makes more sense, but this is just for this tutorial. This should match your actual production application as closely as possible (And would ideally be using the same exact files to keep things DRY).

app/k8s/app.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: app
  name: app
  namespace: default
spec:
  containers:
  - image: blog-skaffold-go/app:latest
    imagePullPolicy: IfNotPresent
    name: app

Skaffold

And finally, the skaffold.yaml file. We’ll only use the build and deploy steps to start

app/skaffold.yaml

apiVersion: skaffold/v4beta10
kind: Config
metadata:
  name: blog-skaffold-go
build:
  artifacts:
    - image: blog-skaffold-go/app
      context: .
      docker:
        dockerfile: ./Dockerfile
      sync:
        infer:
        - '**/*'
        hooks:
          after:
            - container:
                command: ["sh", "-c", "pgrep devinit | xargs kill -USR1"]
deploy:
  kubectl: {}

Build

The build step is the most important. We give it the Dockerfile and the context (cwd), which is enough just to build the image. In addition, we tell skaffold to infer which files are needed for syncing (hence the importance of .dockerignore). And finally, we tell skaffold that after a file sync happens, to signal our devinit process and tell it to restart the application.

Now spin it up and making file changes to your .go files will sync the files and restart the application.

Of course, there are still problems to be solved. The next step is to get debugging to work, but that will be part 2.

Deploy

We’ll use the simple kubectl deploy method. This is great for small applications, but once you outgrow this, take a look at the helm deploy method and other options.

Try it out

Now that we have all the pieces together, let’s try it out.

terminal

skaffold dev

output

Generating tags...
 - blog-skaffold-go/app -> blog-skaffold-go/app:1958d06-dirty
Checking cache...
 - blog-skaffold-go/app: Found Locally
Tags used in deployment:
 - blog-skaffold-go/app -> blog-skaffold-go/app:2db85477c26067b58cf6b84289fb14f113b354a8cf7f8ac371154ae5c4480543
Starting deploy...
 - pod/app created
Waiting for deployments to stabilize...
 - pods is ready.
Deployments stabilized in 3.075 seconds
Listing files to watch...
 - blog-skaffold-go/app
Press Ctrl+C to exit
Watching for changes...
[app] Hello, World!
[app] Hello, World!

Now change the main.go file and you should see:

output

[app] Hello, World!
Syncing 1 files for blog-skaffold-go/app:2db85477c26067b58cf6b84289fb14f113b354a8cf7f8ac371154ae5c4480543
Starting post-sync hooks for artifact "blog-skaffold-go/app"...
Completed post-sync hooks for artifact "blog-skaffold-go/app"
Watching for changes...
[app] Goodbye, World!

Next steps

This is a relatively simple set up, but can easily be used to scale to multiple services. But, we’re not done yet. This handles the happy path well, in that changing a single file will refresh the cluster quickly. But we still need to handle the scenarios where we need to modify dependencies efficiently. In addition, we want to make sure that our debugger can connect to the service so we step through our live code with breakpoints and all the other tools we’re used to. I will save that for the next posts!