Developing for Kubernetes with Skaffold

There are a growing number of solutions that enable software developers to develop their cloud-native applications on Kubernetes. We've discussed a couple of these solutions in this series already (Tilt and Garden), and in this article we'll take a look at another, Skaffold. Unlike the tools we've already discussed, which have emanated from small, innovative cloud-native start-ups, Skaffold comes from the house of Google. When it comes to cloud-native tooling, Google seems to have a play in just about every category.

The fact that Skaffold originated from Google probably accounts for its relative popularity on GitHub. As I write, it has over 11 000 stars and in excess of 270 contributors, which outstrips the other tools that provide a similar developer experience. This gives the project, which is open source under an Apache Licence, a distinct advantage over its competitive rivals. More contributors bring more capacity to the cause, and the stars help to increase the project's profile. So, is this higher profile warranted? Let's lift the lid on Skaffold.

Pipeline Architecture

As a tool with the purpose to aid software developers, it comes as no surprise that Skaffold is designed around a pipeline construct. The pipeline is segmented into stages, some or all of which can be incorporated into a workflow to suit the circumstances. Some of these stages seem very logical (e.g. build, test, deploy), but some feel as if they're an affectation that simply serves to highlight a feature. Let's take an example; the tag stage is really a detail of the build stage rather than a workflow stage in itself. This is borne out by the way Skaffold defines a workflow using YAML-based declarative configuration. The important top-level keys in the schema are build, test and deploy, with image tagging details, for example, defined as a sub-key of the build key.

The reason some of these features are given prominence in the documentation concerning stages, is that Skaffold provides the developer with a lot of flexibility in implementing a pipeline, which it achieves with a pluggable system. Images, for example, can be built using a number of different tools, and tagged using a variety of different 'taggers' or policies. To expand on this a little, Skaffold can invoke a build using Docker, Jib, Bazel or Buildpacks, and can tag images using git references, image IDs, environment variables, and so on. This flexibility speaks to the abundant choice in tooling that's available in the cloud-native domain, and the many different ways that workflows can be configured. Kudos to the project for providing the flexibility and choice.

Workflows

Skaffold, like Tilt and Garden, uses declarative configuration to define the developer's workflow. As is generally the norm with cloud-native configuration, it's stored in YAML format in a file called skaffold.yaml, which is generally located in the project's root directory. The declarative configuration is consumed by the Skaffold CLI, in order to influence the actions it takes in processing the defined pipeline.

Building Images

---
apiVersion: skaffold/v2beta16
kind: Config
metadata:
  name: my-app
build:
  artifacts:
    - image: my-app
      docker:
        dockerfile: Dockerfile.build
  local:
    useBuildkit: true
    push: false

In this simple snippet of a skaffold.yaml file, the build key declares that an artifact, a Docker image, is to be built from a Dockerfile called .Dockerfile.build, which will subsequently be tagged my-app. This assumes a build context exists in the working directory, of course. Running a skaffold build command triggers an image build, which results in an image with two tags; one with the most recent git tag or git commit (e.g. my-app:ae7926d), and as the image isn't pushed to a container registry, a tag which uses the 64 character ID of the image. Skaffold is big on image immutability, and the latter tag is a content addressable reference to the image in the local cache. If it had been pushed to a registry, the immutable reference would be the image's digest instead. This ensures we only use the image Skaffold has built, and not a nefarious image masquerading as one and the same.

As well as providing flexibility on the method of build that we mentioned earlier (i.e. Docker, Jib and so on), Skaffold can accommodate different locations for performing image builds. They can be performed locally, inside a Kubernetes cluster (using Kaniko), or remotely using (unsurprisingly) Google Cloud Build.

Deploying to Kubernetes

Just as there is a skaffold build command to build images, there is a skaffold deploy command to deploy applications that are defined in the build section of the configuration. Once again, Skaffold provides some flexibility to accommodate the three most popular methods for deploying applications to Kubernetes clusters. It supports good old kubectl apply, Helm charts, and Kustomize variants and overlays. Skaffold also supports kpt as a deployer, which you can expect to play a bigger role in the future. The skaffold.yaml file schema has a number of different keys for defining deployment activity for whichever deployer is used, all under the primary deploy key.

Testing Applications

As far as testing goes, Skaffold has an in-built test stage which sits between the build and deploy stages. At present, there are two types of tests that can be conducted; container structure tests, and custom tests, both defined in the 'skaffold.yaml' file. Whenever a build is initiated after a change, any defined tests for specific images are executed before deployment. That is, unless Skaffold is instructed to ignore the defined tests. If tests fail, the pipeline is aborted and no deployment takes place.

So, what type of testing can be achieved? Well, container structure tests are an opinionated approach to testing the content of a container image. They are designed to check for expected filesystem content, image metadata (default user, for example), environment variables, correct command output, and so on. Unfortunately, the GitHub repo that hosts this project suggests that it isn't an officially supported Google project, and that it's currently in maintenance mode. It seems the reason for this is that there isn't sufficient staff to maintain the project. Best to use with caution.

A custom test executes any command the developer wants, and allows for defining dependencies for the test. The test is executed locally, and might be used to run unit tests, scan for image vulnerabilities, or whatever is deemed important prior to deployment.

Developer Experience

So far we've discussed the different stages and tools that can be plugged into the pipeline, but we haven't really discussed the overall developer workflow. A skaffold build, skaffold test, or skaffold deploy command executes actions for the appropriate stage in question. To get the seamless inner loop experience, a developer needs to use skaffold dev, which puts Skaffold into a development loop. It watches for source file changes, syncs files to running containers or builds images, runs tests, deploys the services to the target Kubernetes cluster, and then returns to watching for changes. This is considered the main mode of operation for Skaffold.

But the development loop can also be augmented with debugging. The skaffold debug command behaves much like the skaffold dev command, but configures the application's container to allow debugging for the language used for the application service. How this works depends heavily on the language in question, but Skaffold supports .NET, Python, Java/JVM, Golang and Node.js at the time of writing.

Conclusion

Skaffold has all the essential ingredients for successful inner loop development in a Kubernetes cluster, and even offers an API in dev or debug mode for integration with IDEs like Visual Studio Code and Intellij. This further improves the overall developer experience. But, it can also be used in CI/CD pipelines, which is where Skaffold deviates away from what's offered by other competitive tools that tend to have more focus on the inner loop.

By its own admission, Skaffold is an opinionated development environment for developing applications for Kubernetes. Whilst it's open source and provides a degree of flexibility in tool choice, there is an undeniable leaning towards all things Google (Jib, Bazel, Kaniko, Kpt). This might deter some, but for others, having a project supported by a large public cloud provider might be an attractive proposition. Either way, Skaffold sits comfortably alongside the other competing tools in this area of cloud-native developer tooling.

We're always interested to see how cloud-native tools are adopted in the real world. So, let us know if you're a Skaffold fanboi, and share your experiences using it to develop your software applications.