Application Configuration Management with Kapitan

If you've been following along with this series on application configuration management in Kubernetes, then you'll already have an understanding of some of the challenges that this presents. We've already covered how Helm and Kustomize seek to address these challenges. In this article, we'll be taking a look at another community solution, Kapitan.

What is Kapitan?

Open-sourced in 2017, Kapitan originated from the company DeepMind Health (before its acquisition by Google) and has subsequently gained a sizable following. GitHub stars may not be the most reliable measure of the popularity of an open-source software project, but Kapitan's 1,150+ stars suggest the project has some tangible credence. It may not have the same number of stars or contributors that Helm and Kustomize have, but then it doesn't have the close association that these projects have with the Kubernetes project.

Kapitan was originally devised as a technology-agnostic tool for managing configuration but took on a Kubernetes focus as an antidote to Helm's perceived shortcomings. Whilst the main focus is on rendering YAML configuration for Kubernetes resources, it can also render templated scripts, documents (i.e. READMEs), or any other artefact that can be expressed as a template. This flexibility allows it to be applied in numerous different domains simultaneously.

How Kapitan Works

Initially, Kapitan is difficult to get your head around, so let's use a diagram to help explain how it works.

Kapitan is a Python-based tool that melds inventory configuration items with a set of templates and code to render artefacts as desired outputs. The inventory is simply a set of values expressed as YAML, that can be hierarchically split into classes, with the ability to reference Jinja templates and/or Jsonnet code. In addition to classes, inventory targets are abstract entities that provide an association for lots of different configuration elements. For Kubernetes environments, it's suggested that a target is synonymous with a Kubernetes namespace. Classes define configuration values, and targets stitch the values together.

One of the motivations behind the creation of Kapitan was the desire to keep all configuration in one central location. Not just for a single project, but all projects. This is in contrast to Helm, for example, where all values tend to be chart specific, and not easily shareable between different applications. Hence, Kapitan's inventory may be configured to comprise multiple applications, and many different target environments (e.g. dev, test, prod). The different targets dictate which components come into play during the compilation of rendered output by Kapitan.

The use of Jsonnet as a templating mechanism has also been a key factor in Kapitan's relevance to application configuration management in Kubernetes. Whilst Kapitan was originally developed to use Jinja templates (and still supports their use today), Jsonnet offers a far more expressive environment for templating. Jsonnet is a data templating language, which is effectively an extension of JSON, and provides programming constructs familiar to developers; library importing, functions, conditional branching, variables, and so on. The Jsonnet language has long been seen as a potential ingredient for resolving some of the issues in application configuration management in Kubernetes.

Let's lift the covers on some of Kapitan's key attributes.

Composability

One of the big strengths of Kapitan is its use of composable units of configuration definition. This greatly aids in the quest to be efficient by reducing the need to replicate configuration for different environments or scenarios. Different targets can be defined to use the configuration that is relevant for its purpose, and new application or environment targets can be configured swiftly without having to reinvent the wheel. The componentizing of configuration in this way facilitates composition and reuse. This snippet of YAML details a hypothetical hierarchy defined in a target:

classes:
  - common
  - component.env.qa-common
  - component.payment-common
  - component.payment-jsonnet
 
parameters:
  target_name: payment-qa

Each of the definitions under the classes key references a separate YAML file containing parameter definitions. Immediately before rendering its output, Kapitan assembles the inventory parameters into a single entity for processing (the ‘kapitan inventory’ command can provide this output without rendering the output). The parameters defined in classes can be overridden by subclass definitions or targets.

Clearly, a hierarchical structure of parameter values can work much better than an equivalent flat structure. Additionally, it can ensure that organizational policy and convention is adhered to when defining configuration items. This can be a great benefit for teams and organizations, but it does require a degree of initial planning and ongoing maintenance.

Templating

It's great that Kapitan allows you to compose and organize configuration, but how does the templating work? Templating is a key feature of Kapitan and a generally accepted requirement of application configuration management solutions.

parameters:
  namespace: payment-qa
  kapitan:
    compile:
      - output_path: manifests
     input_type: jsonnet
     output_type: yaml
     input_paths:
         - components/namespace/namespace.jsonnet

In this inventory class component, a reference is made to some Jsonnet code in the input path shown. The syntax informs Kapitan to expect input of type Jsonnet, and that it needs to generate YAML as output, which will be placed in a directory called `manifests` (which resides under a directory named after the target).

local kube = import "lib/kube.libjsonnet";
local kap = import "lib/kapitan.libjsonnet";
local inventory = kap.inventory();
local p = inventory.parameters;
 
{
    "namespace": kube.Namespace(p.namespace),
}

The corresponding Jsonnet code snippet imports some code from two libraries; one relevant to Kapitan, the other relevant to templating Kubernetes resources. The inventory parameters are assigned to a variable before the namespace parameter ('payment-qa') is passed to the ‘kube.Namespace’ function, which renders the YAML for the resource. The key, ‘namespace’, is the basename of the file that is created by Kapitan for the YAML definition. Assuming the target is called ‘payment_service’, the rendered Namespace resource is located at the path:

./compiled/payment_service/manifests/namespace.yml

Input Types

Jsonnet is one of Kapitan’s input types, let’s have a look at some of the others.

Documents and Scripts

In addition to the creation of YAML or JSON output using Jsonnet code, Kapitan can also render artefacts using Jinja templates. Consider that a project is also likely to consist of scripts, documents, Dockerfiles, and a whole lot more besides. Having the ability to produce multiple, nuanced outputs based on a single source using templates is extremely compelling.

Helm Charts

Kapitan also added support for the use of Helm charts as an input in v0.24 released in the second half of 2019. The input type needs to be specified as ‘helm’ and can be accompanied by a ‘helm_values’ YAML object, and a ‘helm_params’ YAML object. The former allows for overriding the Helm chart's default values, and the latter allows for setting the namespace, release name, and so on. Kapitan's rendering of Helm charts is equivalent to running the `helm template` command but without the Helm binary itself.

A bespoke Helm chart can be stored in one of the designated input paths, but what about off-the-shelf Helm charts provided by third parties? These are generally located in remote repositories or on the Helm Hub. How does Kapitan deal with this scenario? Well, Kapitan has a built-in dependency manager, which handles the retrieval of remote content. Provided an inventory parameter specifies the location of a remote chart dependency using the ‘--fetch’ flag with the ‘kapitan compile’ command will cause Kapitan to download the chart for its subsequent use in rendering output.

The addition of the Helm input, which is currently considered alpha in terms of maturity is important. Providers of cloud-native applications for Kubernetes generally use Helm as the preferred means of packaging their applications for distribution. In creating the ability to consume and customize these off-the-shelf applications, Kapitan provides access to the current standard for consuming packaged applications for Kubernetes. Without it, Kapitan would be a far less attractive proposition.

Conclusion

Kapitan is a genuine competitor in the application configuration management space and provides solid techniques for addressing the issues that surround the problem. It has other interesting features that are not covered in this article; manifest validation and management of secrets, for example. However, despite its functional strengths, there are some barriers to adoption.

Firstly, there's a relatively steep learning curve to conquer. Kapitan works off of a directory structure with snippets of configuration values distributed amongst lots of classes and targets. For the uninitiated, by the time you've added in the different inputs (Jsonnet code, Jinja templates, Helm charts) in their different directories, it's very hard to see how all the different components relate and fit together. However, if you can get past this perception problem, Kapitan's use of composition can pay great dividends. This is especially true if you desire to manage all of your configuration for different applications and environments from a single location.

Secondly, there's a requirement to learn how to program in Jsonnet. There is no gain without pain, they say. After all, if you're looking for an advanced experience for application configuration management then it's necessary to make the skills investment to achieve it. It's worth pointing out that Jsonnet is relatively straightforward as a language, and shouldn't challenge existing developers too much. It's also a feature of some of the other solutions that attempt to tackle this same problem.

Finally, it's worth pointing out that Kapitan doesn't enjoy the same level of adoption and contribution that other peer projects have. We've already mentioned that it isn't coupled with the Kubernetes project via a Special Interest Group (SIG) or another channel, it isn't a Cloud Native Computing Foundation (CNCF) project, and doesn't appear to have contribution support from any big cloud-native companies. This is in stark contrast to other solutions and ultimately may dictate the extent of its adoption. We'd love to hear about your experiences with Kapitan — do get in touch!

ICYMI: Part 1 — Kubernetes, Part 2 — Helm, and Part 3 — Kustomize.