GitOps with GitLab Agent for Kubernetes

Following our articles on ArgoCD, Flux, and Fleet, this next article explores the features of the GitLab Agent for Kubernetes. The Agent enables a number of different DevOps features for teams, GitOps workflows amongst them.

Given that an outcome of GitOps adoption is the continuous deployment of software applications, it shouldn’t be a surprise to learn that GitLab provides a solution in this area. GitLab, after all, is as much a CI/CD platform, as it is a version control system (VCS). But, what makes GitLab’s approach a little different from the other tools we’ve discussed in this series so far, is that it’s targeted exclusively at GitLab users (SaaS or self-hosted). And, furthermore, the agent’s GitOps capabilities are only available to its ‘premium’ or ‘ultimate’ fee-paying customers. This will obviously be a barrier to some organizations, but if you’re invested in GitLab as your primary VCS tooling, then it could be a good contender for all your GitOps needs.

GitLab Agent for Kubernetes

The Agent for Kubernetes is the component in GitLab’s toolchain that allows for GitOps-style deployments. Implementing GitOps is a big part of its function, but its utility stretches way beyond the GitOps use case. In fact, its purpose is to provide a generalized means of secure communication between GitLab and one or more Kubernetes clusters, without the need for exposing the Kubernetes API. Exposing the API, and storing credentials in third-party systems, will be ‘beyond the pale’ for many security-conscious organizations. Early GitLab solutions for integrating with Kubernetes clusters relied on certificate-based access using the Kubernetes API, so the introduction of the agent is a welcome progression.

So, the Agent provides a channel of communication, and some GitOps features are conducted over that secure channel. In addition, the Agent allows for running the GitLab Runner in-cluster, providing secure cluster access for CI/CD jobs, scanning images for vulnerabilities in the cluster, and much more.

Architecture

From a GitOps perspective, the Agent works differently from those agents that we’ve previously discussed for other GitOps tools. The other tools are characterized by agents that follow the Kubernetes controller pattern, each introducing their own custom resource definitions. The GitLab Agent for Kubernetes, on the other hand, is much more straightforward in nature. It has a number of functions, which all rely on its primary function, which is to establish a secure channel of communication between a GitLab instance and a Kubernetes cluster. So, how does this secure communication channel get established?

The Agent is implemented in a client-server configuration; the cluster-side component is called agentk and the GitLab component is called KAS (also known as the GitLab Agent Server). Communication between these agent components is initiated by agentk, conducted using a secure tunnel, and is a bidirectional gRPC stream. As the channel is bidirectional in nature, either side can establish communication with its counterpart. In the context of GitOps, this enables either push or pull-based operation.

Being of a relatively light nature, GitLab recommends at least one instance of agentk per cluster, and preferably an agent per application environment (i.e. dev, staging, production).

Registration

In order to establish trust between the client and server components of the Agent, the agentk client must present a token to KAS. The token will need to have been generated in advance in your GitLab project and be associated with the client’s name. To achieve this, each client has a configuration file defined for it in a repository at the location .gitlab/agents/<name>/config.yaml. The token is provided to agentk at the time of installation, which it then subsequently presents when it dials out to KAS. If authenticated correctly by KAS, the Agent’s configuration file is downloaded to the cluster, which gives agentk its scope of operation. This file is constantly monitored for changes in the scope of operation.

Creation of the token can be achieved manually through the GitLab user interface, or programmatically using GitLab’s API. If performed using the user interface, a helpful command is provided for generating the custom manifests required for installing agentk to the cluster; a Docker container is executed with its output piped to kubectl apply.

 docker run --pull=always --rm \
  registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable \
  	generate --agent-token= \
           	--kas-address=wss://kas.gitlab.com \
           	--agent-version stable \
           	--namespace gitlab-kubernetes-agent \
| kubectl apply -f -

This isn’t the only mechanism of installation, of course, and it’s possible to use more advanced methods, which might involve kpt and kustomize, for example. It’s also worth pointing out that the default configuration provides agentk with liberal access to resources in the cluster. If you need to restrict instances of the Agent to particular namespaces (which is a more realistic scenario), then the manifests need amending to reflect the required security posture. This may involve the creation of additional namespaces, and roles and bindings that control the Agent's activity in those namespaces.

Assuming that all is well, and the channel has been established correctly, operations can be performed on the cluster, whilst using GitLab as the source of truth for your application configuration.

GitOps Functions

When compared with its competitive peers in the GitOps arena, the GitLab Agent for Kubernetes is fairly basic in terms of function. But, it does exactly what you expect it to do; that is, to scan for any Kubernetes manifests at path locations provided in the config file, and then reflect this desired state in the cluster as an actual state. Originally, the GitOps Engine was used to achieve this, this is now replaced by the cli-utils package which allows server-side apply to be used which is beneficial when resources are owned by multiple tools (e.g. GitOps and HPA).

gitops:
  manifest_projects:
  - id: '31983943'
    paths:
    - glob: 'manifests/**/*.{yaml,yml,json}'

There are no fancy value-added GitOps features, such as; automated application image updates, notification alerting, multi-step synchronization, and so on. To many, these additional capabilities may not be as important as the basic automated syncing of the declarative state to the cluster. After all, GitLab’s integration with Kubernetes via the Agent provides a host of CI/CD capabilities that provide significant value.

However, there is one thing that does diminish the effectiveness of the Agent in my view. In the Kubernetes world, there are a variety of ways to define an application’s configuration for deployment to a cluster; Helm charts and templates, Kustomize overlays, raw manifests, and a number of other more exotic methods. At present, the agent only supports the application of raw Kubernetes manifests (as YAML or JSON). If you’re using Kustomize or Helm to define your app’s configuration, you’re either not able to use the Agent for GitOps deployments, or you’ll need to hydrate the configuration before committing it to the watched repo(s). Again, some may not see this as prohibitive, especially as the additional steps can be incorporated into an existing GitLab CI/CD pipeline. Some, however, might prefer to use ArgoCD or Flux as their chosen GitOps tool, because of the better config management options that are catered for. It’s a deficiency that GitLab have recognized, and intends to address in due course.

Conclusion

It's probably fair to say that the GitLab Agent for Kubernetes is a late arrival to the GitOps party with many of its peer technologies further down the line to maturity. So, it has some catching up to do. But, unlike the other solutions, which are git host agnostic, GitLab is focused on its own platform. For GitLab adopters, it's good to have a fully integrated solution that extends to GitOps deployments too. But, if you're wedded to another VCS provider such as GitHub, or perhaps a cloud provider solution, you'll be precluded from using this GitOps solution. That may not worry too many people at this juncture, as there is a sizeable chasm in available features when compared with the likes of ArgoCD and Flux. Hopefully, the imbalance will get redressed over time.

We're always interested in other people's views, especially when they're contrary to our own. Let us know about your GitOps experiences with the GitLab Agent for Kubernetes, and how you think it stacks up against other solutions.