At Giant Swarm, I work on Microservice Infrastructure and a lot of my work is centered around improving how we run clusters as well as software on clusters. However, this work isn’t reflected in my local development environment – as much effort as we put into improving the server experience, the local development experience still lags behind.
From Snowflake to Cloud
Whilst I have massive amounts of compute available to me in the cloud, this isn’t usable for my development environment. I’m limited to the amount in my laptop. Also, to be a little thrifty, we spend a fair amount on high-spec laptops, even though we have massive servers already running.
Furthermore, the drawbacks of snowflake servers are well known. I regularly preach about how snowflake servers are pure evil, and yet my local development environment is this weird rube goldberg machine of packages and aliases that just magically happens to be capable of producing code - it is a snowflake development environment. We can do better here.
The question then is how can we use lessons learnt from running clusters, to improve development environments? And what does that look like?
Containerizing the Environment
There are two rough parts to my approach. The first is moving my development environment into a container. This has a whole heap of benefits: my dotfiles are finally version controlled; I have far better control over which versions of packages I’m using; I can spin this environment up whenever, and completely destroy it – I don’t have to care about keeping my development environment clean and tidy!
One drawback here, although slightly self-inflicted, is that I’m only using CLI tools in the container – no X11 mounting magic. This isn’t too bad – I’m finally getting into using vim, so I don’t have to only pretend I’m a hacker anymore. This container is also open-source, which is pretty cool. I can share my environment with my colleagues (and you), and people can benefit from anything cool I come up with. Have a look at: https://github.com/JosephSalisbury/dev-workspace-container. It’s geared towards my own setup, so you will need to fork it (unless you want your username to be joseph). The container includes a lot of useful tools, including a golang environment. Docker support is provided by using a privileged container, and mounting the Docker socket.
Running it on Kubernetes
While running your development environment in a container is a good start, it doesn’t address the imbalance in computing power between my machine and my available clusters. We want to run the container on a cluster!
I have access to a bunch of really awesome Kubernetes clusters via Giant Swarm (We have Kubernetes clusters! They’re fantastic! You want one! I think that’s enough of a plug, but I needed to do it). Kubernetes is a good way to run containers on clusters, so let’s use that.
I have a little bash script that sets up a namespace, starts a pod with my dev container, and attaches to it. This lets me spin up my development environment on any Kubernetes cluster. This script also creates a secret with my credentials and keys in it. It lets me keep my sensitive data out of the container, which is pretty cool, and realistically, completely essential. It also means that auditing and rotating my credentials is fairly straightforward – they’re all listed in this script. The dev env container is set up to consume these secrets.
I’m planning on converting this script into an open-source binary that can use a service, like LastPass, to fetch my sensitive data, so it doesn’t include this sensitive data itself. This should mean that all I need on a machine is this binary (and Chrome and Slack), and I’m good to go! I like the idea of my next laptop being really low power, and pushing all my heavy lifting onto a cluster.
Conclusion
There are some pretty nifty benefits to running your development environment as a Kubernetes Pod. My current Giant Swarm projects involve writing Kubernetes controllers and operators; I can execute them straight away using in-cluster configuration. You can also use the Kubernetes service account to configure kubectl!
I have some thoughts on how to extend this idea even further. It should be possible to write a shell that runs every command as a separate Kubernetes Pod, and spread the load over the cluster – you could run a large compilation job on one node in your cluster, and use other nodes in the meantime. There are also some thoughts to consider regarding storage – having a volume containing all your repositories and data available on your cluster, and mounting it to your Pods as required, or using the gitRepo volume support. Lots of ideas!
I don’t know how to wrap this up, as it’s fairly rambling. tl;dr – put your development environment in a container, and then run it in a Kubernetes cluster. There’s some pretty cool things you can do with it. It’s tidier, more powerful, your laptop can be cleaner, and if you’re building stuff for Kubernetes, you can test natively right away!
