Getting Started With A Local Kubernetes Environment

Update: This post has seen some great response and I’ve been receiving lot’s of nice suggestions. I updated the post with two more options that can be found in the official Kubernetes repo. There’s some more options in the comments below. Thanks everyone for your feedback.

At Giant Swarm we’re currently working on the next generation of our Microservice Infrastructure and I’m going to tell you a secret: It’s gonna be based on Kubernetes and fully-managed (yes, even the nodes) just like our current infrastructure. Ok, so now that this is out, you might want to know when you can start trying it out. There’s two answers to this question:

1. When it’s ready ;)
2. Right now!

Now, I got you confused, so let’s paddle back a bit. Firstly, we’re working on the new platform with full steam and will release it step by step, first to a selected group of alpha users, and sure soon enough also open to more people out there. Secondly, as you will be getting a fully-managed Kubernetes that includes all functionality of any vanilla Kubernetes, you can start right now with a local development environment and get familiar with it.

Finding The Right Local Development Environment

Like many things in Kubernetes world there’s no single source of truth and rarely can you easily find a common best-practice. This is both nice (diversity!) but also can get quite confusing for getting started. There’s many local development environments out there and I’ll try myself at giving you a bit of an overview over a selected few and a bit of help with choosing one that works for you. Disclaimer: I have no aspiration to completeness here and limited it to several common cases, but if you think I’ve overlooked an important option, just shoot me a message and I might add it to the list.

CoreOS Kubernetes with Vagrant

Quite commonly used are the Vagrant boxes by our friends at CoreOS, they run on almost any kind of host system and have both a single node version as well as a multi node (or cluster) variant. The setup is based on CoreOS VMs and they take care of setting up everything including TLS encryption between the components as well as DNS and even (if you want) Calico networking. You can configure things like how many nodes with how much RAM you want and which version of CoreOS should be running.

Choices:

• Single Node Vagrant Setup
• Multi Node Vagrant

Kube Solo & Cluster for OS X

Another popular choice is based on xhyve and thus supports Mac OS X only. However, in turn it comes with refreshingly low requirements as to prerequisites (neither Virtual Box nor Vagrant, just a recent version of Mac OS X). They have a single node and a multi node variant as well. And like the Vagrant boxes above, these are based on CoreOS and fleet as an underlying base layer and come with a running DNS setup. Additionally they come with UIs for fleet and Kubernetes. There’s even a nice menu icon, which gives you access to all kinds of commands and settings, like updating Kubernetes and CoreOS.

Choices:

• Kube Solo
• Kube Cluster

Kubernetes in Docker with Hyperkube

Further, there’s the option to just run a local Kubernetes in Docker, which runs pretty much anywhere that Docker runs. There’s several images and guides out there, but maybe the most used and most updated, as it’s built and pushed automatically on every release of Kubernetes, is hyperkube. There’s official documentation for running it. However, that one is quite long and includes many steps to get it to a state comparable to the above mentioned solutions, e.g. you need to deploy DNS manually, which can be tricky if you’re new to Kubernetes.

Your best choice:

• Kubernetes in Docker

Recent Developments In The Kubernetes Repo

There’s further some more recent projects appearing in the official Kubernetes GitHub repo.

One of them is Minikube, which is a single command way of getting a single-node cluster up in a VM (currently VirtualBox and VMware Fusion), incl. DNS and the dashboard, and is aiming to become the default local-cluster setup.

The other one comes curtesy of our friends from Weaveworks and is called Kubernetes Anywhere. It uses Weave Net and Docker to set up a Kubernetes cluster anywhere. The only prerequisite are said two tools. It works locally the same as it would work on AWS and supports multi-node setups. Maybe partly due to its goal of getting a Kubernetes up anywhere, it’s a bit more manual then some of the above-mentioned local-only setups, but the Weave team is working hard on further documenting and automating the steps, so I’m optimistic.

Choices:

• Minikube
• Kubernetes Anywhere

Decisions, Decisions, Decisions

Deciding on one solution mainly depends on four points:

1. Your host system
2. What you’d like to test out
3. How much flexibility you want to have
4. How easy and fast it should be

First, if you have Mac OS X the xhyve based environments are the more native solution. Docker moving towards an xhyve based local environment in the current Docker for Mac beta shows that there’s a trend towards more native virtualization on Mac OS. If you don’t run on Mac OS X then your choice gets limited to Docker or VMs by default. If you want an environment for your whole organization/team, then going with one that runs on any system might not be a bad idea.

Second, if you want to test out things that pertain to high availability, you should naturally choose a cluster (multi-node) solution. Further, if you like to test out certain networking solutions or other “under the hood” extensions consider point 3.

Third, changing stuff under the hood is usually easier when there’s a well documented system like CoreOS and fleet running below it. VMs give you a bit more manual control as it’s easier to fiddle with them. Further, if you want to test out different networking solutions, like e.g. Calico (or soon Canal) the CoreOS setup comes with Calico on board, but can also be used to test others like Weave Net.

Fourth, xhyve and Docker based solutions are generally faster (in startup) than VM based solutions like Vagrant. Further, the Kube Solo/Cluster solution is generally as easy as it will get and definitely currently the nicest from a UX perspective.

I personally use Kube Solo/Cluster for most things enduser side, but go for a CoreOS Vagrant setup when I want more flexibility and work more deeply within the system. However, if you generally don’t want to install anything (I presume you have Docker running), you can get a first glimpse of Kubernetes quickly with the Docker solutions.

Now What?

The next question after having solved the local development environment is where to start. Again, documentation and tutorials are a plenty, but finding the right one for you and not getting crushed by the huge amount of new concepts you might want to learn is not easy.

Starting with a shameless plug, you could try out the little Hello World I detailed out in my last blog post just to test out that things are working.

Then, however, I would recommend not actually trying out more getting started guides and examples, but to start getting familiar with the concepts of Kubernetes first. This will give you a good basis to actually understand what it is, you will be doing and help with finding the “right way” of doing things.

For a starter I’d recommend reading up on Pods, Labels, and Replica Sets. Then, once you have those down and maybe tested out a more advanced example, you should get more serious and look at Deployments, Services, Secrets, Namespaces, and DNS. Once you have understood these concepts, you’ll already be pretty well off in terms of being able to conceptualize your own services and run and manage them on Kubernetes. Go off and get some practical experience with running an actual (side-)project on your local Kubernetes.