Docker and Related Services Enable a Future of Microservices for Everyone

Earlier in 2014 Benjamin Wootton wrote an insightful post on how microservices are not a free lunch. In that post he writes about the complexity of microservices in the real world and the many hurdles we have to take to make microservice architectures a reality. Especially, he writes about the “significant operations overhead” and “substantial DevOps skills required” to get microservices running, but also about the implicated problems that arise when trying to split up your monolith or even new app into many (small) microservices, e.g. implicit interfaces, increase in complexity, testing issues, and asynchronicity. Few months later Richard Clayton also wrote about their experiences with microservice and the common pitfalls in moving to this new pattern.

Dependencies from a developer’s perspective (PWC Technology Forecast 2014)

Basically we can summarize that there are two categories of problems arising with moving to microservice patterns. One is software architecture and paradigm related problems, e.g. how do we define a microservice, how big or thin should it be, how is communication handled, etc. The other is deployment and operations related problems, e.g. increased complexity, DevOps skills, etc.

The first category is a common problem, when moving from existing development patterns to new ones. It was the same when people were moving from procedural to object-oriented programming. This is even worse when a pattern is not well defined, yet. Microservices and microservice architecture are still in en early stage of definition (for a decent introduction to the topic see Martin Fowler and James Lewis’ article), but are gaining traction recently as can be seen from the Google Trends chart for the term “microservices”, which shows how more and more people are actually interested in this concept now, even if search volume is still extremely low.

As many will argue, it’s also maybe not a completely new concept. However, finally having a more or less fixed term for it, enables the community to communicate it better (e.g. show your CTO/CIO the great presentation by Adrian Cockcroft at DockerCon Europe 2014) and discuss what a true microservice architecture should look like. I'm seeing more and more discussion on topics like fat vs. thin services, inclusion vs. exclusion of DBs, "SOA = microservice vs. SOA as a predecessor", and many other parts of the greater microservice topic nowadays.

The operations related problems, however, are a different beast. There is increased complexity and a lot of other problems arising with microservices. And next to all those problems, there are also a lot of new tools and concepts that need to be learned and deployed. However, some stronger trends these days are also pushing the microservice trend. Namely the revival of container technologies, especially Docker getting more and more widely known and stable. And it's easy enough for people to actually use it (at least for dev and testing).

Docker and similar (or future) container technologies can help with at least some of the problems (especially when it comes to making boundaries between dev and ops more explicit and offering a means of communication for both groups). However, many of the problems still remain, especially when it comes to production and deployment. Handling your operations yourself is already quite some work without microservices, but with them, as stated in the articles mentioned above, the work increases manyfold. Suddenly you have to take care of various servers, environments, message queues, load balancers, clustering, etc..

We ran into the same problems and we tried a lot of approaches. Sure big startups and tech companies like Amazon, Twitter, Netflix, and others with more than 100 devs can pull this feat off. But even those have a lot of self-developed tooling and "hacks" running in between to make it bearable for dev and ops alike. Thus, we thought we should build something for the rest of us and that’s how Giant Swarm came to be. It's a simple microservice infrastructure, built for developers. At the moment it helps you deploy, run, and scale your dockerized apps. With time we are also adding more specific microservice "enablers", like a ready to use message queue and a circuit breaker for example. We also know that splitting up your apps into microservices is hard also on the conceptual side, so we try to write more content that gives you examples of common stacks and apps. We as everyone else in the ecosystem are not the authority in defining perfect microservices, so we are happy to let you try out our system and tell us what you need from a perfect microservice infrastructure, so we can build it for you.