Avoiding the TMT Stack
I work on a variety of applications that I design, build and maintain for my clients. At the same time, I’m constantly learning about new technologies, programming techniques, languages and infrastructure.
It’s very tempting to reach for the obvious technology and want to integrate it into your application, but the case that I will make in this article is that you should not do this unless absolutely necessary.
Lets start with an example. I have a modular monolithic application, that is a property booking system. It is a good few years old, but it works well and does what the client needs. The stack is pretty standard with PHP, MySQL, Nginx. There’s really nothing exciting here - this is the bare minimum that could be used to provide the application.
However, I also have to consider performance, and often-times this means implementing a caching layer for whatever heavyweight feature is slowing things down. Immediately I fire up a Redis container by adding it to my Docker configuration, install a couple of libraries and start configuring my application to cache whatever it is in the Redis instance. Great! It’s all working and it’s made the relevant performance improvements, tests are all green, we’re good. I’ve got a managed Redis instance ready in my live environment, so I can merge the changes and deploy to production. All goes well, everyone is happy.
A few months later, I’m asked to implement a search feature, and I know that Elasticsearch is great for this kind of thing, just like Redis is great for caching. Lets just say I implement things in a similar way to adding Redis for caching, by adding Elasticsearch to help me implement my search feature. Great, we’re all live, everything is working, everybody is happy.
Next up, we look at our message bus which we use for a limited amount of async processing. The currently implementation is using the database as a message store, which for our needs has been fine. However, this is not the recommended approach, and it should be better if we use a proper message broker such as RabbitMQ. Great, now repeat the above stories for Elasticsearch and Redis, but this time replacing our database message queue with RabbitMQ. Everybody is happy, the application is still working well.
So what’s the problem?
Suddenly, we realise that we’ve gone from a simple technology stack of PHP, MySQL, Nginx, to having a stack TWICE the size, with the addition of Elasticsearch, Redis and RabbitMQ.
We’ve ended up with what I like to call the TMT Stack - Too Much Technology.
The production environment is now twice as complex, with twice as many services, twice as much configuration, twice as many points of failure.
For small teams, this really does become a problem quickly. Is all this extra complexity really necessary?
In my view, 95% of the time, we’ve been distracted by all the promises of these additional technologies, and didn’t spend enough time evaluating whether we could perhaps still meet our feature requirements using our existing stack.
In this example, we should have spent some time considering things before adding more tech to our stack:
Do we actually need a cache? Can we optimise the code a bit instead? Does the cache need to be in Redis? Could we cache stuff in our database instead? Or on disk?
For the search features we’ve been asked for, could we not just spend some time crafting a decent SQL query that would provide similar, good enough results? Do we really need the full power of Elasticsearch to provide what we need?
Why are we even bothering with RabbitMQ when our previous strategy of using the database as a message queue was actually working fine for us? Why invest extra development effort when we didn’t even have a problem in the first place?
Sometimes, answering these questions will result in a decision to add to your tech stack, but I’d wager that the majority of the time, you’ll realise that you don’t need to invest the time and effort in setting up, configuring, and crucially maintaining these extra services into the future.
All this effort could be spent working within the existing tech stack on reducing technical debt or implementing new features.
Conclusion
Every time you get the opportunity to add a new feature, and the obvious way to implement it involves adding another layer to your technology stack, take some time to investigate whether this is really necessary, and how you might be able to implement the feature using the existing stack.
Simple technology stacks result in simpler applications, which are easier to maintain and add new features to. The KISS principle does not only apply to code, but to infrastructure too.
