My journey into devops

Effective monthly server and hosting costs down from 32800€ a month to 580€ a month. A cost reduction of 98,2%. When I finally crunched the numbers for my last rapport I felt ecstatic and sad at the same time.

I knew before I made all the changes to their hosted application infrastructure that I was going to be able to significantly help this company with the monthly recurring server and backup costs. But this reduction was only made possible because of something far more troublesome.

There were no business controls for IT infrastructure. No business controls on infra means there’s something seriously wrong with the business. Needlessly leaking upwards of 30k a month without someone sounding the alarm? This small scaleup was in trouble.

I wish I could say this was the first time I encountered something like this, but alas it’s not. This is, in fact, the third scaleup for which I cleaned up infra. Each and every time the problem wasn’t that noone knew what good infrastructure looked like. It was that skilled developers who were short on time had to meet deadlines, and they took shortcuts. Which in turn meant there were more servers than needed, backups were not in order, configurations didn’t meet redundancy specs, there was no documentation - you name it.

Combine this with a cloud provider which provides server backups without limiting either time or size, storage volumes being backed up but not in use, over-optimistic capacity planning and you have the makings of a financial equivalent of a black hole.

What you need to know about devops

DevOps seems to mean many different things to many different people and organizations, so I’m not going to attempt to describe some canonical definition for what someone in devops does.

What I can tell you is that — from my experience — people need someone on the ops team that’s experienced with deploying an infrastructure to the cloud which can run the desired or developed application stack.

This task seems to be appointed to either ops or development most of the time, but it’s really the sysadmin with development experience, or the developer with sysadmin experience which you should assign this task to.

Hence they look for either a sysadmin or developer - but what they really need is someone who loves the sysadmin/dev hybrid work.

That missing link is often the ‘devops’ role.

Devops mindset: a latticework of mental models spanning disciplines

To me it seems that devops is mostly an excercise in creative problem solving using any automation and configuration means necessary to provision robust and scalable infrastructures.

It’s ops to which software engineering principles are applied.

Being good at devops means you have to pull from an very broad spectrum of knowledge among which network- and system administration, security, virtualization, configuration management, CI/CD, cloud provider specific details and even software development.

This means that the required knowledge foundation for someone doing devops is very broad.

I find myself relying heavily on the latticework of mental models I’ve acquired during my entire career.

From the security principles and configuration management I did as a sysadmin to thinking in neatly set abstractions, components and scaling during my software development career - even tinkering with virtualization - they all helped me build a latticework for what is now called ‘devops’.

The road that lead me to devops

My first job ever was as a PHP developer in the wild west of the early internet. The company that hired me mostly used visual basic 6, and I was hired for creating a web-facing application in php. And (of course) mysql.

As they didn’t have anyone who could provision a server for me I had to do it myself. From setting up the servers and the networking to deploying and running the production server - it was on my plate. This is when I learnt shellscripts could both save me buckets of time and ensure some form of deployment uniformity.

The job following that wasn’t any different: devs had to manage and provision their own servers. The number of outdated, internet-facing servers was frightening. Unknown to the business, there was this huge risk hiding in plain sight. Out of sheer angst - coming from my dabbling in infosec - I talked to the devs who owned the outdated servers and convinced them to let me resegment the network and manage their servers. No more outdated public facing database servers please!

From that point on there was the sysadmin stuff for the rest of the business - the official sysadmin. And then there was me. The server- network- and virtualization management guy for software development people. And although I made mistakes (MongoDB - allow unauthed connections by default, really? I get the ‘solve it in the network’ thing, but at least provide secure defaults!) I feel I’ve contributed more to the primary business processes and the chain of value for the business as an devops worker than I ever did being a developer.

This pattern of ‘appoint that dev with sysadmin experience to the servers’ kept going as I proceeded throughout my career. And I didn’t mind. In fact, I enjoyed doing it. My official title was still something along the line of “software developer” or “application engineer” though.

From scripts to CI/CD

We’ve come a long way from semi-manually setting up new service accounts for our applications. Today we can provision entirely disposable, scalable, robust and fault-tolerant infrastructure from scratch given enough time and resources.

We have tools like terraform, ansible, kubernetes, docker and the list goes on and on. The tools - for every part of what is now called devops - have proliferated. Cloud providers like Amazon and Digital Ocean hand tools and offer automation at every step.

We can merge code into master and have it online a minute later. Life is good. Done well, devops can be of incredible value to a business, saving both cost and time on top of being able to guarantee virtually no downtime.

And it is exactly this transparency with regards to business value which made me switch from a developer to a devops role. Delivering value for the business and making the developers’ life easier. A win/win situation - and I get paid to apply all my technical experience to make that happen. I can’t think of anything that’s more satisfying.

Want to know more? Let’s talk!

The nature of devops work entails that, if I do my work well, I make myself all but superfluous. This contradicts my personality though, as I value making strong personal connections to the point that I think of any potential client through the lens of the ‘beer’ test; ergo “Would I want to grab a beer with the client on a sunny day?”.

So here’s the deal. I’m going freelance. I’d love to be part of different teams, implement a kickass infra stack - and if you’re interested we can also talk about a retainer for monthly maintenance (or even for severity-1 security issues/updates!). Sound interesting? Poke me on social media, or send me a mail at kapott@aivd.33mail.com and let’s build something awesome.