Heredia, Costa Rica, 2022-11-05

Series: From Volkswagen Beetle to Ferrari: Accelerating ASP.Net Microservices, Article 1

Welcome to my personal nightmare. In this first article of the series From Volkswagen Beetle to Ferrari: Accelerating ASP.Net Microservices, I will be describing the general conditions of a microservices project I am currently working on. This is the start of a scary story that has given me personal, first-hand experience on how bad habits like not following DRY can become an absolute terror.

The objective of this series is twofold:

1. Share with the world my acquired knowledge on ASP.Net performance and how to actually achieve it.
2. Share with the world how bad programming practices can turn your love for programming into a living nightmare.

Without further ado, here it is.

Introduction

I entered my team as a senior software developer. They were looking for a person with experience in the microservices arena, and so there I was. Long story short, I got the job.

The system, which is a medium-sized project (for a top Fortune 500 company anyway), had already had around 7 months worth of work towards being converted from monolithic to microservices. I was therefore late to many recommendations.

State of the Conversion

As I stated, I was late to the project conversion start time, so I could not give any inputs on many things, like how to divide the application, where to start, how to set up authentication and security, etc.

So here is what I got when I entered.

1. The microservices were created as exact and almost complete copies (if not 100% complete copies) of the monolithic application.
2. The microservices received some basic refactoring to make them live in K8s pods; just enough for them to function.
3. The microservices underwent some minimal code conversion so they would compile as .Net Core v3.1 applications. The monolithic is (because it is still the production version) .Net Framework v4.8.
4. The people doing the migration of code made no investigation on how ASP.Net Core functions, so they dragged along the service locator anti pattern from the monolithic application, including the Autofac IoC container, among other vices.
5. There was no conversion from synchronous to asynchronous programming.

At this point in time, I can even tell you more details that I have discovered recently, but I think this is enough to get you started. More details will be revealed as the story continues in the article series.

Go-Live Nightmare

After about 1.5 years into the making, the microservices project attempted a Go-Live. While it was stress tested before this, and while the results were not the best, it still happened. The result was a disaster. The 600 or so user base the application has quickly proved the pods were unable to handle the concurrent users, which even weren't the full 600. The system was brought down to its knees with merely 200 concurrent users. It was literally unusable.

So enter the nightmare. As the senior developer with the most experience in microservices (which in all honesty wasn't that much to start with), I was given the responsibility to fix the system as fast as humanly possible.

Follow the blog series if you wish to know what happened next. I will detail my troubleshooting attempts, my discoveries, K8s configurations I tried, and most importantly, how I progressed through clearing the bottlenecks.