There has to be a different way
A struggle that engineers go through every single day!
When I joined my previous company as the director of engineering, my manager asked me two questions: are we efficient in our cloud infrastructure? Are we moving fast enough? My answer to him after 24 hours was: No & No. We agreed to be heads down to reorganize and introduce more automation to fix those problems. 6 months later we saved our company millions of dollars and our team was able to move 5 times faster than before. We were ready for our bonuses, but 3 weeks after, the whole engineering team regressed back to old habits, overprovisioning the infrastructure and cutting corners to meet their deadlines. Most of our progress ended up being lost. Three months later we were able to improve our situation again. But sure enough, after a little more time, we kept going back and forth between inefficiencies and improving them. I asked myself a question then: are we just a bunch of idiots repeating the same mistakes again and again? Or is there something fundamentally wrong about how the technology works? So, I reached out to many engineers and technical leaders in the community to learn about their journey. It was an eye-opener to realize that it is a common problem regardless of the company size or skill level, it is a struggle that engineers go through every single day!
There something fundamentally wrong about how the DevOps technology works?
This problem was frustrating, I was clueless about how to fix things. I thought at this point that there must be a different way for engineers to control cloud infrastructure and automate it. The current tools that require a steep learning curve, regular maintenance, and long hours to make them work are just dump technologies that steal engineers’ life and energy.
Something has to be done. I just didn’t see anyone doing it.
This problem will get worse if we don’t rethink the tools we use to write and manage software applications. The business will progress much slower than desired, engineers will have mounting pressures to keep up and will find themselves working more and more on non-differentiating tasks that take away time from making a difference for their teams, business, and customers. Something has to be done. I knew what was needed, I just didn’t see anyone doing it.
One day in a conference I asked the speaker at the end of a session if we’ll ever have a cloud infrastructure or technology that can be given high-level goals and be smart enough to achieve them and maintain them? The answer back then was: “It is coming, but no one knows when or how yet.” I asked the same question to senior leaders at AWS back then. I got the same answer. There was some incremental progress in the industry, however, the way I saw it, their approach was about making faster horses, not about rethinking the current technology. I was hoping that someone, somewhere is already taking the initiative to think differently, but nobody was. I became fed up waiting and decided to do it myself.
Finding the right balance between engineers' involvement and AI control is a key to next gen technology
So, I founded Magalix with the idea of building a new AI-powered technology that could help engineers focus on what matters most in their cloud infrastructure. We had many product iterations ranging from giving the AI full control of the infrastructure to not having it at all. We realized that there has to be the right balance between engineers' involvement and AI control to make them productive and confident about their work. We understand that these tools need to observe what engineers are doing in different scenarios and apply those learned practices in similar scenarios. We also learned how to keep developers in full control by adding the right mechanisms to engage and disengage those automation scenarios. After many iterations, we believe we have found the right balance.
The current tools and thought processes about infrastructure automation are not working. We need to find smarter, faster, and more fulfilling solutions to be productive, relevant, and empowered to do more. That’s why it’s so exciting that my team and I feel that we are closer to it than ever.
We have to start thinking differently.
Software engineers are applying machine learning in many different fields, such as voice and image recognition in consumer space. However, it is the least applied technology in software development and operations. I want to challenge you to really consider how machine learning can help build more impactful tools and systems to support us in our everyday engineering tasks/work. You will do 10x more relevant work and end your day much more fulfilled.
