Skip to main content

Paradigm shift

· 6 min read
Chafik Belhaoues

Introduction#

The Cloud(1) has been around for 20 years now and has changed radically our way of running business as it accelerates innovation and lowers a lot of risks for companies (like wasting money before finding a market). But most importantly it triggered a paradigm shift in the IT world as companies (and engineers) started thinking differently about how to access computing resources to run applications and businesses.

New paradigm shift#

Paradigm shift

When we analyze the history of IT we can clearly see how and why the first paradigm shift happened. Before the emergence the Cloud (before ~2006), companies had to buy or loan Data centers, buy servers and hire a lot of people to build and operate their infrastructures to provide computing resources. Which means investing a lot before seeing any value.

With the Cloud, companies started moving into something else, they don't only changed the way they operate their infrastructures but also the tools and solutions they use.

This explosion of tools was driven mainly by the open source that helped and still helping to bring maturity that allowed companies to be confident to run their workload in the Cloud.

Now and after more then 16 years of tools at all levels, the industry is shifting into something else because engineers feel more than ever the urgency to have an ecosystem to build and manage the Cloud. They no longer want to spend their timing chasing tools and gluing them together.

Why?#

Because:

  1. The Cloud is the default choice for almost all companies (90% of companies are in the cloud according to Statista).
  2. We have created a lot of complexity and the Cloud now is not only complex but also costly.
  3. The same challenges since the creation of the Cloud are still there: people, processes and tools.

People Process & Tools

People, processes and tools#

The main question that is constant since the inception of the Cloud is:

How do engineers work collaboratively, by respecting processes and using tools, to build and manage Cloud infrastructures?

The industry brought a lot of solutions/tools to the very same question/problem, but let's explore the 3 challenges together:

  1. People:

    To build and operate the Cloud companies need qualified engineers, skilled Cloud architects and trained DevOps people. At the same time, these engineers need to master between 30% to 40% of cloud services to be able to do their work correctly; and just to give an overview, we are talking about:

    • AWS has more than 280(2) services and ~ 500 Terraform resources
    • Azure has more than 600(3) services and Terraform resources
    • GCP has more than 200(4) services and Terraform resources

    Adding into that all the necessary tools that engineers need to learn like Terraform, Ansible, Jenkins, CI/CD... which shows clearly the high learning curve.

    Without talking about the absolute necessity to keep up with new technologies because the benefit is beyond personal aspects (one of the vectors to reduce costs is to use new services & product), and as we are talking about people, collaboration is key (again the same question: how engineers collaborate to create a value)?

  2. Processes:

    People within the organization generally follow processes & workflows to manage their Cloud infrastructure and generally these workflows are built manually which brings a lot of questions about security and how policies are created and respected (or not 😎 ).

    The main question is:

    Who has access to what and why

    These processes need to be maintained as far as the infrastructure and applications on top of it are alive.

  3. Tools:

    It is just crazy the number of tools we created to manage the Cloud / applications and this scary picture from CNCF tells a lot(5)

    CNCF landscape

    With all these tools we do not have an ecosystem for the management of the Cloud and engineers spend a huge amount of time to manually combine tools and processes / workflows to manage Cloud infrastructures.

Structured Cloud infrastructure#

At the actual stage of the industry we need more than ever to structure the way we operate the Cloud, and not only in terms of assets but also people and processes in mainly 2 verticals that empower each other mutually:

  1. Operational consistency to operate the Cloud in a consistent way we need:
    1. Centralized Cloud management: not only assets but also people, expertise and skills needed to manage infrastructures
    2. Unique source of truth: because with all the tools we have the source of the truth could be in different places, eg:
      • The source code (especially with Infrastructure-as-Code)
      • It could be in the head of the engineer (most of the time)
      • It could be in the console of the Cloud provider
      • Or simply in the bill especially the spicy one 🔥
    3. Security: we need to increase security and talk about security by design
  2. Productivity to increase productivity (and also operational consistency) it is mandatory to:
    1. Lower dependency between teams to avoid bottlenecks because usually Cloud architects or DevOps teams are responsible to provide and operate the Cloud infrastructure for other teams, which means the output and the speed of other teams depend on the output and the speed of the Cloud and DevOps teams.
    2. Increase collaboration within the same team and this is achieved by having the right solutions and tools that unify the culture of the company and teams.
    3. Documentation: this is indisputable as it has many benefits beyond the organizational memory. It helps people to be autonomous by making things clear in a way that anyone within the organization will be able to find any kind of information on his/her own.

Ecosystem vs tools#

Ecosystem

Shifting our thinking out of glued tools into ecosystem will help connect people, processes and business because we'll be:

  • Creating error-free environments
  • Increasing productivity
  • Making clear governance on who does what and why
  • Reducing time to delivery
  • Increasing the impact on business

Promise of change#

Driving any kind of change within an organization takes time / a lot of energy and in most cases it is a business strategy, so before investing any valuable resources (time, energy and money) we need to think about the output of the approach and not reinvent the wheel. Hopefully we are seeing emerging solutions that create ecosystems to build & manage Cloud infrastructures and are focused on business values.


References#
  1. Cloud computing history - wikipedia
  2. AWS services
  3. Azure services
  4. GCP services
  5. CNCF landscape