Chapter 6

Speed and stability: Redefining IT with DevOps

BMK Lakshminarayan

Marc Andreessen coined the famous phrase ‘Software is eating the world’ in his 2011 Wall Street Journal op-ed. To this effect, technology has become a significant differentiator for businesses to stand out and outperform. In some cases, only those with the right kind of technological advantages survive. For example, the recent pandemic taught our industry that those with digital capabilities managed to endure and continue to do business. Crises like this remind us — public agencies, service organisations, financial institutions, retail — about the need for the digital revolution and associated transformation. It is not just a matter of staying ahead of the curve with SMACT (Social, Mobile, Analytical, Cloud and Internet of Things).

Patrick Debois, back in 2009 announced his conference as ‘DevOpsDays’, and laid the foundation for the need for ‘dev’ and ‘ops’ to work closely together, to drive the speed for innovation and delivery. Dev means the developers/engineers, including people who build software applications, services and components. Ops means the operations engineers, including system administrators and database administrators, who deploy the software created by developers to production environments, manage and look after it. These build and deploy functions are traditionally in siloed units, even today.

Hence, the dev team creates the ‘deployment instructions’ for deploying the software. It describes what order/sequence it should be in, what configuration entries to update for production, and handover instructions to the operations team. The operations team are the only people who have access to production environments, follow the deployment instructions and deploy the software. This results in disconnected functions. Developers have detailed knowledge of the functionality of their software and how the different components work together, however, they don’t appreciate how that software will run day-to-day in the operating environment. The operations team do have that in-depth knowledge of infrastructure and their focus is on service management rather than a detailed understanding of the different software components.

Donovan Brown from Microsoft defines DevOps as a ‘Union of people, process, and products to enable continuous delivery of value to our end users.’ Why we do DevOps comes down to that one big word Donovan highlights — value.

This chapter is about how the last decade of DevOps and Reliability Engineering practices have emerged and changed how we plan, manage, deliver and operate software at scale.

The two critical capabilities are first, the speed of innovation (that is, time to value, not just time to market). Second, the IT system and services’ stability to provide excellent customer service.

Time to market is a measure of how quickly the software product or its services are launched or made available to customers. In contrast, the time to value measures how long it takes existing or new customers to realise value from the software product.

In New Zealand, most enterprises, government organisations and banks have been catching up with DevOps since 2013. This chapter will paint the need for DevOps, provide an update on the availability of tools, training and skills, outline the current skills gap, and discuss the current state and the future of Kiwi DevOps.

Agile and DevOps — the culture

Twenty years since the first publication of the Agile Manifesto, many organi­sations of all sizes are still on their Agile transformation journey. Agile methodology sounds like a common-sense approach to delivering business value in small incremental batches and provides the organisations with much needed ‘predictability’.

Over the years, our industry has seen many Agile frameworks built to suit small to large enterprises, such as Scrum, Kanban, SAFe, LeSS, Nexus. On top of this, various providers have multiple certifications to help practitioners and coaches.

In 2008, DevOps emerged out of a discussion between Andrew Clay and Patrick Debois. The idea started slowly gaining momentum after Patrick (considered Godfather of DevOps), held the first DevOpsDays at Ghent, Belgium, in 2009. DevOpsDays events started spanning out to multiple countries, and like-minded technology professionals, both Dev and Ops, were attracted to this concept of DevOps. Jez Humble and Dave Farley wrote Continuous Delivery (2010), a book that describes reliable software releases through build, test and deployment automation. In general, people consider that DevOps addressed the gap that Agile left in this space — the ‘speed’.

Speed can be achieved by automating:

• Continuous Integration (CI) — automating the software artefact build process
• Continuous Testing — automated testing via scripting test cases that executed part of CI & CD
• Continuous Delivery (CD) — automated deployments of software artefacts to all environments from development to production

CALMS* (Culture, Automation, Lean, Measure and Share) is used as a framework to assess the way to define how organisations adapt to DevOps. John Willis, Damon Edwards and Jez Humble introduced CALMS:

• Culture refers to the organisational culture — changing the way people work. Dr Ron Westrum, the famous American sociologist, published a typology of organisational culture, referring to ‘generative organisations’ as those actively seeking and sharing information to better achieve their mission.
• Automation is an effort to automate repetitive manual tasks to reduce effort, maintain consistency and increase speed. It includes how infrastructure is provisioned and how software code is reviewed, tested, released and deployed.
• Lean refers to simplifying processes and eliminating any waste. The idea comes from the manufacturing industry and includes wait time, over-processing and minimising unused/unwanted artefacts.
• Measure is about metrics that need to be collected to analyse and improve the various processes.
• Share refers to spreading the good work of the DevOps team to the rest of the organisation and sharing the learnings and experiences, including common pitfalls, asking for help and offering help.

Gene Kim introduced the three ways approach, to help get a different perspective of DevOps:

• First way: Flow and System Thinking — emphasise performance of the whole system instead of just one portion, unit or silo.
• Second way: Amplify feedback loops — focus on shortening and amplifying the feedback loops.
• Third way: Culture of continual experimentation and learning — create the right organisational mindset to experiment, risk, fail fast and learn.

A few organisations in the forefront demonstrate how they can apply Agile at scale and how quickly the software can get shipped to customers. Interestingly these enterprises are usually technology companies. Most of us have read the stories about Spotify and other highly successful companies that are leaders in their field, such as FAANG (Facebook, Amazon, Apple, Netflix and Google). These enterprises deliver software at scale and continuously deliver software features, bug fixes and security updates multiple times a day.

One common thing that stands out in any transformation journey is the organisational culture. Most practitioners and coaches agree that culture plays a significant role in the transformation, creating an impact in the organisation and increasing enterprise agility.

DevOps momentum in New Zealand

In 2013, the legendary John Willis, co-author of The DevOps Handbook (Kim et al 2016), introduced DevOps to New Zealand and held the first DevOpsDays in Auckland.

Rob England, our New Zealand-based Information Technology Service Management expert, known as ITSkeptic and well respected globally, has said that he attended the first DevOpsDays in Auckland. He was curious to learn more about this concept. Rob’s mission led him to the first DevOps Enterprise Summit in 2014 in San Francisco. Gene Kim, co-author of the famous The Phoenix Project and The DevOps Handbook, chaired this conference. This conference attracted people who were trying to understand how to introduce DevOps in enterprises.

In 2016, a few like-minded DevOps advocates came together to run the DevOpsDays conference in Wellington. The growing adoption of DevOps in New Zealand became very evident from the number of conference attendees.

In March 2016, Chris Mazur hosted the very first session of the Wellington DevOps Group meetup. The meetup group has organised many sessions in the last five years, bringing together DevOps practitioners and thought leaders. As of June 2021, the Wellington meetup group has over 1800 members. The Auckland meetup organised by Alexander Taler has more than 2000 members.

DevOpsDays New Zealand conference and DevOps meetups are well attended by people from many different industries, including government agencies, banks, health, retail, hospitality, travel.

DevOps, SRE and tools

In the last decade, the software industry has witnessed a plethora of tools in the DevOps space. Tools are now available to automate continuous integration; automate software code quality checking; ensure continuous delivery by release automation and automated deployments; test automation tools and infrastructure automation. There are also tools for configuration management, secrets manage­ment, monitoring and logs aggregation.

Some big players in the market have acquired small start-ups with a niche in DevOps tools, which lets these enterprise software vendors expand their product portfolio to address gaps in the tools ecosystem. These tools are used by enterprises to ideate, create, release and operate software, The ecosystem includes tools owned by the vendor, for example they may have a tool for building the software but not for deploying it.

Ten or even just five years before, provisioning infrastructure was considered a time-consuming and expensive exercise. ‘Virtualisation’ led to a change in the way infrastructure and platforms were approached. Platforms and tools like Docker Containers, and orchestration tools like Kubernetes have made a significant difference in the way hardware resources are utilised. They provide process isolation along with the ability to build the resiliency and backing needed to create a fault-tolerant platform to provide stability for the software and services.

To operate software at a scale that can be used by millions of users worldwide, Google introduced Site Reliability Engineering (SRE) practices in 2003. DevOps and SRE go together well as DevOps provides speed, and SRE provides the principles and practices of reliability engineering and operating software at scale.

SRE focuses on operational task automation to reduce the toil, take measure­ments, introduce service-level objectives and meet service-level agreements. Just delivering software to production at speed in itself does not help enterprises. It is also necessary to continuously monitor software services and applications by collecting and analysing logs and triggering automated email and incident alerts.

DevOps adoption breaks organisational silos and aligns organisations with customer value by continuously delivering software to production. The fact is that some organisations were successful with DevOps adoption, however, some still struggled to overcome and break functional silos to gain enterprise agility. The problem seems to have more to do with the culture and the organisation’s ability to continuously improve software delivery. The successful organisations were ready to introduce a new way of working and a holistic approach to the end-to-end software delivery. At the same time, other organisations struggled to identify suitable opportunities. They focused on local optimisation, improving just one functional silo, failing to realise the importance of upskilling people and underinvestment in tools and technology.

Measures

As DevOps gained momentum, the need for measuring and understanding DevOps metrics became critical. Metrics provide a reality check for organisations to see how they are doing when it comes to DevOps adoption.

In the report ‘Accelerate’,* Nicole Forsgren defined four critical metrics for DevOps capability. Dr Forsgren, a researcher by profession, did a number-crunching exercise for the ‘State of DevOps’ report to better understand the patterns, practices and organisational capabilities. She defined metrics for measuring speed and stability:

Speed metrics

• Lead time for changes (from code commit to code deploy)
• Deployment frequency

Stability metrics

• Mean time to restore (MTTR)
• Change fail rate

It is essential to capture all these four metrics as they represent both DevOps’, Dev and Ops side.

Regarding speed metrics, questions need to be asked, such as: How long does it take to get from the code commit to code runs in production? How often do we deploy? Successful organisations deploy multiple times a day. The trick is to deploy minor changes more frequently, thereby reducing the impact of any one production change.

On the other hand, stability metrics are about quickly fixing something in the production system if it is broken. Even a minor production outage can impact the customer experience. Consider these examples:

• Customers not checking out their shopping cart affecting their buying experience
• The system being unavailable during regular working hours

When we introduce a change in the production software system, the change failure rate measured is the likelihood of causing a service outage or incident.

These four metrics provide a great way to measure the organisational capability and maturity to understand DevOps adoption better. They reflect organisation capability in terms of practices and technology tools available within the organisation.

Skills and training

Though the number of available tools for DevOps has grown, the skills required to create, test and manage the infrastructure, platform and software services have become very scarce. The pace of change in the software industry is phenomenal. To stay up-to-date, practitioners must constantly upskill by learning new tools, practices and unlearning some of the old methods and techniques.

The number of roles advertised in the New Zealand job market show that DevOps skills such as infrastructure automation and test automation are much sought after here. Public cloud adoption by government and other enterprises, has accelerated the growth in demand for staff with DevOps skills.

DevOps provides a vast opportunity and filling these DevOps roles has become critical for organisational success. Modern software engineering, architecture, continuous integration/ continuous delivery (CI/CD),* build and test automation plus observability demands more from the IT professional these days. The ‘E-shaped’ or ‘comb-shaped’ engineers who have more than one specialisation are in more demand than ‘T-shaped’ engineers who are specialised in just one area.

T-shaped engineers are generalists; they have deep expertise in one area and have broad skills across many areas. These individuals can step up to remove bottlenecks and make planning flexible, and they absorb variability. For example, T-shaped engineers might have deep expertise in writing software code and may know how the infrastructure is set up. However, they do not have the skill or expertise to script and provision the infrastructure.

E-shaped engineers have deep knowledge in few areas and have working experience across many areas. They have proven execution skills and are constantly innovating. They have greater potential to achieve the best outcome. For example, E-shaped engineers might have extensive expertise in writing software code and as well as experience in writing infrastructure automation and test automation scripts.

Today’s engineers can find plenty of resources online; they can connect with global thought leaders, practitioners, authors and experts online. Learning opportunities can also be amplified by attending relevant conferences and local meetups.

Various certifications, up to excerpt level, are available from the DevOps Foundation. Most New Zealand organisations and employers support training and help their staff to upskill by providing certification and training programs. Training has now become an integral part of the work as the demand for highly skilled engineers grows.

DevOps adoption acceleration

Agile and DevOps transformation is an organisational change needed to survive and thrive in this digital age. Enterprises that can advance their technology and achieve digital transformation by driving these changes will grow their business and stay competitive.

Agile and DevOps help organisations and businesses respond to market needs and changes using proper practices and tools. Investing in tools alone does not enable an organisation to make the transformational changes required to fully integrate DevOps; organisations also need to invest in people, re-engineer their processes and modernise software architecture. DevOps Research and Assessment (DORA)* reported that IT performance contributes to overall organisational performance.

In New Zealand, many public sector agencies, banks, retail, supermarkets, travel and hospitality industry, health organisations and business have adopted DevOps either in a small pocket of implementation or right across the enterprise. Global software and tool vendors like Redhat, Sonatype, CloudBees have established their offices in this region to work closely with their customers and accelerate DevOps adoption in the recent past.

A few organisations are still struggling to take their first step on the DevOps journey. Reasons include the organisational culture, lack of talent and reluctance to upskill employees and invest in technology tools. Some large enterprises have many disconnected tools, overloaded processes, an over-engineered tech stack, and are just not set up to deliver the best customer outcomes. However, any organisation that wants to transform should invest in DevOps. Most savvy CIOs have DevOps on their plan as a means of improving the efficiency of their technology and doing more with less.

Summary

When it comes to speed and stability, customers often demand better services from their service providers. They want an innovative organisation that can constantly improve its products and services, deliver value at speed, and care about a reliable system and services.

Organisations that understand the pulse of their customers know what they need and how to deliver it successfully, gain and retain customers. Delivering value at speed and providing a reliable platform, application and services to customers makes organisations more trustworthy. Teaching and adopting a DevOps mindset, approach and technical practices show that they are committed to serving their customers better.

BMK Lakshminarayanan is an inspiring and passionate DevOps Advocate and Value Stream Architect promoting DevOps, Lean and Value Stream Management practices. He serves as New Zealand ambassador for the Cloud-Native Computing Foundation, DevOps Institute and Continuous Delivery Foundation, and a board advisor for the Value Stream Management consortium.

With over 25 years of ICT experience, he has worked on various challenging assignments as an architect, designer and developer ranging from desktop applications to distributed systems. He brings his wealth of knowledge and practical experience to introduce DevOps, Continuous Delivery practices challenging the status quo of the old way of thinking and traditional enterprise software development and delivery technical practices.

His primary focus these days is on developer productivity, building high performing IT delivery teams, and driving strategy and execution to deliver value to customers. He frequently writes and speaks on DevOps, continuous delivery, cloud-native, open-source systems and architecture at various global conferences and local meetups. BMK is one of the core organising committee members for DevOpsDays New Zealand and co-chairs the Cloud-Native Summit Wellington conference. He is passionate about engaging, connecting and learning with community members and leaders. His mantra for amplifying his learning and bringing the best to the community is: learn, share, care and grow together.

References

Humble, J & Farley, D (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley Professional

Kim, G, Willis, J, Debois, P & Humble, J (2016). The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press