DevOps can be defined as “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality”
DevOps (development and operations) is an enterprise software development phrase used to mean a type of agile relationship between development and IT operations. The goal of DevOps is to change and improve the relationship by advocating better communication and collaboration between these two business units.
Patrick Debois, who’s often called “the father of DevOps”, coined the word “DevOps” in 2009. As the word depicts, it was formed by combining two words: “development” and “operations”. DevOps is a collaborative way of developing and deploying software. DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operation professionals.
DevOps is the union of people, process, and products to enable continuous delivery of value to our end users. The contraction of “Dev” and “Ops” refers to replacing siloed Development and Operations to create multidisciplinary teams that now work together with shared and efficient practices and tools. Essential DevOps practices include agile planning, continuous integration, continuous delivery, and monitoring of applications.
History of DevOps
2007 – Patrick Debois, a software development consultant, had a goal of learning all aspects of IT. Over fifteen years, Patrick had taken on many different roles in IT in order to work in every role of an IT organization and gain a holistic understanding of IT. He worked as a developer, network specialist, system administrator, tester and project manager.
Patrick had taken a consulting job for a large datacenter migration. He was in charge of the testing, which meant he was spending a lot of time with Dev and Ops. Patrick had always been bothered by the differences between how Dev and Ops worked, but he became particularly frustrated with the challenges of managing work across the two groups on this datacenter migration.
Continuous integration was gaining popularity in the agile community and was moving Dev closer to deployment, but there was still nothing out there that fully crossed the divide of Dev and Ops. Patrick was sure there had to be a better way for these two teams to work together.
2008 – Andrew Shafer posted an idea for an agile infrastructure “birds of a feather” session at the Agile 2008 Conference. Patrick Debois saw the post and went to the session. Unfortunately, he was the only one who showed up. The idea was so poorly received that Andrew didn’t even show up to his own discussion.
Fortunately, Patrick was so excited to see that someone else was interested in solving the challenges of Dev and Ops working together that he tracked down Andrew and they decided to start a Google group named Agile System Administration.
2009 – John Allspaw, senior vice president of technical operations at Flickr, and Paul Hammond, director of engineering at Flickr, gave a presentation at the O’Reilly Velocity Conference in San Jose, “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” The presentation laid the groundwork for how Dev and Ops can effectively work together to improve software deployment.
Patrick Debois watched the presentation in Belgium via a live stream and was inspired to start his own conference, DevOpsDays, in Ghent, Belgium. The conference brought together an energetic group of forward-thinking minds trying to improve software deployment. What may be even more important is that this group of people kept the conversation going on Twitter with the hashtag #DevOpsDays. In an effort to save Twitter character space, people soon dropped days and the hashtag became #DevOps.
2010 – The following year, DevOpsDays were held in Australia and the U.S. Over time, there were more and more DevOpsDays that were hosted in different countries and cities around the world. The face-to-face meetings ignited more and more people to get energized about DevOps until it had become a full-on grassroots movement.
2011 – Up until 2011, the DevOps movement has been fueled by individuals and open source tools with little attention from analysts or vendors. But in 2011, the movement began to go mainstream, catching the attention of analysts like Cameron Haight from Gartner and Jay Lyman of 451 Research. Big vendors started to market DevOps.
2012 – By 2012 DevOps was quickly turning into a buzzword and DevOpsDays continued to grow.
2013 – The public thirst for DevOps information inspired several authors to write books on the topic. Notable examples are The Phoenix Project by Gene Kim, Kevin Behr and George Spafford and Implementing Lean Software Development by Mary and Tom Poppendiek.
2014 – Large companies such as Target, Nordstrom and LEGO became some of the first companies to bring DevOps into the enterprise.
DevOps Features
- DevOps is an approach based on agile and lean principles in which business owners, development, operations, and quality assurance team collaborate to deliver software in a continuous stable manner
- DevOps is an environment that promotes cross practicality, shared business tasks and belief
- DevOps is a movement that improves IT service delivery agility
- DevOps is a culture that promotes better working relationship within the company
- DevOps is a set of practices that provides rapid, reliable software delivery
Goal of DevOps
Improve collaboration between all stakeholders from planning through delivery and automation of the delivery process in order to:
- Improve deployment frequency
- Achieve faster time to market
- Lower failure rate of new releases
- Shorten lead time between fixes
- Improve mean time to recovery
Simple processes become increasingly programmable and dynamic, using a DevOps approach. DevOps aims to maximize the predictability, efficiency, security, and maintainability of operational processes. Very often, automation supports this objective.
DevOps integration targets product delivery, continuous testing, quality testing, feature development, and maintenance releases in order to improve reliability and security and provide faster development and deployment cycles. Many of the ideas (and people) involved in DevOps came from the enterprise systems management and agile software development movements.
Why DevOps
DevOps allows Agile Development Teams to implement Continuous Integration and Continuous Delivery. This helps them to launch products faster into the market. Other Important reasons are:
- Predictability: DevOps offers significantly lower failure rate of new releases
- Reproducibility: Version everything so that earlier version can be restored anytime.
- Maintainability: Effortless process of recovery in the event of a new release crashing or disabling the current system.
- Time to market: DevOps reduces the time to market up to 50% through streamlined software delivery. This is particularly the case for digital and mobile applications.
- Greater Quality: DevOps helps the team to provide improved quality of application development as it incorporates infrastructure issues.
- Reduced Risk: DevOps incorporates security aspects in the software delivery lifecycle. It helps in reduction of defects across the lifecycle.
- Resiliency: The Operational state of the software system is more stable, secure, and changes are auditable.
- Cost Efficiency: DevOps offers cost efficiency in the software development process which is always an aspiration of IT companies’ management.
- Breaks larger code base into small pieces: DevOps is based on the agile programming method. Therefore, it allows breaking larger code bases into smaller and manageable chunks.
DevOps Lifecycle
DevOps is deep integration between development and operations. Understanding DevOps is not possible without knowing DevOps lifecycle. A brief detail on the Continuous DevOps lifecycle is listed below.
- Development – In this DevOps stage the development of software takes place constantly. In this phase, the entire development process is separated into small development cycles. This benefits DevOps team to speed up software development and delivery process.
- Testing – QA team use tools like Selenium to identify and fix bugs in the new piece of code.
- Integration – In this stage, new functionality is integrated with the prevailing code, and testing takes place. Continuous development is only possible due to continuous integration and testing.
- Deployment – In this phase, the deployment process takes place continuously. It is performed in such a manner that any changes made any time in the code, should not affect the functioning of high traffic website.
- Monitoring – In this phase, operation team will take care of the inappropriate system behavior or bugs which are found in production.
Relationship to other approaches
Agile – Agile and DevOps both often utilize practices such as automated build and test, continuous integration, and continuous delivery. Agile can be viewed as addressing communication gaps between customers and developers, while DevOps addresses gaps between developers and IT operations / infrastructure. Also, DevOps has focus on the deployment of developed software, whether it is developed via Agile or other methodologies.
Continuous delivery – Continuous delivery and DevOps have common goals and are often used in conjunction, but there are subtle differences.
While continuous delivery is focused on automating the processes in software delivery, DevOps also focuses on the organization change to support great collaboration between the many functions involved.
DevOps and continuous delivery share a common background in agile methods and lean thinking: small and frequent changes with focused value to the end customer. They are well communicated and collaborated internally, thus helping achieve faster time to market, with reduced risks.
DevOps Sub-Concepts
DevOps includes different sub concepts such as
Continuous Integration: It is more of a culture of integrating application code or source code into the source code repositories such as SVN and GIT several times a day. Build tools such as Ant and Maven are used to verify check in by integrating source code repositories and continuous integration tools such as Jenkins, Atlassian Bamboo, etc.
Continuous Testing: Automated Testing or Unit testing to be more precise. Junit or Selenium is used for such kind of automated testing and these tools are integrated in the continuous integration behavior by tools.
Continuous Provisioning or Cloud Provisioning: Virtual machines or cloud instance are highly available, flexible and pay per use. It is beneficial to create different types of instances for different environment such as development, testing, staging, and production. Different cloud service providers such as Microsoft Azure, Amazon Web services can be used. Different Cloud service models can be taken into consideration such as Infrastructure as a Service or Platform as a Service.
Configuration Management: Configuration management is useful to make runtime environment ready in consistent manner across all the environments. Chef, Puppet or Ansible can be used as configuration management tools. It is very easy to manage resources with such tools in Cloud environment.
Continuous Delivery or Continuous Deployment: Continuous Delivery or Continuous Deployment is terms which used loosely in day to day use. Continuous Delivery can be used when package file is deployment ready into any environment while Continuous Deployment term can be used when package file is production deployment ready.
It is always useful to orchestrate end to end process of application release management. Visibility into orchestration gives insight into end to end automation process and that is highly useful in creating and maintaining DevOps culture.
DevOps Automation
DevOps automation can be achieved by repackaging platforms, systems, and applications into reusable building blocks through the use of technologies such as virtual machines and containerization.
Implementation of DevOps automation in the IT-organization is heavily dependent on tools, which are required to cover different areas of the systems development lifecycle (SDLC):
- Infrastructure as code — Ansible, Terraform, Puppet, Chef
- CI/CD — Jenkins, TeamCity, Shippable, Bamboo, Azure DevOps
- Test automation — Selenium, Cucumber, Apache JMeter
- Containerization — Docker, Rocket, Unik
- Orchestration — Kubernetes, Swarm, Mesos
- Software deployment — Elastic Beanstalk, Octopus, Vamp
- Measurement — NewRelic, Kibana, Datadog, DynaTrace
- ChatOps — Hubot, Lita, Cog
Integration and Delivery
Continuous Integration (CI) – Developers integrate code into a shared repository multiple times a day and each isolated change to the code is tested immediately in order to detect and prevent integration problems. Examples of vendors and tools: Jenkins, Teamcity, TravisCI, CircleCI
Continuous Delivery (CD) – As an extension of CI and the next step in incremental software delivery, continuous delivery (CD) ensures that every version of the code that is tested in the CI repository can be released at any moment. Examples of vendors and tools: Jenkins, Teamcity, TravisCI, Electric Cloud, Go, Codeship, AWS CodeDeploy
Continuous Deployment – Continuous deployment can be thought of as an extension of continuous integration, aiming to have a new code deployed in production to be used by live users. Supported by CI, when tests meet the release criteria, it is released immediately.
Configuration Management (CM)
In a nutshell, the process of maintaining up-to-date, detailed records of hardware and software — including versions, requirements, network addresses, and design and operational information — is known as configuration management (CM). You can use CM tools such as Chef, Puppet, or Ansible to aid the process. You can also use Bash and Python to build your own configuration management automation. Examples of vendors and tools: Chef, Puppet, Ansible, Saltstack, Vagrant, CFEngine
Resource Orchestration
When it comes to micro-services, service-oriented architecture, converged infrastructures, virtualization, and provisioning, the coordination and integration between computer systems is known as orchestration. By leveraging defined automated workflows, orchestration makes sure that business needs are aligned with your IT infrastructure resources.
Containers
Containers are lightweight virtualization components that run isolated application workloads. They run their own processes, file systems, and network stacks, which are all virtualized using the root operating system (OS) running on the hardware. Examples of vendors and related tools: Docker, CoreOs, Kubernetes, Mesos, ElasticBox
Source (Version) Control
Version control includes practices and tools that help R&D organizations maintain and control changes within their source code repository. R&D members use source control tools to document and track system configuration files as well. Examples of vendors and tools: GitHub, Bitbucket, JFrog, Artifactory
Bug Tracking
Bug tracker is a system that aggregates and reports software bugs and defects. It helps R&D organizations with task management and is part of the consistent feedback loop that the DevOps methodology requires. Examples of vendors and tools: BUGtrack, JIRA, GitHub
Test Automation
Test automation facilitates test engineer work by supporting multiple tests that run continuously. It enhances test coverage while supporting efficient release cycles. For example, test automation tools help manage, execute, and measure functional tests and load tests. Examples of vendors and tools: Selenium, Cucumber, JUnit, TestNG, JMeter
Unit Test – Unit testing is a process that allows testers to examine small parts of an application, such as a specific code or module. This test is usually automated and reused in order to support continuous testing and integration.
Monitoring
Monitoring is a primary element of IT performance management and is one of the most important aspects when operating online services. Monitoring tools are essential and provide crucial information that helps to ensure service robustness in terms of availability, security, and performance.
Application Performance Monitoring (APM) – APM allows you to automatically detect and be alerted about hotspots in your application framework that include the application and database layers. Examples of vendors and tools: New Relic, AppDynamics, DataDog
Infrastructure Monitoring – Tools in this category automatically detect and alert about degradations in underlying physical or virtual resource performance and availability. Examples of vendors and tools: AWS CloudWatch, Nagios, Zabbix, Sensu, Icynga
Log Management
Log management (or log analytics) is the practice of dealing with large volumes of computer-generated messages. They can be operational messages (e.g., when tracking service performance or security) or for BI purposes (e.g., when tracking online user behavior). Vendors and tools: Logz.io (ELK), Splunk, Sumo Logic
