Open Source DevOps in Software Development
ABSTRACT
CI and CD are foundational practices being adopted by agile organizations to accelerate their development and test processes to meet the new demands of mobile and IoT. This paper looks at the benefits of CI and CD together and the requirements for effectively deploying these practices in Schneider Electric.
- Arka Ghosh
Introduction to DevOps
DevOps culture is gaining rapid momentum in the IT industry as it enables business to adopt agile software delivery methodologies like Continuous Integration (henceforth referred to as CI) & Continuous Delivery (henceforth referred to as CD) .These methodologies enable quicker issue resolution, instant feedback loops, improved software quality and cost saving to meet the ever-increasing demand to deliver better software faster. It would not be wrong to say that CICD practices will soon become the de-facto software delivery standards across the industry. Though both these methodologies (CI and CD) complement each other, CI is the pre-requisite phase for enabling CD as the latter is built on top of the former. The primary goal of CI is not only to enable build automation through continuous test and quality checks but also to provide project insights through reports and dashboards. Since this phase is all about tools, it imposes various integration challenges. Having a good knowledge of the tools involved, their integration aspects and the best practices to follow, will definitely enable their smoother adoption and rollout across enterprises looking to adopt CI practices.
• What is CI/CD in software development
• What are the advantages of setting up a CI/CD architecture
• What open source tools can we use to achieve full automation
• How to integrate all the tools together to build a complete workflow
Continuous Integration and Continuous Delivery (CI/CD)
Continuous Integration and Delivery (CI/CD) is a set of software practices and techniques that enable the frequent release of small batches of code changes, with extensive visibility and traceability. It typically involves the creation of a largely automated pipeline that orchestrates the build, test and deployment of software across staged environments, ultimately leading to deployment in production.
Continuous Integration Architecture
The below diagram illustrates the end to end Continuous Integration (CI) setup which we have been following across projects. As seen below, the main actors include the Development team, the Source Control Server and the Continuous Integration server. Developers check-in the code into source control server which is integrated with CI server. For each build, CI server is configured to run the JUnit test cases, static code quality checks, build binaries, publish artifacts, build container images and deploy them to environments. This continuous automation process helps in reducing the overall defect density and thereby improving the release automation process.
Advantages of doing CI/CD automation
Faster identification & resolution of defects
CI/CD allows an elegant way to establish the appropriate quality gates in the development and testing process. A fast feedback loop to the developers ensures that bugs are addressed early in the development cycle.
Better Quality assurance
CI/CD enables QA teams to release deployable software at any point in time. Without it, projects are prone to delayed releases because of unforeseen issues which arise at any point in the traditional development and test process.
Faster time to market
Faster test and QA cycles enable organizations to get quality products and services to market faster and more efficiently.
Reduced overhead cost
Finding a bug at the development stage is the cheapest possible way to find it. If the same bug was to be fixed in any other environment, it would cost more. CI/CD requires some upfront overhead cost, but these are more than offset by the time and expense saved along the way.
Infrastructure on Cloud
I have used 3 EC2 machines(dev, qa & prod) on Azure Cloud to setup my workflow These can be setup using the available MSDN Enterprise subscription.
CI/CD Tool Landscape
CI depends mostly on adopting the correct set of tools and their proper usage. The selection of tools is generally driven by various IT policies in the organization, existing technology landscape, current infrastructure setup, and other considerations. In my project I have used a set of Open Source tools that can be deployed by any team. The figure below shows the tools which we have been using successfully across various projects for CI enablement. As seen, we have adopted Jenkins as the continuous integration platform but there are other CI platforms (Bamboo, TeamCity etc.) to choose from. The same case holds good for other toolsets as well.
GitHub - Source Code Management
Gradle - Build Tool Automation
Junit Tests
Jfrog Artifactory Manager
Jacoco – Java Code Coverage
SonarQube – Static Code Analysis
Jenkins – CI Engine
Docker - Containerization
Ansible -Configuration Deployment
• Jenkins is the most famous Continuous Integration tool. (other examples - circle CI,
Jenkins• Jenkins is an open source automation tool written in Java with plugins built for Continuous Integration purpose.
• Jenkins integrates development life-cycle processes of all kinds, including build, document, test, package, stage, deploy, static analysis and much more.
• Continuous Integration with the help of plugins. Some examples like Amazon EC2, HTML publisher, GitHub, Copy Artifact etc.
• http://52.179.9.125:8080/
Advantages of Jenkins
• It is a free, open source tool with great community support.
• It has 1000+ plugins to ease your work. If a plugin does not exist, you can code it and share with the community.
• It is built with Java and hence, it is portable to all the major platforms.
GitHub – Source Code Management
GitHub is a web-based hosting service for version control using Git. It offers all of the distributed version control and source code management (SCM) functionality of Git as well as adding its own features.
It provides access control and several collaboration features such as bug tracking, feature requests, task management, and wikis for every project.
GitHub offers plans for enterprise, team, pro and free accounts which are commonly used to host open-source software projects.
In our project we have created a GitHub account and checked -in a sample project that we will build and deploy. Below screenshot shows how we ca create a repository and push it to the remote GitHub
Gradle - Build Automation
Groovy based DSL that adds some tasks to your project which will compile and unit test your Java source code, and bundle it into a JAR file.
Start by downloading the Gradle Binary-only zip file in the /tmp directory using the following wget command: wget https://services.gradle.org/distributions/gradle-5.0-bin.zip -P /tmp
Once the download is completed, extract the zip file in the /opt/gradle directory:
sudo unzip -d /opt/gradle /tmp/gradle-*.zip
Verify that the Gradle files are extracted by listing the /opt/gradle/gradle-5.0 directory:
ls /opt/gradle/gradle-5.0
bin getting-started.html init.d lib LICENSE media NOTICE
After Gradle installation the final project directory should look like this. Notice the build.gradle file.
JaCoCo - Java Code Coverage
• JaCoCo is a free code coverage library for Java Unit and Integration Tests.
• It can be easily integrated with CI tools like Jenkins
• Integration is also possible with standard IDE like Eclipse, IntelliJ etc.
• It publishes easily understandable HTML reports about code coverage
Jacoco configuration
1. Add JaCoCo to the Gradle configuration.
2. Add the code coverage stage to the pipeline.
3. Optionally, publish JaCoCo reports in Jenkins. In the pipeline steps 1and 2 should be enough to run Code coverage analysis. However to integrate coco results in Jenkins : Install HTML plugin and Install Jacoco plugin
publishHTML (target: [ reportDir: 'build/reports/jacoco/test/html', reportFiles: 'index.html', reportName: "JaCoCo Report" ])
Once Jacoco is configured in our project the results can be viewed in Jenkins. Clicking on the “JaCoCo Report” link will give is the code coverage results.
The given HTML report shows us the code coverage based on various parameters (lines, methods etc).
SonarQube – Static Code Quality
SonarQube is an open-source platform for continuous static analysis of code to detect bugs, code smells, and security vulnerabilities on 25+ programming languages.
Post SonarQube installation is done we can run analysis on our source code to view the quality score. For it we need to go to the root directory of our project and run the command –“sonarscanner”. Based on the values that we have earlier set in the sonar.properties, sonar project.properties and sonar-scanner.properties configuration files, the scanner runs an analysis as shown below.
Docker - Containerization
Docker is a tool to create, deploy, and run applications by using containers. Docker is a software platform designed to make it easier to create, deploy, and run applications by using containers. Developers can package an application and all its dependencies in a container ship out as a package. Docker Hub is a cloud-based repository in which Docker users and partners create, test, store and distribute container images.
https://hub.docker.com/
I need to create a Docker image with JRE, the compiled and packaged Java application, and all 3rd party libraries.
Create a Dockerfile
FROM ubuntu:latest
MAINTAINER Arka Ghosh "arka.ghosh@se.com"
WORKDIR .
# Install Java 8
RUN apt-get update -y
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get update -y
RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
RUN apt-get install oracle-java8-installer -y
ADD build/libs/onlinelibrary.jar onlinelibrary.jar
CMD java -jar onlinelibrary.jar
Build an image from the Dockerfile
>docker build -t arghosh/onlinelibrary .
Run the image as container
>docker run arghosh/onlinelibrary
See the image has been created or not
> docker images
Push the image to the Docker Hub registry
> docker login username <username> password <password>
> docker push arghosh/onlinelibrary:latest
Ansible – Provisioning & Deployment
Ansible is an open-source provisioning, configuration management, and application deployment tool.
• Provisioning: Set up the various servers you need in your infrastructure.
• Configuration managementI: mplement a security policy, start and stop services; install or update applications.
• Application deployment: Automating the deployment of internally developed applications to your production system
Inventory - Description of the nodes that can be accessed by Ansible. Usually defined in “/etc/ansible/hosts” file.
Playbooks - Playbooks are YAML files that express configurations, deployment, and orchestration in Ansible
Modules – Can be executed directly on remote hosts or via playbooks. Example are users, packages, shell etc.
So, if I want to run a shell command on a remote host to find all running processes via Ansible, it would like :
> ansible -i /etc/ansible/hosts -m shell -a "ps -eaf" 198.168.0.1
The Ansible deploy.yaml file will look like this. We can invoke this script from Jenkins via command:
sh "ansible-playbook /home/sesa475934/projects/Library/ansible/deploy.yml extra-vars 'host_name=prod_server remote_username=sesa475934'"
Trigerring Job on Jenkins
Once the entire CI/CD setup is complete, we can run it from the Jenkins page or by checking in code in GitHub Enterprise.
We can change over to the Blue Ocean page for better Jenkins GUI experience.
At the same time we can check the logs on the screen while each phase of the pipeline is running.
Conclusion
So, in this whitepaper I have researched and shown how we can achieve complete CI/CD for our R&D work using Open Source tools. Such an integration targets product delivery, quality testing, feature development, and maintenance releases in order to improve reliability and security and provide faster development and deployment cycles.