Building CI/CD Pipelines

# Building CI/CD Pipelines

Introduction

CI/CD (Continuous Integration/Continuous Deployment) pipelines are automated workflows that build, test, and deploy software changes. CI/CD pipelines enable teams to deliver code changes frequently, reliably, and with minimal manual intervention. This tutorial covers pipeline design, stages, quality gates, deployment strategies, rollback procedures, and pipeline-as-code implementation.

Learning Objectives

By the end of this tutorial, you will be able to:

• Design CI/CD pipeline architecture with appropriate stages
• Implement CI/CD pipelines using Jenkins, GitHub Actions, and GitLab CI
• Configure quality gates and automated testing
• Apply deployment strategies (blue-green, canary, rolling)
• Implement rollback procedures for failed deployments
• Manage pipeline security and secrets
• Monitor and optimize pipeline performance

Technical Background

CI/CD Pipeline Stages

flowchart LR
    Commit[Code Commit] --> Build[Build]
    Build --> Test[Test]
    Test --> Package[Package]
    Package --> Deploy-Staging[Deploy Staging]
    Deploy-Staging --> Test-Staging[Integration Tests]
    Test-Staging --> Gate[Quality Gate]
    Gate --> Deploy-Prod[Deploy Production]
    Deploy-Prod --> Verify[Verify]
    Verify --> Monitor[Monitor]
    
    subgraph CI
    Commit
    Build
    Test
    Package
    end
    
    subgraph CD
    Deploy-Staging
    Test-Staging
    Gate
    Deploy-Prod
    Verify
    Monitor
    end

Pipeline Stages

| Stage | Purpose | Tools | |-------|---------|-------| | Build | Compile code, create artifacts | Maven, Gradle, npm, Docker build | | Test | Run unit, integration, E2E tests | Jest, PyTest, JUnit, Selenium | | Package | Create versioned artifacts | Docker, Helm, npm publish | | Deploy | Deploy to target environment | kubectl, AWS ECS, Azure DevOps | | Verify | Smoke tests, health checks | curl, k6, postman | | Monitor | Track deployment health | Prometheus, Datadog, CloudWatch |

Deployment Strategies

| Strategy | Description | Risk | Rollback Speed | |----------|-------------|------|----------------| | Rolling | Replace instances gradually | Low | Fast | | Blue-Green | Switch traffic between environments | Very Low | Instant | | Canary | Route small % traffic to new version | Lowest | Fast | | Recreate | Stop old, start new | High | Slow |

Prerequisites

• Source code repository (GitHub, GitLab, Bitbucket)
• CI/CD platform (Jenkins, GitHub Actions, GitLab CI, Azure DevOps)
• Container registry (Docker Hub, ECR, ACR, GCR)
• Target deployment environment (Kubernetes, ECS, VMs)
• Understanding of application architecture and dependencies

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | CI Server | Jenkins, GitHub Actions runner, or GitLab runner | | Container Runtime | Docker | | Orchestration | Kubernetes, ECS, or similar | | Registry | Docker Hub, ECR, ACR, or GCR | | Cloud Account | AWS, Azure, or GCP | | CLI Tools | kubectl, awscli, azure-cli, or gcloud |

Step-by-Step Implementation

Phase 1: Pipeline Design

#### Pipeline Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Build     │───▶│   Test      │───▶│  Package    │
│ (compile,   │    │ (unit, int, │    │ (container, │
│  lint, scan)│    │  e2e, sec)  │    │  version)   │
└─────────────┘    └─────────────┘    └─────────────┘
       │                   │                   │
       ▼                   ▼                   ▼
┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Deploy     │───▶│   Verify    │───▶│  Monitor    │
│ (staging)   │    │ (smoke test)│    │ (alerting)  │
└─────────────┘    └─────────────┘    └─────────────┘
       │
       ▼ (if approved)
┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Deploy     │───▶│   Verify    │───▶│  Monitor    │
│ (production)│    │ (smoke test)│    │ (alerting)  │
└─────────────┘    └─────────────┘    └─────────────┘

#### Quality Gates

Quality gates prevent low-quality code from progressing:

1. **Code quality:** Linting passes, no critical issues 2. **Test coverage:** Minimum 80% line coverage 3. **Security scan:** No high/critical vulnerabilities 4. **Performance:** Response time under threshold 5. **Dependencies:** No outdated or vulnerable packages

Phase 2: Implement CI/CD with GitHub Actions

#### Complete CI/CD Pipeline

# .github/workflows/cicd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [develop]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # CI Stage
  build-and-test:
    runs-on: ubuntu-latest
    
    outputs:
      image-tag: ${{ steps.tagging.outputs.image-tag }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha
            type=ref,event=branch
      
      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }
          labels: ${{ steps.meta.outputs.labels }}
      
      - name: Run Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.meta.outputs.version }}
          format: table
          exit-code: 1
          severity: CRITICAL,HIGH
      
      - name: Set image tag output
        id: tagging
        run: echo "image-tag=${{ steps.meta.outputs.version }}" >> $GITHUB_OUTPUT

  # Deploy to Staging
  deploy-staging:
    needs: build-and-test
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}
      
      - name: Deploy to staging
        run: |
          kubectl set image deployment/my-app \
            my-app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ needs.build-and-test.outputs.image-tag }} \
            -n staging
      
      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/my-app -n staging --timeout=300s
      
      - name: Run smoke tests
        run: |
          curl -f http://staging.myapp.com/health || exit 1

  # Deploy to Production (Manual Approval)
  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://myapp.com
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.KUBE_CONFIG_PROD }}
      
      - name: Blue-Green Deploy
        run: |
          # Deploy to green environment
          kubectl apply -f k8s/green/
          kubectl rollout status deployment/my-app-green -n production --timeout=300s
          
          # Run smoke tests on green
          curl -f http://green.myapp.com/health || exit 1
          
          # Switch traffic to green
          kubectl patch service my-app -n production \
            -p '{"spec":{"selector":{"version":"green"}}}'

Phase 3: Implement CI/CD with Jenkins

#### Jenkins Pipeline (Declarative)

// Jenkinsfile
pipeline {
    agent any
    
    environment {
        REGISTRY = "docker.io/myorg"
        IMAGE_NAME = "my-app"
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        
        stage('Build') {
            steps {
                script {
                    dockerImage = docker.build("${env.REGISTRY}/${env.IMAGE_NAME}:${env.BUILD_NUMBER}")
                }
            }
        }
        
        stage('Test') {
            steps {
                sh 'npm test -- --coverage'
                sh 'npm run lint'
            }
            post {
                always {
                    junit '**/test-results/*.xml'
                }
            }
        }
        
        stage('Security Scan') {
            steps {
                sh 'trivy image --exit-code 1 --severity HIGH,CRITICAL ${env.REGISTRY}/${env.IMAGE_NAME}:${env.BUILD_NUMBER}'
            }
        }
        
        stage('Push Image') {
            steps {
                script {
                    docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-credentials') {
                        dockerImage.push("${env.BUILD_NUMBER}")
                        dockerImage.push('latest')
                    }
                }
            }
        }
        
        stage('Deploy Staging') {
            steps {
                script {
                    sh '''
                    kubectl set image deployment/my-app \
                        my-app=${env.REGISTRY}/${env.IMAGE_NAME}:${env.BUILD_NUMBER} \
                        -n staging
                    kubectl rollout status deployment/my-app -n staging --timeout=300s
                    '''
                }
            }
        }
        
        stage('Production Approval') {
            steps {
                input message: 'Deploy to production?', ok: 'Deploy'
            }
        }
        
        stage('Deploy Production') {
            steps {
                script {
                    sh '''
                    kubectl set image deployment/my-app \
                        my-app=${env.REGISTRY}/${env.IMAGE_NAME}:${env.BUILD_NUMBER} \
                        -n production
                    kubectl rollout status deployment/my-app -n production --timeout=300s
                    '''
                }
            }
        }
    }
    
    post {
        always {
            cleanWs()
        }
        failure {
            mailtop(
                subject: "Pipeline Failed: ${env.JOB_NAME} - ${env.BUILD_NUMBER}",
                to: "devops-team@example.com"
            )
        }
    }
}

Phase 4: Deployment Strategies

#### Blue-Green Deployment with Kubernetes

# k8s/blue-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-blue
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: blue
  template:
    metadata:
      labels:
        app: my-app
        version: blue
    spec:
      containers:
      - name: my-app
        image: myapp:blue-version
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-green
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: green
  template:
    metadata:
      labels:
        app: my-app
        version: green
    spec:
      containers:
      - name: my-app
        image: myapp:green-version
---
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: production
spec:
  selector:
    app: my-app
    version: blue  # Change to green to switch traffic
  ports:
  - port: 80
    targetPort: 8080

#### Canary Deployment with Argo Rollouts

# k8s/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app-rollout
  namespace: production
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 100
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: myapp:new-version

Phase 5: Rollback Procedures

#### Kubernetes Rollback

# View rollout history
kubectl rollout history deployment/my-app -n production

# Rollback to previous revision
kubectl rollout undo deployment/my-app -n production

# Rollback to specific revision
kubectl rollout undo deployment/my-app -n production --to-revision=3

# Check rollout status
kubectl rollout status deployment/my-app -n production

#### Automated Rollback in CI/CD

# .github/workflows/cicd.yml - rollback step
      - name: Rollback on failure
        if: failure()
        run: |
          kubectl rollout undo deployment/my-app -n production
          echo "Rolled back to previous version"

Expected Output

• Automated CI/CD pipeline building, testing, and deploying code
• Container images built and pushed to registry
• Applications deployed to staging and production
• Quality gates preventing bad code from progressing
• Rollback capability for failed deployments
• Pipeline monitoring and alerting

Verification Procedures

| Check | Method | Expected Result | |-------|--------|-----------------| | Pipeline execution | Trigger pipeline | All stages pass successfully | | Container build | Check registry | Image pushed with correct tag | | Deployment | Check Kubernetes | Pods running and ready | | Smoke test | Curl health endpoint | Returns 200 OK | | Rollback | Trigger rollback | Previous version restored | | Quality gate | Introduce failing test | Pipeline blocks deployment |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Build fails | Compilation error or missing dependency | Fix code; update build configuration | | Test fails | Test assertion failure or environment issue | Fix test or test environment | | Security scan fails | Vulnerable dependency | Update dependency or add exception | | Deployment fails | Image pull error or resource quota | Check image tag; increase quota | | Rollback doesn't work | Previous revision unavailable | Ensure revision history is maintained | | Pipeline slow | Large dependencies or sequential steps | Cache dependencies; parallelize steps |

Best Practices

• **Pipeline as code:** Store pipeline definitions in version control
• **Fail fast:** Run fast checks (lint, unit tests) before slow ones
• **Cache dependencies:** Speed up builds by caching npm, Maven, pip packages
• **Use container images:** Consistent build and test environments
• **Implement quality gates:** Block deployments that don't meet criteria
• **Automate rollback:** Test rollback procedures regularly
• **Monitor pipeline:** Track build times, success rates, and failure causes
• **Separate environments:** Use different clusters/namespaces for staging and production

Security Considerations

• **Secret management:** Store CI/CD secrets in platform secret store, not in code
• **Least privilege:** CI/CD service accounts should have minimum required permissions
• **Image scanning:** Scan container images for vulnerabilities before deployment
• **Signed commits:** Require GPG-signed commits for production deployments
• **Pipeline access:** Restrict who can trigger production deployments
• **Audit logging:** Log all pipeline executions and changes

Conclusion

CI/CD pipelines automate the software delivery process from code commit to production deployment. By implementing proper pipeline design, quality gates, deployment strategies, and rollback procedures, teams can deliver software faster, more reliably, and with less risk. Pipeline as code ensures consistency, version control, and reproducibility across all environments.

References

• Jenkins Pipeline: https://www.jenkins.io/doc/book/pipeline/
• GitHub Actions: https://docs.github.com/en/actions
• GitLab CI/CD: https://docs.gitlab.com/ee/ci/
• Kubernetes Deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
• Argo Rollouts: https://argoproj.github.io/argo-rollouts/
• Continuous Delivery: https://continuousdelivery.com/