my blog

posts
omfsakib Sept. 23, 2024, 8:44 a.m. frappe, ci/cd, erpnext, devops, continuous integrations, continuous development.

Frappe CI/CD Integration Local -> Git -> VPS With Only Customizations

Push Script / push.sh

Introduction

This shell script is designed to streamline the process of backing up a Frappe site and managing Git commits and pushes. It performs the following operations:

  • Backup: Creates a backup of the Frappe site’s database with specific settings.
  • Git Add: Stages all changes in the Git repository.
  • Git Commit: Commits the changes with a user-provided message.
  • Git Push: Pushes the committed changes to a specified branch.

Purpose

The script ensures that a Frappe site is backed up before committing and pushing changes to a Git repository. This process helps maintain data integrity and version control by:

  • Automating the backup process to avoid manual errors.
  • Ensuring that changes are consistently committed and pushed to the correct branch.
  • Providing clear error messages if any step fails.

Script Code

#!/bin/bash

# Default values
MESSAGE=""
SITE_NAME=""
BRANCH_NAME=""

# Parse command-line arguments
while getopts "m:s:b:" opt; do
  case ${opt} in
    m )
      MESSAGE=$OPTARG
      ;;
    s )
      SITE_NAME=$OPTARG
      ;;
    b )
      BRANCH_NAME=$OPTARG
      ;;
    \? )
      echo "Invalid option: -$OPTARG" 1>&2
      exit 1
      ;;
    : )
      echo "Invalid option: -$OPTARG requires an argument" 1>&2
      exit 1
      ;;
  esac
done
shift $((OPTIND -1))

# Check if all mandatory variables are provided
if [ -z "$MESSAGE" ]; then
    echo "Error: Commit message (-m) is required."
    exit 1
fi

if [ -z "$SITE_NAME" ]; then
    echo "Error: Site name (-s) is required."
    exit 1
fi

if [ -z "$BRANCH_NAME" ]; then
    echo "Error: Branch name (-b) is required."
    exit 1
fi

# Step 1: Run the backup
echo "Starting Backup for site $SITE_NAME..."
bench --site $SITE_NAME backup --only 'Scheduled Job Type,Log Settings,Logs To Clear,Document Naming Settings,Amended Document Naming Settings,System Settings,User Type,Navbar Settings,Navbar Item,Installed Applications,Installed Application,Server Script,Translation,Language,Report,Report Filter,Report Column,DocType,DocPerm,DocType Action,DocType Link,DocType State,DocField,Patch Log,SMS Settings,SMS Parameter,Module Def,Role,Page,Custom HTML Block,Changelog Feed,Form Tour,Form Tour Step,Number Card,Dashboard Settings,Workspace,Workspace Number Card,Workspace Chart,Workspace Shortcut,Workspace Link,Workspace Quick List,Workspace Custom Block,List View Settings,Global Search Settings,Global Search DocType,Dashboard Chart Source,Dashboard,Dashboard Chart Link,Number Card Link,Dashboard Chart,Dashboard Chart Field,Custom Field,DocType Layout,DocType Layout Field,Customize Form,Customize Form Field,Property Setter,Client Script,Auth Page Customize,Theme Customize,Service Worker,Web App Manifest,Web App Manifest Icon,Web App Manifest Category,Web App Manifest Screenshot,Web App Manifest Related Application,Audit Trail,Energy Point Settings,Energy Point Rule,Account,Email Domain,Singles,Assignment Rule,Assignment Rule Day,Assignment Rule User' --backup-path-db $SITE_NAME/private/backups/database_backup.sql.gz
if [ $? -ne 0 ]; then
    echo "Backup failed. Exiting."
    exit 1
fi
echo "Backup completed successfully."

# Step 2: Run Git Add
echo "Adding changes to Git..."
git add .
if [ $? -ne 0 ]; then
    echo "Git add failed. Exiting."
    exit 1
fi
echo "Git add completed successfully."

# Step 3: Run Git Commit
echo "Committing changes with message: $MESSAGE"
git commit -m "$MESSAGE"
if [ $? -ne 0 ]; then
    echo "Git commit failed. Exiting."
    exit 1
fi
echo "Git commit completed successfully."

# Step 4: Run Git Push
echo "Pushing changes to branch $BRANCH_NAME..."
git push origin $BRANCH_NAME
if [ $? -ne 0 ]; then
    echo "Git push failed. Exiting."
    exit 1
fi
echo "Git push completed successfully."

Setup Script / setup.sh

Introduction

This shell script automates the setup and configuration of a Frappe environment. It includes steps for environment setup, configuration, database setup, application installation, and customizations. This script ensures that all necessary components are properly configured and ready for use.

Purpose

The primary purpose of this script is to streamline the setup process for a Frappe environment, making it easier to deploy and configure a new site. The script handles the following tasks:

  1. Environment setup
  2. Configuration setup
  3. Redis setup
  4. Installation of development, Python, and Node.js requirements
  5. Database configuration
  6. Site installation
  7. Application installation
  8. Customization restoration
  9. Database migration
  10. Build process

Script Code

#!/bin/bash

# Function to check if a variable is set and not empty
function check_var() {
    if [ -z "$2" ]; then
        echo "$1 is required but not set. Exiting."
        exit 1
    fi
}

# Parse command line arguments
while getopts "s:d:u:p:a:" opt; do
    case $opt in
        s) SITE_NAME="$OPTARG"
        ;;
        d) DB_NAME="$OPTARG"
        ;;
        u) DB_USER="$OPTARG"
        ;;
        p) DB_PASS="$OPTARG"
        ;;
        a) APP_NAMES="$OPTARG"
        ;;
        *) echo "Invalid option"; exit 1;
        ;;
    esac
done

# Check if all required variables are set
check_var "SITE_NAME (-s)" "$SITE_NAME"
check_var "DB_NAME (-d)" "$DB_NAME"
check_var "DB_USER (-u)" "$DB_USER"
check_var "DB_PASS (-p)" "$DB_PASS"
check_var "APP_NAMES (-a)" "$APP_NAMES"

# Step 1: Run env setup
echo "Setting up ENV"
bench setup env
if [ $? -ne 0 ]; then
    echo "ENV Setup failed. Exiting."
    exit 1
fi
echo "ENV Setup completed successfully."

# Step 2: Run procfile setup
echo "Setting up Procfile"
bench setup procfile
if [ $? -ne 0 ]; then
    echo "Procfile Setup failed. Exiting."
    exit 1
fi
echo "Procfile Setup completed successfully."

# Step 3: Run config setup
echo "Setting up Config"
bench setup config
if [ $? -ne 0 ]; then
    echo "Config Setup failed. Exiting."
    exit 1
fi
echo "Config Setup completed successfully."

# Step 4: Run redis setup
echo "Setting up Redis"
bench setup redis
if [ $? -ne 0 ]; then
    echo "Redis Setup failed. Exiting."
    exit 1
fi
echo "Redis Setup completed successfully."

# Step 5: Run requirements dev setup
echo "Setting up Requirements --DEV"
bench setup requirements --dev
if [ $? -ne 0 ]; then
    echo "Requirements --DEV Setup failed. Exiting."
    exit 1
fi
echo "Requirements --DEV Setup completed successfully."

# Step 6: Run requirements python setup
echo "Setting up Requirements --PYTHON"
bench setup requirements --python
if [ $? -ne 0 ]; then
    echo "Requirements --PYTHON Setup failed. Exiting."
    exit 1
fi
echo "Requirements --PYTHON Setup completed successfully."

# Step 7: Run requirements node setup
echo "Setting up Requirements --NODE"
bench setup requirements --node
if [ $? -ne 0 ]; then
    echo "Requirements --NODE Setup failed. Exiting."
    exit 1
fi
echo "Requirements --NODE Setup completed successfully."

# Step 8: Run DB-Name setup
echo "Setting up DB-Name"
bench --site $SITE_NAME set-config db-name "$DB_NAME"
if [ $? -ne 0 ]; then
    echo "DB-Name Setup failed. Exiting."
    exit 1
fi
echo "DB-Name Setup completed successfully."

# Step 9: Run DB-USER setup
echo "Setting up DB-USER"
bench --site $SITE_NAME set-config db-user "$DB_USER"
if [ $? -ne 0 ]; then
    echo "DB-USER Setup failed. Exiting."
    exit 1
fi
echo "DB-USER Setup completed successfully."

# Step 10: Run DB-PASSWORD setup
echo "Setting up DB-PASSWORD"
bench --site $SITE_NAME set-config db-pass "$DB_PASS"
if [ $? -ne 0 ]; then
    echo "DB-PASSWORD Setup failed. Exiting."
    exit 1
fi
echo "DB-PASSWORD Setup completed successfully."

# Step 11: Run Site Install
bench --site $SITE_NAME reinstall
if [ $? -ne 0 ]; then
    echo "Site Install Setup failed. Exiting."
    exit 1
fi
echo "Site Install Setup completed successfully."

# Step 12: Run APP Install
bench --site $SITE_NAME install-app $APP_NAMES
if [ $? -ne 0 ]; then
    echo "APP Install Setup failed. Exiting."
    exit 1
fi
echo "APP Install Setup completed successfully."

# Step 13: Run Customization Install
bench --site $SITE_NAME --force partial-restore $SITE_NAME/private/backups/database_backup.sql.gz
if [ $? -ne 0 ]; then
    echo "Customization Install Setup failed. Exiting."
    exit 1
fi
echo "Customization Install Setup completed successfully."

# Step 14: Run DB Migration
bench --site $SITE_NAME restore-schema
if [ $? -ne 0 ]; then
    echo "DB Migration Setup failed. Exiting."
    exit 1
fi
echo "DB Migration Setup completed successfully."

# Step 15: Run Build
bench build
if [ $? -ne 0 ]; then
    echo "Build Setup failed. Exiting."
    exit 1
fi
echo "Build Setup completed successfully."

CI/CD Pipeline for Deployment

Introduction

This document provides the configuration for a CI/CD pipeline that automates the deployment of a Frappe application to an EC2 instance. The pipeline is triggered on push events to the development branch and handles tasks such as code checkout, SSH installation, and deployment using a shell script.

Purpose

The purpose of this CI/CD pipeline is to streamline the deployment process of a Frappe application by automating the following tasks:

  1. Checking out the latest code.
  2. Installing necessary tools.
  3. Connecting to an EC2 instance using SSH.
  4. Running deployment commands on the EC2 instance.

GitHub Actions CI/CD Pipeline Configuration

name: CI/CD Pipeline

on:
  push:
    branches:
      - development

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install sshpass
        run: sudo apt-get update && sudo apt-get install -y sshpass

      - name: Deploy to EC2
        run: |
            sshpass -p "${{ secrets.EC2_PASSWORD_DEV }}" ssh -o StrictHostKeyChecking=no -p "${{ secrets.EC2_PORT_DEV }}" "${{ secrets.EC2_USERNAME_DEV }}"@"${{ secrets.EC2_IP_DEV }}" << 'EOF'
            echo "Executing remote commands..."
            cd "${{ secrets.PROJECT_LOCATION_DEV }}"
            bash shell/deploy.sh -s "${{ secrets.SITE_NAME_DEV }}"
            echo "Remote commands executed."
            EOF

Shell Script / deploy.sh for Deployment

#!/bin/bash

# Default values
SITE_NAME=""

# Parse command-line arguments
while getopts "s:" opt; do
  case ${opt} in
    s )
      SITE_NAME=$OPTARG
      ;;
    \? )
      echo "Invalid option: -$OPTARG" 1>&2
      exit 1
      ;;
    : )
      echo "Invalid option: -$OPTARG requires an argument" 1>&2
      exit 1
      ;;
  esac
done
shift $((OPTIND -1))

# Check if SITE_NAME is provided
if [ -z "$SITE_NAME" ]; then
    echo "Error: SITE_NAME is required. Please provide it using the -s option."
    exit 1
fi

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm use v18.19.1
export PATH="$PATH:$HOME/.yarn/bin"

# Step 1: Enable VirtualENV
echo "Enable VirtualENV"
source env/bin/activate
if [ $? -ne 0 ]; then
    echo "VirtualENV Enable failed. Exiting."
    exit 1
fi
echo "VirtualENV Enabled"

# Step 2: Run Git Pull Request
echo "Running Git Pull Request"
git stash
git pull
if [ $? -ne 0 ]; then
    echo "Run Git Pull Request failed. Exiting."
    exit 1
fi
echo "Run Git Pull Request Successfully"

# Step 3: Run requirements dev setup
echo "Setting up Requirements --DEV"
bench setup requirements --dev
if [ $? -ne 0 ]; then
    echo "Requirements --DEV Setup failed. Exiting."
    exit 1
fi
echo "Requirements --DEV Setup completed successfully."

# Step 4: Run requirements python setup
echo "Setting up Requirements --PYTHON"
bench setup requirements --python
if [ $? -ne 0 ]; then
    echo "Requirements --PYTHON Setup failed. Exiting."
    exit 1
fi
echo "Requirements --PYTHON Setup completed successfully."

# Step 5: Run requirements node setup
echo "Setting up Requirements --NODE"
bench setup requirements --node
if [ $? -ne 0 ]; then
    echo "Requirements --NODE Setup failed. Exiting."
    exit 1
fi
echo "Requirements --NODE Setup completed successfully."

# Step 6: Run Customization Install
echo "Running Customization Install"
bench --site $SITE_NAME --force partial-restore $SITE_NAME/private/backups/database_backup.sql.gz
if [ $? -ne 0 ]; then
    echo "Customization Install Setup failed. Exiting."
    exit 1
fi
echo "Customization Install Setup completed successfully."

# Step 7: Run DB Migration
echo "Running DB Migration"
bench --site $SITE_NAME restore-schema
if [ $? -ne 0 ]; then
    echo "DB Migration Setup failed. Exiting."
    exit 1
fi
echo "DB Migration Setup completed successfully."

# Step 8: Run Bench Build
echo "Running Bench Build"
bench build
if [ $? -ne 0 ]; then
    echo "Bench Build failed. Exiting."
    exit 1
fi
echo "Bench Build completed successfully."