Rsync Project - Linux Bash

Submitted by brett.ferringer on Tue, 09/14/2021 - 20:17

This next code is a work in progress. This is to backup my Unraid share to my OpenMediaVault server that is hosted virtually through Unraid. To ensure that it is a good backup, I passed the hard drives through Unraid and into the OpenMediaVault server. I have the rsync Linux virtual machine set up so that everyday at 1am and 1pm, it will back up the source location to the destination. In my design of this, I wanted good logging for when there is an issue with transferring a file. This code will log any issues within the mysql server. An example of this is below:

Image removed.

After the script below is completed, it will email me to let me know that it is completed. Eventually I would like to also include any errors that arise during this

rsync.sh file:

 

#!/bin/bash

dun='<database username>'

dpw='<database password>'

dip='<server ip or hostname>'

dn='logs'

dt=$(date +%s)

source='/mnt/source/'

destination='/mnt/destination/'

mediaFolder='<folder name>'

personalFolder='<folder name>'

 

function FileCheck()

{

    if [[ ! -d "$1" ]];

        then

            echo "The $1 file does not exist. The $1$2 file will mount now."

            mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, end, ERROR, server) VALUES ('rsync - $1$2 folder', 'FAIL', now(), now(), '$1 was not mounted. $1$2 mounted.', 'rsyncServer')"

            sudo mount $1$2

    else

        echo "The $1$2 file exists. Moving on."

        mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, end, server) VALUES ('rsync - $1$2 folder', 'PASS', now(), now(), 'rsyncServer')"

    fi

}

 

function RsyncProcess()

{

    ERR_OUTPUT="/tmp/rsync_err_$$"

   

    mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, tmp, server) VALUES ('rsync - $4', 'start', now(), 'rsync$dt', 'rsyncServer')"

    rsync -$5 --exclude=.* $1$3/$4/* $2$3/$4/ 2> "$ERR_OUTPUT"

    ERR=$(sed "s/'/_/g"  $ERR_OUTPUT)

 

    if [ $? -eq 0 ]

        then

            mysql --host=$dip --user=$dun --password=$dpw $dn -e "UPDATE processes SET end = now(), tmp = '', status = 'finished', ERROR = '$ERR' WHERE tmp = 'rsync$dt'"

    else

        mysql --host=$dip --user=$dun --password=$dpw $dn -e "UPDATE processes SET end = now(), tmp = '', status = 'FAIL', ERROR = '$ERR' WHERE tmp = 'rsync$dt'"

    fi

 

    sleep 1

    rm -f $ERR_OUTPUT

}

 

function RsyncPerm()

{

    chmod -R 777 $1

    if [ $? -eq 0 ]; 

        then

            echo "Permissions were changed successfully."

            mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, end, server) VALUES ('Permissions - $1', 'finished', now(), now(), 'rsyncServer')"

    else

        mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, end, server) VALUES ('Permissions - $1', 'FAIL', now(), now(), 'rsyncServer')"

    fi

}

 

echo "Starting rsync incremental backup."

FileCheck $source $mediaFolder

FileCheck $destination $mediaFolder

 

sleep 1

 

mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, tmp, server) VALUES ('rsync', 'start', now(), 'srsync$dt', 'rsyncServer')"

 

for p_name in $source$mediaFolder/*

    do

        p_name=${p_name##*/}

        case "$p_name" in

        *.*) echo 'Not a folder. Skipping';;

        *       ) RsyncProcess $source $destination $mediaFolder $p_name 'a';;

    esac

done

 

RsyncProcess $source $destination '' $personalFolder 'a'

RsyncProcess '/mnt/' '/mnt/destination/' '' 'System' 'avz'

RsyncPerm $destination

 

mysql --host=$dip --user=$dun --password=$dpw $dn -e "UPDATE processes SET end = now(), tmp = '', status = 'finished' WHERE tmp = 'srsync$dt'"

 

edt=$(date +'%m/%d/%Y')

curl --url 'smtps://smtp.gmail.com:465' --ssl-reqd \

  --mail-from '<mail from email address>' \

  --mail-rcpt '<The receiver email address' \

  --user '<smtps email address to use>:<smtps email password to use>' \

  -T <(echo -e "From: <from email address>\nTo: <to email address>\nSubject: Rysnc Complete - $edt\n\nThe Rsync process completed at $edt")

Auto Update Script for Linux

Submitted by brett.ferringer on Tue, 09/14/2021 - 20:16

 This script is meant to automate my Linux server update process. It logs the results to my Maria DB database.

 

#!/bin/bash

dun='<username>'

dpw='<password>'

dip='<ipaddress>'

dn='logs'

serverName='<servername>'

mysql --host=$dip --user=$dun --password=$dpw $dn -e "UPDATE processes SET tmp = '' WHERE tmp = 'sSM$serverName'"

mysql --host=$dip --user=$dun --password=$dpw $dn -e "INSERT INTO processes (name, status, start, tmp, server) VALUES ('systemMaintenance', 'start', now(), 'sSM$serverName', '$serverName')"

apt-get update

apt-get -y upgrade

apt-get -y autoremove

apt-get autoclean

mysql --host=$dip --user=$dun --password=$dpw $dn -e "UPDATE processes SET end = now(), tmp = '', status = 'finished' WHERE tmp = 'sSM$serverName'"

SQL Commands Batch File

Submitted by brett.ferringer on Tue, 09/14/2021 - 20:14

 

I wrote the following batch script to automatically run SQL commands and log the results. This particular script was originally meant for automated select queries.

 

 

@echo off

set SName="<ip or server name>"

set DbName="<databaseName>"

set RtnFolder=<output folder>

set mydate=%date:~10,4%-%date:~4,2%-%date:~7,2%

set logName=<log name>

@echo on rem checks to see if the logfile exists

IF NOT EXIST %~dp0\logs\%logName%.log ( ECHO Name,Status,Date,Time>>"%~dp0logs\%logName%.log" )

rem adding line to log that the shows the start of the full process

ECHO %logName%,Start,%DATE%,%TIME%>>"%~dp0logs\%logName%.log"

rem running through the sql commands in the sql folder to run all select queries available rem this logs each start and finish to the log file rem this only picks up .sql files. All other files are ignored

FOR %%i IN ("%~dp0sql\*.sql") DO ECHO %%~ni,Start,%DATE%,%TIME%>>"%~dp0logs\%logName%.log"^

&& sqlcmd -S %SName% -E -d %DbName% -i "%%i" -v PARM1=%mydate% -o "%RtnFolder%%%~ni - %mydate%.csv" -W -w 999 -s"|"^

&& IF ERRORLEVEL 1 (ECHO %%~ni,FAIL,%DATE%,%TIME%>>"%~dp0logs\%logName%.log") ^

ELSE (ECHO %%~ni,Finish,%DATE%,%TIME%>>"%~dp0logs\%logName%.log") rem adding line to log the end of the full process ECHO %logName%,Finish,%DATE%,%TIME%>>"%~dp0logs\%logName%.log"