rsync script for large jobs

Before I run scripts below I first mount both the source and the destination paths if necessary. Then the systax would be

similar to the following. Of course the paths differ. (Important to add the last /.)

/home/…/scripts/rsync.WRAP    ///

This first script, rsync.WRAP runs script rsync.PRJ.

*** Begin script rsync.WRAP ***

#!/bin/sh -x
cwd=`dirname $0`

if [ $# -lt 2 ] ; then
echo “USAGE: rsync.WRAP  ”



[ -f /var/tmp/$SOURCE_PROJECT_NAME ] && rm /var/tmp/$SOURCE_PROJECT_NAME.*
[ -f /var/tmp/$SOURCE_PROJECT_NAME.tmp ] && rm /var/tmp/$SOURCE_PROJECT_NAME.tmp

cd $SOURCE_PROJECT_PATH ; find . -maxdepth 3 -type d \( -name .snapshot \) -prune -o -type d -print | cpio -pvdum $DEST_NFS_SERVER > /var/tmp/$SOURCE_PROJECT_NAME.tmp 2>&1

cat /var/tmp/$SOURCE_PROJECT_NAME.tmp | grep -v “^0 blocks” | sed -e “s,$DEST_NFS_SERVER/,,g” > /var/tmp/$SOURCE_PROJECT_NAME && rm /var/tmp/$SOURCE_PROJECT_NAME.tmp

cd /var/tmp ; split -l 20 $SOURCE_PROJECT_NAME $SOURCE_PROJECT_NAME.

for j in `ls -1 /var/tmp/$SOURCE_PROJECT_NAME.[a-z]*`; do
/home/thierse/files/professional/unix/scripts/rsync.PRJ $j $SOURCE_PROJECT_PATH $SOURCE_PROJECT_NAME $DEST_NFS_SERVER &

# Do a one time top level rsync to the project to get the files at top level
rsync -lptgoDWSH  –stats –exclude $SOURCE_PROJECT_PATH/.snapshot $SOURCE_PROJECT_PATH ${DEST_NFS_SERVER}

*** End Script rsync.WRAP ***

*** Begin Script rsync.PRJ ***


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s