First, did you know that Solaris' UFS is a journaling filesystem? Make crashes more manageable by putting 'logging' in the options component of /etc/vfstab. Second did you know you can freeze your filesystems to cut your backup down time to a few seconds? Read...

A classic sysadmin dilemna is that you have to stop writing files if you want to get a clean backup. This usually means stopping the service that uses the files for the duration of a backup. But that could be a *very* long time in which you service is down. In high availability environments, this is unacceptable. Many applications (oracle) provide mechanisms to support archiving without cold backups. But most don't. Besides a cold backup is usually the cleanest.

Solaris UFS provides a compromise between hot & cold backups. It's called fssnap which stands for file system snapshot. Fssnap take a snapshot of a filesystem at a specific point in time and creates devfs psuedo devices for accessing the snapshot. Then changes to the original filesystem are written to a backing store. So by comparing the backing store and the original filesystem fssnap can produce the filesystem *exactly* as it was when you snapshotted it.

The *beauty* of this model is that you can stop your services for just a few seconds, take a snapshot, restore services, and then take your sweet time backing up.

The are of course disadvantages:

1. You *should* stop services long enough to take a snapshot. You *can* keep services running, but the snap might not succeed if locks aren't released before a timout or you might get half written files. Keeping the app running presents the same problems as a hot backup, but is usually good enough to restore single files (just not whole installations).
2. While the snapshot device is available all disk writes to the original filesystem must also be written to the backing store. So double write time.
3. Overhead. Now the backup, fssnap, and your application are contending for disk I/O, memory and CPU. Usually not a problem though. A well chosen backup time will alleviate any difficulties.

Here's a script I use on solaris to take nightly backups of an apache/modperl/tomcat server.

#!/bin/bash

# This is the backup routine for the blackboard application server. It uses
# fssnap to take a snapshot of the current state of the filesystems.
# Only really good for Solaris. I haven't used the snapshotting in the linux md
# kernel stuff (but md rox!). If you know how to use the linux one that would be an easy change.
# This script would take modifications if you're going to backup to tape instead of
# a dmp file.
#
# 6/28/03 5:06 PM Jim Weller


# SOME GUIDANCE
#
# see http://docs.sun.com/db/doc/806-7502/6jgce01t7?a=view
# 
# This is a little longer and more generic version Anthony G. Gunia's (from
# Purdue) script. I'm not real fond of ufsdump because you can't throw it on
# a cd and look at it on a desktop pc/mac w/out trauma. But I put it in their
# to be true to the original. I also don't have a tape drive in my app
# servers. I deposit my archives on an NFS mounted NAS. Then little bots back
# it up to external media.
# 
# You *SHOULD* have a backing store that is as big as the sum of all your
# drives. That's the true upper bound.  But you *SHOULD* be able to get by
# with a backing store just a little larger than your largest mount to
# backup.
# 
# Also your backup location needs to be large enough to hold all the
# tar/ufsdump files that you are creating.
# 
# This is best done at a low activity time because writes to the drives being
# backed up are expensive because both the backing store has to be updated,
# and the actual file write has to be performed. This shouldn't hurt most
# system performance, but a 1am backup isn't a bad idea.
# 
# To shutdown blackboard or not? I say yes because you're doing it at some
# bewitched hour. But hypothetically there could be somebody using it during
# the 10-30 seconds that it will be down. And yes, their session will get
# screwed, and their next click will return either no server or server error.
# Either way, it's only 30 seconds. BUT, if you don't shut it down then
# fssnap will take slightly longer waiting for lock releases AND you might
# have bb data in a transient state. Those are the choices. It's up to you.


#  NOTE: the devfsadm daemon must be running for the device special nodes to
#  be created. I tend to strip servers, and pulled the daemon thinking I'd
#  not use it. Now I use it.

# the place where you want your backups put, tape or folder or device
export BACKUP_LOCATION=/mnt/backups

# the location where you want the fssnaps backing store to live it is a
# temporary file(s) used to maintain differences between the original
# filesystem and it's snapshot. It cannot be the same mount that is being
# backed up (e.g not in LIST_OF_MOUNT_POINTS. I recommend a seperate mount
# (preferably on an otherwise unused disk). The backing store must be
# slightly larger than the the mount that is being archived. See man
# fssnap_ufs for details
export FSSNAP_BACKING_STORE_LOCATION=/mnt/fssnap-backing-store

# A space seperated list of devices/mountPoints that you plan on archiving.
# Try not to include the trailing slash (e.g. /home/). I've seen solaris mount
# tools screw that up.

# this will do *ALL* ufs mounts ( careful not to backup your backing store ;)
# useful for backing up to a NON-ufs filesystem (like tape or nfs)
#export LIST_OF_MOUNT_POINTS=`df -F ufs | awk  -F\( '{print $1;}'`

# or do it by hand here
#export LIST_OF_MOUNT_POINTS="/ /usr /export/home /var /usr/local/blackboard"
export LIST_OF_MOUNT_POINTS="/ /usr/local/blackboard"

# or debug it ;)
#export LIST_OF_MOUNT_POINTS="/tmp/crap"

# Some OS's use different date format 
# specifiers. This works on Sol8 (ex 20030623.1334 )
export DATE_PREFIX_COMMAND="date +%Y%m%d.%H%M"


# if you needed to umount a tape or disk you'd put the command here
# mount $BACKUP_LOCATION


# maybe you want to stop blackboard first?
#  my init script kills java procs to
/etc/init.d/blackboard stop  >/dev/null
wait

# loop though the list of things to be backed up
#  - take a snapshot
for i in $LIST_OF_MOUNT_POINTS
do
    # create the snapshots
    fssnap -F ufs -o backing-store=$FSSNAP_BACKING_STORE_LOCATION $i > /dev/null
    wait
done

# it's ok to restart bb now. we have a snapshot.
/etc/init.d/blackboard start  >/dev/null
wait

# loop though the list of things to be backed up
#  - build some useful strings
#  - ufs dump the snapshot (or tar so files readable on PC's?)
#  - delete this snapshot
#  - delete this backing store
for i in $LIST_OF_MOUNT_POINTS
do

    # last arg in mount point, probably unique usr var export blackboard etc etc.
    export SUFFIX=`basename $i`

    # we don't want wierd filenames for the root dir so replace / with the word root
    export SUFFIX=`echo $SUFFIX | sed -e 's/\//root/' `

    # date stamp now
    export DATE_PREFIX=`$DATE_PREFIX_COMMAND`

    # maybe you like ufsdump?
    # create the dump file
    #export FILE_NAME=$DATE_PREFIX-$SUFFIX.ufsdump
    #/usr/sbin/ufsdump 0uf $BACKUP_LOCATION/$FILE_NAME /dev/rfssnap/`fssnap -i -o snapnumber $i`
    #wait
    
    # or maybe you like tar better? (don't use sun tar!)
    export FILE_NAME=$DATE_PREFIX-$SUFFIX.tar
    mkdir /tmp/$DATE_PREFIX
    mount -F ufs -o ro `fssnap -i -o blockdevname $i` /tmp/$DATE_PREFIX
    pushd /tmp/$DATE_PREFIX > /dev/null
    /usr/local/bin/tar -cf $BACKUP_LOCATION/$FILE_NAME .
    popd > /dev/null
    wait
    umount /tmp/$DATE_PREFIX
    wait
    rmdir  /tmp/$DATE_PREFIX
    
    export BACKING_FILE=`fssnap -i -o backing-store $i`

    # delete the snapshot
    fssnap -d $i  > /dev/null
    wait

    # clean up after this snapshot
    rm -f $BACKING_FILE
    
done


# if you needed to unmount a tape or disk or share you'd put it here
# umount $BACKUP_LOCATION

exit