July 2003 Archives

Marquez's "Love in the Time of Cholera" is one of those overley poignant heady literature novels you'd have to read for a class. What can I say I'm a big hairy brain pervert and have been on heavy lit kick.

This is the story of Florentino Ariza, a bastard son of a river trader, whose romantic quixoticism toward shady mule traders Daughter Fermina Daza touches and mortifies. She marries into a more notable family and enjoys a happy life. Florentino maintains a distant love worship and holds a "love grudge" for 50+ years.

The book is absolutely touching. It analyzes deep questions about love, age, and happiness. The writing is so rich. I found myself stopping at moments to savor a phrase that tweaked a sequence. Realize that there are almost no breaks in continuity. There are no chapters or breaks. Everything flows as if in a poetic dream.

The book culminates with the wizened old/young lovers on a honeymoon trip aboard a beatiful river boat on a ravaged river. The damn metaphors are so deep I don't even want to get into it. But it was good. I dug it.

I read "100 Years of Solitude" in a daze, overwhelmed by names, culture and content. This book was easier to approach. I really "bonded" with it's humanity.

First, did you know that Solaris' UFS is a journaling filesystem? Make crashes more manageable by putting 'logging' in the options component of /etc/vfstab. Second did you know you can freeze your filesystems to cut your backup down time to a few seconds? Read...

A classic sysadmin dilemna is that you have to stop writing files if you want to get a clean backup. This usually means stopping the service that uses the files for the duration of a backup. But that could be a *very* long time in which you service is down. In high availability environments, this is unacceptable. Many applications (oracle) provide mechanisms to support archiving without cold backups. But most don't. Besides a cold backup is usually the cleanest.

Solaris UFS provides a compromise between hot & cold backups. It's called fssnap which stands for file system snapshot. Fssnap take a snapshot of a filesystem at a specific point in time and creates devfs psuedo devices for accessing the snapshot. Then changes to the original filesystem are written to a backing store. So by comparing the backing store and the original filesystem fssnap can produce the filesystem *exactly* as it was when you snapshotted it.

The *beauty* of this model is that you can stop your services for just a few seconds, take a snapshot, restore services, and then take your sweet time backing up.

The are of course disadvantages:

1. You *should* stop services long enough to take a snapshot. You *can* keep services running, but the snap might not succeed if locks aren't released before a timout or you might get half written files. Keeping the app running presents the same problems as a hot backup, but is usually good enough to restore single files (just not whole installations).
2. While the snapshot device is available all disk writes to the original filesystem must also be written to the backing store. So double write time.
3. Overhead. Now the backup, fssnap, and your application are contending for disk I/O, memory and CPU. Usually not a problem though. A well chosen backup time will alleviate any difficulties.

Here's a script I use on solaris to take nightly backups of an apache/modperl/tomcat server.

#!/bin/bash

# This is the backup routine for the blackboard application server. It uses
# fssnap to take a snapshot of the current state of the filesystems.
# Only really good for Solaris. I haven't used the snapshotting in the linux md
# kernel stuff (but md rox!). If you know how to use the linux one that would be an easy change.
# This script would take modifications if you're going to backup to tape instead of
# a dmp file.
#
# 6/28/03 5:06 PM Jim Weller


# SOME GUIDANCE
#
# see http://docs.sun.com/db/doc/806-7502/6jgce01t7?a=view
# 
# This is a little longer and more generic version Anthony G. Gunia's (from
# Purdue) script. I'm not real fond of ufsdump because you can't throw it on
# a cd and look at it on a desktop pc/mac w/out trauma. But I put it in their
# to be true to the original. I also don't have a tape drive in my app
# servers. I deposit my archives on an NFS mounted NAS. Then little bots back
# it up to external media.
# 
# You *SHOULD* have a backing store that is as big as the sum of all your
# drives. That's the true upper bound.  But you *SHOULD* be able to get by
# with a backing store just a little larger than your largest mount to
# backup.
# 
# Also your backup location needs to be large enough to hold all the
# tar/ufsdump files that you are creating.
# 
# This is best done at a low activity time because writes to the drives being
# backed up are expensive because both the backing store has to be updated,
# and the actual file write has to be performed. This shouldn't hurt most
# system performance, but a 1am backup isn't a bad idea.
# 
# To shutdown blackboard or not? I say yes because you're doing it at some
# bewitched hour. But hypothetically there could be somebody using it during
# the 10-30 seconds that it will be down. And yes, their session will get
# screwed, and their next click will return either no server or server error.
# Either way, it's only 30 seconds. BUT, if you don't shut it down then
# fssnap will take slightly longer waiting for lock releases AND you might
# have bb data in a transient state. Those are the choices. It's up to you.


#  NOTE: the devfsadm daemon must be running for the device special nodes to
#  be created. I tend to strip servers, and pulled the daemon thinking I'd
#  not use it. Now I use it.

# the place where you want your backups put, tape or folder or device
export BACKUP_LOCATION=/mnt/backups

# the location where you want the fssnaps backing store to live it is a
# temporary file(s) used to maintain differences between the original
# filesystem and it's snapshot. It cannot be the same mount that is being
# backed up (e.g not in LIST_OF_MOUNT_POINTS. I recommend a seperate mount
# (preferably on an otherwise unused disk). The backing store must be
# slightly larger than the the mount that is being archived. See man
# fssnap_ufs for details
export FSSNAP_BACKING_STORE_LOCATION=/mnt/fssnap-backing-store

# A space seperated list of devices/mountPoints that you plan on archiving.
# Try not to include the trailing slash (e.g. /home/). I've seen solaris mount
# tools screw that up.

# this will do *ALL* ufs mounts ( careful not to backup your backing store ;)
# useful for backing up to a NON-ufs filesystem (like tape or nfs)
#export LIST_OF_MOUNT_POINTS=`df -F ufs | awk  -F\( '{print $1;}'`

# or do it by hand here
#export LIST_OF_MOUNT_POINTS="/ /usr /export/home /var /usr/local/blackboard"
export LIST_OF_MOUNT_POINTS="/ /usr/local/blackboard"

# or debug it ;)
#export LIST_OF_MOUNT_POINTS="/tmp/crap"

# Some OS's use different date format 
# specifiers. This works on Sol8 (ex 20030623.1334 )
export DATE_PREFIX_COMMAND="date +%Y%m%d.%H%M"


# if you needed to umount a tape or disk you'd put the command here
# mount $BACKUP_LOCATION


# maybe you want to stop blackboard first?
#  my init script kills java procs to
/etc/init.d/blackboard stop  >/dev/null
wait

# loop though the list of things to be backed up
#  - take a snapshot
for i in $LIST_OF_MOUNT_POINTS
do
    # create the snapshots
    fssnap -F ufs -o backing-store=$FSSNAP_BACKING_STORE_LOCATION $i > /dev/null
    wait
done

# it's ok to restart bb now. we have a snapshot.
/etc/init.d/blackboard start  >/dev/null
wait

# loop though the list of things to be backed up
#  - build some useful strings
#  - ufs dump the snapshot (or tar so files readable on PC's?)
#  - delete this snapshot
#  - delete this backing store
for i in $LIST_OF_MOUNT_POINTS
do

    # last arg in mount point, probably unique usr var export blackboard etc etc.
    export SUFFIX=`basename $i`

    # we don't want wierd filenames for the root dir so replace / with the word root
    export SUFFIX=`echo $SUFFIX | sed -e 's/\//root/' `

    # date stamp now
    export DATE_PREFIX=`$DATE_PREFIX_COMMAND`

    # maybe you like ufsdump?
    # create the dump file
    #export FILE_NAME=$DATE_PREFIX-$SUFFIX.ufsdump
    #/usr/sbin/ufsdump 0uf $BACKUP_LOCATION/$FILE_NAME /dev/rfssnap/`fssnap -i -o snapnumber $i`
    #wait
    
    # or maybe you like tar better? (don't use sun tar!)
    export FILE_NAME=$DATE_PREFIX-$SUFFIX.tar
    mkdir /tmp/$DATE_PREFIX
    mount -F ufs -o ro `fssnap -i -o blockdevname $i` /tmp/$DATE_PREFIX
    pushd /tmp/$DATE_PREFIX > /dev/null
    /usr/local/bin/tar -cf $BACKUP_LOCATION/$FILE_NAME .
    popd > /dev/null
    wait
    umount /tmp/$DATE_PREFIX
    wait
    rmdir  /tmp/$DATE_PREFIX
    
    export BACKING_FILE=`fssnap -i -o backing-store $i`

    # delete the snapshot
    fssnap -d $i  > /dev/null
    wait

    # clean up after this snapshot
    rm -f $BACKING_FILE
    
done


# if you needed to unmount a tape or disk or share you'd put it here
# umount $BACKUP_LOCATION

exit

It's *pure* voodoo. You make oracle's "normal" snmp agent, called 'master_peer', act as the master agent for the system. Then , oracle's 'encap_peer' will look for an snmp "subagent" on port UDP:1161. That's where you put ucd-snmp or net-snmp. Oracle will essentially proxy all requests that aren't it's own to the other program. This let's you get to the normal SNMP, HOST, et al MIBs. Read on...

The purpose of this document is to walk through installation, configuration, and examination of oracle's smart agent's SNMP capabilities. This document is specific to Unix, but directions exist for windows. Oracle provides it's own SNMP master agent. You'll use an oracle wrapper to net/ucd-snmp. I tried to do it the other way. I tried to make net-snmp the master and *just* use dbsnmp's port 1748 (snmp port), but it didn't work. So this is the way for now.

First connect to the instance(s) in sqlplus as sysdba and run 'catsnmp.sql'. This script creates an SNMPAGENT role in the database that has privs on V$'s to harvest stats. It also creates a user called DBSNMP that is assigned the SNMPAGENT role. There is also an undo script called 'catnsnmp.sql'.

@?/rdbms/admin/catsnmp.sql

The mib's end in .v1 and are in $ORACLE_HOME/network/doc read the top couple of lines to see which MIB is which. The oracleDDB and rdbms were most important for my needs. You'll probably want to copy those to your MIB directory (/usr/local/share/snmp/mibs ymmv). One of the MIB's is broken for net-snmp. Edit onrs.v1

        Counter
                FROM RFC1155-SMI

becomes
 
        Counter, enterprises
                FROM RFC1155-SMI

Now we need to make some small changes in oracle's configuration. Look in $ORACLE_HOME/network/snmp/peer. You'll see

• snmp.conf - crap, they expect you to feed this to your SNMP daemon. We don't.
• master_peer - the "master" agent. It's an SNMP server.
• CONFIG.master - config for master server (oracle's snmp server)
• encap_peer - the encapsulation server shim between master_peer and snmpd. It passes requests to native snmp on 1161
• CONFIG.encap - config for the encap server
• start_peer - a silly wrapper script to start master, encap, and then your native snmp server.

You can probably leave CONFIG.master except maybe to change the IP of your SNMP trap server. I don't have one so I just setup a readonly community that only accepts from localhost.

COMMUNITY public
ALLOW GET OPERATIONS
USE NO ENCRYPTION
MEMBERS localhost

Edit CONFIG.encap and add any extra oids. Add any other OID's you want encapsulated and passed to your native snmp server. The extra OID's shown are host, internet and rdbms.

AGENT AT PORT 1161 WITH COMMUNITY public 
SUBTREES        1.3.6.1.2.1.1, 
                1.3.6.1.2.1.2, 
                1.3.6.1.2.1.3, 
                1.3.6.1.2.1.4, 
                1.3.6.1.2.1.5, 
                1.3.6.1.2.1.6, 
                1.3.6.1.2.1.7, 
                1.3.6.1.2.1.8, 
                1.3.6.1.2.1.25, 
                1.3.6.1.2.1.39, 
                1.3.6.1.4.1.2021, 
                1.3.6.1.4.1.11.2 
FORWARD ALL TRAPS; 

Now to fix oracle's start_peer script. This script starts master, encap, and snmpd. Edit start_peer (another lame oracle script). Make sure SNMPD= points to your snmpd daemon (/usr/local/sbin/snmpd ymmv). You should point SNMPD_CONF= to a vlid snmpd.conf file, but I've found you can do it without. But you get whatever the defaults are. Snmpd config files are outside of scope, but see your snmpd.conf man page to write a suitable one. Then fix the line at the end that calls ucd/net snmp. Fix it to use (or not use) the right config file. Note the difference in how the two programs get listening addresses.

        $SNMPD -c $SNMPD_CONFIG -p $NEW_SNMPD_PORT >snmpd.out 2>&1 &
to (ucd snmp)
        $SNMPD -C -c $SNMPD_CONFIG -p $NEW_SNMPD_PORT  >snmpd.out 2>&1 &
or to (net-snmp)
        $SNMPD -C -c $SNMPD_CONFIG $HOSTNAME:$NEW_SNMPD_PORT  >snmpd.out 2>&1 &

Make sure your native snmpd isn't running. Now you can start daemons with './start_peer -a'. You'll get output like this:

Starting master_peer ...
./master_peer CONFIG.master NOV >master_peer.out 2>&1 &
Done!

Starting encap_peer ...
./encap_peer -t 1162 -s 1160 -c CONFIG.encap >encap_peer.out
2>&1 &
Done!

Starting /usr/sbin/snmpd ...
/usr/sbin/snmpd -C -c /usr/local/shar/snmp/snmpd.conf -p 1161 >snmpd.out 2>&1 &
Done!

Now you can start oracle's dbsnmp processes. You use lsnrctl to manage this. Start the dbsnmp service by typing 'lsnrctl dbsnmp_start' as the oracle user. There are two dbsnmp processes that start. One listens on TCP:1748 as the SNMP agent (only comms with iAgent). The other is the intelligent agents RPC handler on port TCP:1754 (for oem mainly). I'd like to turn the second one off, but haven't figured it out yet.

Finally, you want to wrap this all up into a nice startup script. Remember that iAgent is now starting your snmpd daemon for you. So, you'll want to remove snmpd from you rc scripts. Oracle, doesn't have an SNMP trap daemon running. So, if this is a trap station keep snmptrapd in init. Here's my init script for snmp (oracle_intelligent_agent). It's not very elegant in the clean up, but oracle doesn't give you much to work with. Make sure this script starts after oracle, and is stopped before oracle is stopped. You have to hax sun init scripts to get the right environment. The one below is for linux. If you need sun help contact me.

#!/bin/bash

# Simple script to start oracle's SNMP subsystem
# which also manages the native snmpd.
#
# This script gets run with /bin/sh on solaris
# so watch your syntax bash-boy
#

ORACLE_HOME=/oracle/product/8.1.7
ORACLE_OWNER=oracle
export ORACLE_HOME
export ORACLE_OWNER

case "$1" in
 'start')
    echo "Starting Oracle Intelligent Agent"
    cd $ORACLE_HOME/network/snmp/peer/
    ./start_peer -a
    sleep 2
    su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/lsnrctl dbsnmp_start"
    ;;
 'stop')  # Stop the Oracle databases and Net8 listener
    echo "Stopping oracle intelligent agent"
    su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/lsnrctl dbsnmp_stop"
    pkill master_peer 
    pkill encap_peer
    pkill snmpd
    sleep 2
    if pgrep "master_peer|encap_peer|snmpd"
    then
      pkill -KILL master_peer 
      pkill -KILL encap_peer
      pkill -KILL snmpd
    fi
    ;;
 *)
    echo "Usage: $0 start|stop"
    ;;
esac

Now check that everything is working. Can you snmpwalk the system? Solaris with net-snmp I used 'snmpwalk -Of -m all -c public -v 1 localhost .' Which basically says use all MIB's to give FQ descriptions of all OID's on public@localhost using SNMPv1 which is oracle's only version I can find. Using ucd-snmp on linux I used 'snmpwalk -Of -m all localhost public .' You should get a HUGE list that includes HOST and many of the oracle OIDs.

[...]
application.applTable.applEntry.applIndex.5 = INTEGER: 5
application.applTable.applEntry.applName.2 = STRING: "bb6"
[...]
rdbmsMIB.rdbmsObjects.rdbmsDbTable.rdbmsDbEntry.rdbmsDbIndex.2 = INTEGER: 2
rdbmsMIB.rdbmsObjects.rdbmsDbTable.rdbmsDbEntry.rdbmsDbIndex.5 = INTEGER: 5
[...]

Cool huh? You're basically back where you started with a working SNMP agent, but with oracle doing proxxy magic to net/ucd snmp. You should shore up the security by tailoring the config files and your firewall to your needs. Now you just need to figure out which datums are important to you and throw them at MRTG or HPOV or whatever.

Problems? The 'peers' have some kind of lock on a socket or file that takes 5 minutes to release. So you either have to wait or reboot to start master_peer or encap_peer. These are files to check for messages:

• $ORACLE_HOME/network/logs/dbsnmpw.log
• $ORACLE_HOME/network/logs/dbsnmpw.log
• $ORACLE_HOME/network/snmp/peer/master_peer.out
• $ORACLE_HOME/network/snmp/peer/encap_peer.out
• $ORACLE_HOME/network/snmp/peer/snmpd.out

Some references I used
http://download-west.oracle.com/docs/cd/A87860_01/doc/em.817/a85249/toc.htm
Google groups discussion
http://sourceforge.net/docman/display_doc.php?docid=5201&group_id=12694

Moving on the oracle thread I'm working on stats