Troubleshooting HP-UX

Troubleshooting/Maintenance of HP-UX

Where to get information about problems:

dmesg --> provides a finite list of diagnostic messages
/var/adm/syslog/syslog.log --> system log
/opt/resmon/log/error.log -->
/etc/shutdownlog --> shutdown information
/etc/rc.log --> system startup log
/var/tombstones/ts99 --> crash analysis file
cstm - command line support tool manager
mstm - menu based support tool manager

<alt><underlined letter of command>
<tab> --> to move to another portion of the screen, such as the drop down menu area

Service Processor

<ctrl> <b> from a serial console
he - help
co - return to console mode (exits the program)
sl - show log

Panic Reboots

Check these files for clues:

/var/tombstones/ts99
/etc/shutdownlog

Bad disk

1. Check the syslog (/var/adm/syslog/syslog.log) looking for disk errors.
2. Check the ioscan (ioscan -fnC disk), looking for NO_HW rather than Claimed.
3. If diaglogd is running then check STM logs (/var/opt/resmon/log/event.log)
4. Check the volume group to see if the disk is listed and whether there is any problem with it's status (vgdisplay -v | more)
5. Check lvmtab to see if the disk is supposed to be in a volume group (strings /etc/lvmtab | more)

Filesystem do not mount after a reboot

1. Reactivate the Volume Group --> vgchange -a y /dev/<volume group>
2. Remount the filesystems --> mount -a
3. If still no success then perform a filesystem check --> fsck /dev/<volumegroup>/<logicalvolume>
4. Remount the filesystems --> mount -a
5. Check to see if all the filesystems are there:
          a) bdf
          b) compare with /etc/fstab

Filesystem full

du -kx / | sort -rn | more
du -akx | sort -nr | more

Shows directories on the local filesystem and how much space they are taking up

NFS mount - Permission Denied

1. Check to see if the format of the /etc/exports file is correct on the server that is the nfs server.

2. exportfs -av to export the filesystem

3. Check the /etc/fstab file on the client to make sure that it is correct

4. /usr/sbin/showmount -e <server> on the client to show what is being exported

5. To bypass the /etc/exports file execute the following on the nfs server:     exportfs -i -o rw <filesystem>.

NFS Server

/etc/rc.config.d/nfsconf --> NFS_SERVER=1

Verify the proper processes are running:

/sbin/init.d/nfs.server stop

The processes should NOT be running:

# ps -ef|grep nfsd
# ps -ef|grep rpc.mountd
# ps -ef|grep rpc.lockd
# ps -ef|grep rpc.statd

/sbin/init.d/nfs.server start

These processes should be running:

# ps -ef|grep nfsd
    root 3444     1 0 10:39:12 ?         0:00 /usr/sbin/nfsd 4
    root 3451 3444 0 10:39:12 ?         0:00 /usr/sbin/nfsd 4
    root 3449 3444 0 10:39:12 ?         0:00 /usr/sbin/nfsd 4
    root 3445 3444 0 10:39:12 ?         0:00 /usr/sbin/nfsd 4
# ps -ef|grep rpc.mountd
    root 3485     1 0 10:42:09 ?         0:00 rpc.mountd
# ps -ef|grep rpc.lockd
    root 3459     1 0 10:39:12 ?         0:00 /usr/sbin/rpc.lockd
# ps -ef|grep rpc.statd
    root 3453     1 0 10:39:12 ?         0:00 /usr/sbin/rpc.statd

To start a process if it is not running:

# ps -ef|grep rpc.mountd
# rpc.mountd   or /usr/sbin/rpc.mountd
# ps -ef|grep rpc.mountd
    root 3485     1 0 10:42:09 ?         0:00 rpc.mountd

/etc/inetd.conf needs to have the proper services active (not commented out)

##
# WARNING: The rpc.mountd should now be started from a startup script.
#          Please enable the mountd startup script to start rpc.mountd.
##
#rpc stream tcp nowait root /usr/sbin/rpc.rexd     100017 1    rpc.rexd
rpc dgram udp wait    root /usr/lib/netsvc/rstat/rpc.rstatd   100001 2-4 rpc.rstatd
rpc dgram udp wait    root /usr/lib/netsvc/rusers/rpc.rusersd 100002 1-2 rpc.rusersd
rpc dgram udp wait    root /usr/etc/rpc.mountd 100005 1 rpc.mountd -e
rpc dgram udp wait    root /usr/lib/netsvc/rwall/rpc.rwalld   100008 1    rpc.rwalld
#rpc dgram udp wait    root /usr/sbin/rpc.rquotad 100011 1    rpc.rquotad
rpc dgram udp wait    root /usr/lib/netsvc/spray/rpc.sprayd   100012 1    rpc.sprayd

NIC problems:

The lanadmin utility provides NIC statistics
The nettladmin utility provides packet trace information

Replacing a Mirrored Root Disk:

Replace the disk

Hot swap can be performed while system is up
Not hot swappable means the system must be brought down

Reboot the system into single user mode

shutdown -r 0, unless the system is powered off already, then power it back on
interrupt the boot
bo pri (or bo alt if the disk that was replaced was the primary boot disk)
IPL>hpux -is -lq (;0)/stand/vmunix

vgcfgrestore -n /dev/vg00 /dev/rdsk/c?t?d?
vgsync /dev/vg00
mkboot /dev/rdsk/c?t?d?
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c?t?d?
shutdown -r 0
lvlnboot -v /dev/vg00 to verify that the disk is seen as bootable

Software Installation (swinstall, sd, etc)

    ERROR:   "server::/tmp/omni_tmp/packet": You do not have the
         required permissions to perform this operation. Check
         permissions using the "swacl" command or see your system
         administrator for assistance. Or, to manage applications
         designed and packaged for nonprivileged mode, see the
         "run_as_superuser" option in the "sd" man page.
WARNING: More information may be found in the daemon logfile on this
         target (default location is
         server:/var/adm/sw/swagentd.log).

Bounce swagentd daemon: /usr/sbin/swagentd -r