The purpose of this document is to provide a quick reference as well as additional methods for recovery and adittional best practices.
Introduction
Many administrators choose to use an alternative volume manager to mirror the boot drives because of the difficulties recovering from primary disk failure under VxVM. Typically, Solsitce Disksuite(free product) is used to mirror the boot drive while VxVM is used for all other storage. However, if VxVM is installed and configured correctly, the recovery process is greatly simplified.
The problems associated with recovery are associated in part by the way it encapsulates the boot disk(see above articles for details). We must properly install and configure VxVM so that these issues are avoided. In simple terms, we must first mirror the root disk. Second, we will remove the rootdisk from rootdg and reinitialize the drive. Third, we will add the root disk back into rootdg and re-attach the mirror.
Note: You may be interested in determining if a system's boot drives have already been mirrored properly. One method to identify if the rootdisk was properly mirrored is to look for the presence of the "rootdisk-B0" subdisk. This subdisk is used to map out the boot block of the rootdisk. Use the "vxprint -ht rootvol" command to verify if it exists. If it does exist, the boot drive has NOT been mirrored according to these procedures.
Mirror the Drives using VxVM
Identify the root disk and mirror disk. It is a good idea to have the drives on different controllers if possible for improved redundancy.
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133> <=== Root
/pci@1f,4000/scsi@3/sd@0,0
1. c0t1d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133> <=== Root Mirror
/pci@1f,4000/scsi@3/sd@1,0
Specify disk (enter its number): ^D
Install Veritas Volume Manger according to documentation. Only encapsulate the root disk. Do NOT encapsulate any other drives.
NOTE: You may want to setup your PATH environment variable if you have not already done so. If you installed in VxVM in the default location you can use the following:
# PATH=$PATH:/usr/lib/vxvm/bin; export PATH
Initialize the root mirror disk. Add it to "rootdg" group and name the disk "rootmirror".
# vxdisksetup -i c0t1d0
# vxdg -g rootdg adddisk rootmirror=c0t1d0
Mirror the root disk according to the partition layout.
If you have /, swap, /var, /opt, and /home defined as partitions 0,1,3, 4, and 7 respectively, you must attach the mirrors in this same exact order. If you are not sure, Veritas may have created notes in your /etc/vfstab which describe how the partitions were layed before VxVM encapsulated the boot disk. It is very important to attach each mirror in the exact numerical order as it was originaly carved on the boot disk.
# vxrootmir rootmirror <=== partition 0
# vxassist -g rootdg mirror swapvol rootmirror <=== partition 1
# vxassist -g rootdg mirror var rootmirror <=== partition 3
# vxassist -g rootdg mirror opt rootmirror <=== partition 4
# vxassist -g rootdg mirror home rootmirror <=== partition 7
NOTE: Be sure to mirror any other volumes configured on the boot disk. This may be the case if you have additional volumes configured on the boot drive. Use the "vxprint" command to list all volumes. You can also use "vxprint -p" to make sure there are 2 plexes associated with each volume. Remember to modify the remaining commands to include these additional volumes.
Disassociate and remove the existing plexes from the rootdisk:
# vxplex -g rootdg dis rootvol-01 swapvol-01 var-01 opt-01 home-01
# vxedit -g rootdg -fr rm rootvol-01 swapvol-01 var-01 opt-01 home-01
# vxedit rm rootdiskPriv ### May not be in rootdg - Ignore if Error
Remove the rootdisk from rootdg:
# vxdg -g rootdg rmdisk rootdisk
Re-initialize the rootdisk and add back into rootdg:
# vxdisksetup -i c0t0d0
# vxdg -g rootdg adddisk rootdisk=c0t0d0
Re-attach the mirror in the correct order according to the partition layout:
# vxrootmir rootdisk
# vxassist -g rootdg mirror swapvol rootdisk
# vxassist -g rootdg mirror var rootdisk
# vxassist -g rootdg mirror opt rootdisk
# vxassist -g rootdg mirror home rootdisk
Create the underlying partitions on the rootdisk and mirrordisk - Optional Step - The benefits are worth the effort!
Update: This step can be automated using the Veritas supplied program: vxbootsetup. Simply execute the following command:
# vxbootsetup
Or you may do it the manual way as described below:
Notice how you did NOT create the disk partitions manually when you initialized the mirror disk. This is because VxVM automatically does its partitioning for you. This is also true for the root disk after it has been re-initialized and brought back into rootdg. That is, the original rootdisk partition scheme has been replaced by Veritas's partition scheme. This means that the underlying partitions for each volume(swap, /opt, /var, /home) are not physically layed out on the drives corresponding to the veritas volumes. In some situations though, you might want to be able to access the partitions corresponding to each volume by manually mounting the slice. In a recovery situation for example, you may want to boot from cdrom and access the /opt partition by manually mounting the slice without the veritas drivers loaded. This is not possible unless you create the underlying partition mapping directly to the opt volume using the "vxmksdpart" command. All underlying partitions except / should be created. The / partition is automatically created and is mapped correctly to the corresponding veritas root volume.
One Caveat, you cannot use partition 3 or 4 because this is typically where Veritas stores is private and public region. The following is snipped from the "format" utility listing the Veritas partitions. This shows that the root partition exists but the swap, /var, /opt, home partitions are not mapped on the drive(Veritas knows where they exist on the drive because of its private and public regions):
Part Tag Flag Cylinders Size Blocks
0 root wm 1 - 2282 3.91GB (2282/0/0) 8194662 <=== "/" exists
1 unassigned wm 0 0 (0/0/0) 0 <=== Unassigned
2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084
3 - wu 0 - 0 1.75MB (1/0/0) 3591 <=== Private Region
4 - wu 1 - 4923 8.43GB (4923/0/0) 17678493 <=== Public Region
5 unassigned wm 0 0 (0/0/0) 0 <=== Unassigned
6 unassigned wm 0 0 (0/0/0) 0 <=== Unassigned
7 unassigned wm 0 0 (0/0/0) 0 <=== Unassigned
You may use the unassigned partitions which are 1, 5,6, and 7 to map to the veritas volumes. Be sure to document which partitions map to the Veritas volumes for future reference.
The "vxmksdpart" arguments are the subdisk(which maps directly to the volume), slice, tag and flag. Be sure to use the correct tag and flag for you particular Solaris version. Reference the man page for "fmthard" for correct options. Use the "vxprint" command to get the correct subdisk names:
Rootdisk:
# vxmksdpart -g rootdg rootdisk-02 1 0x03 0x01 <=== swap
# vxmksdpart -g rootdg rootdisk-03 5 0x07 0x00 <=== /var
# vxmksdpart -g rootdg rootdisk-04 6 0x06 0x00 <=== /opt
# vxmksdpart -g rootdg rootdisk-05 7 0x08 0x00 <=== /export/home
Specify the primary swap partition as the dump device:
# dumpadm -d /dev/dsk/c0t0d0s1
RootMirror:
# vxmksdpart -g rootdg rootmirror-02 1 0x03 0x01 <=== swap
# vxmksdpart -g rootdg rootmirror-03 5 0x07 0x00 <=== /var
# vxmksdpart -g rootdg rootmirror-04 6 0x06 0x00 <=== /opt
# vxmksdpart -g rootdg rootmirror-05 7 0x08 0x00 <=== /export/home
For maximum redundancy, configure all private regions to have a copy of rootdg database
# vxedit -g rootdg set nconfig=all rootdg
# vxedit -g rootdg set nlog=all rootdg
# vxdg flush rootdg
Configure VxVM to use host spare rather than hot relocation
# vi /etc/rc2.d/S95vxvm-recover
Comment out the "vxrelocd root &" and uncomment "vxsparecheck root &"
Create OBP device alias for rootdisk and rootmirror. Set PROM to boot to the aliases.
# vxeeprom devalias rootdisk c0t0d0s0
# vxeeprom devalias rootmirror c0t1d0s0
# eeprom "boot-device=rootdisk rootmirror"
Verify the correct eeprom information:
# eeprom | grep disk
Update: The previous "vxeeprom" commands may not work for all boot proms. If you are unable to boot to the "rootdisk" device on reboot, you should set the device alias using the "nvalias" command at the boot prom. This can be done by first doing a "devalias" to display the PATH of each disk, and use the "nvalias" command to set the rootdisk and rootmirror alias manually. The following is an example:
ok devalias
…
disk /pci@1f,4000/scsi@3/disk@0,0 <=== disk is rootdisk"
disk0 /pci@1f,4000/scsi@3/disk@0,0
disk1 /pci@1f,4000/scsi@3/disk@1,0 <=== disk1 is rootmirror"
…
ok nvalias rootdisk /pci@1f,4000/scsi@3/disk@0,0
ok nvalias rootmirror /pci@1f,4000/scsi@3/disk@1,0
ok boot
How to Recover from Primary Disk Failure
Grab your backup drive from the spare parts shelf with the same exact geometry as the failed drive. You do have one; don't you?
Configure system to see drive
# devfsadm ### Prior to Solaris 8 try "# drvconfig; disks; devlinks"
Initialize the new drive, add to rootdg as "newdisk", and replace old "rootdisk" with "newdisk":
# vxdisksetup -i c0t0d0
# vxdg -g rootdg adddisk newdisk=c0t0d0
# vxdg repldisk rootdisk=newdisk
Parition the new drive exactly the same as rootmirror:
I have not figured out a way to do this automatically instead of using format. The fmthard command will not work because it does not allow overlapping partitions. The dd command should not be used as it also copies over the Veritas diskid. In which case, there will be duplicate diskid's on each disk causing the system not to boot. Please use the following procedure instead:
# format
; Enter 0 for the new drive
; Enter p for partition
; Enter p to view current partition layout
; Enter s to select the partition table of rootmirror
; Enter 1 to copy partition table of rootmirror onto new drive
; Enter l to label disk
; Enter y to continue
; Enter q to quit partition menu
; Enter q to quit format utility
Re-attach the mirror for each volume:
# vxplex att rootvol rootvol-01
# vxplex att swapvol swapvol-01
# vxplex att var var-01
# vxplex att opt opt-01
# vxplex att home home-01
How to Recover from Mirror Disk Failure
Grab your backup drive from the spare parts shelf with the same exact geometry as the failed drive. You do have one; don't you?
Configure system to see drive
# devfsadm ### Prior to Solaris 8 try "# drvconfig; disks; devlinks"
Initialize the new drive, add to rootdg as "newdisk", and replace old "rootmirror" with "newdisk"
# vxdisksetup -i c0t1d0
# vxdg -g rootdg adddisk newdisk=c0t1d0
# vxdg repldisk rootmirror=newdisk
Parition the new drive exactly the same as rootdisk:
I have not figured out a way to do this automatically instead of using format. The fmthard command will not work because it does not allow overlapping partitions. The dd command should not be used as it also copies over the Veritas diskid. In which case, there will be duplicate diskid's on each disk causing the system not to boot. Please use the following procedure instead:
# format
; Enter 1 for the new drive
; Enter p for partition
; Enter p to view current partition layout
; Enter s to select the partition table of rootdisk
; Enter 0 to copy partition table of rootdisk onto new drive
; Enter l to label disk
; Enter y to continue
; Enter q to quit partition menu
; Enter q to quit format utility
Re-attach the mirror for each volume:
# vxplex att rootvol rootvol-02
# vxplex att swapvol swapvol-02
# vxplex att var var-02
# vxplex att opt opt-02
# vxplex att home home-02
To mirror your root disk onto another disk
- Choose a disk that is at least as large as the existing root disk.
- If the selected disk is not already under VxVM control, use the vxdiskadd or vxdiskadm command, or the Veritas Enterprise Administrator (VEA) to add it to the bootdg disk group. Ensure that you specify the sliced format for the disk.
- Select menu item 6 (Mirror Volumes on a Disk) from the vxdiskadm main menu, or use the VEA to create a mirror of the root disk. (These automatically invoke the vxrootmir command if the mirroring operation is performed on the root disk.)
Alternatively, to mirror only those file systems on the root disk that are required to boot the system, run the following command: # /etc/vx/bin/vxrootmir altboot_disk where altboot_disk is the disk media name of the mirror for the root disk. vxrootmir creates a mirror for rootvol (the volume for the root file system on an alternate disk). The alternate root disk is configured to enable booting from it if the primary root disk fails. - Set the value of the EEPROM variable use-nvramrc? to true. This enables the use of VxVM boot disk aliases, which identify mirrors of the root disk from which the system can be booted. If the system is up and running, set use-nvramrc? to true using the following command:
# eeprom use-nvramrc?=true You can also set use-nvramrc? at the ok boot prompt: ok setenv use-nvramrc? true - You should now be able to boot the system from an alternate boot disk, vx-altboot_disk, by entering the following command at the ok boot prompt:
You can use the devalias command at the boot prompt to discover the alternate disks from which the system may be booted: ok devalias
