Thursday, December 30, 2010

Recovering plexes from DISABLED RECOVER state.

Problem

Recovering plexes from DISABLED RECOVER state.

Solution

Recovering plexes from DISABLED RECOVER state once lost connectivity has been restored to one or more disks.
#vxdisk list
T41_1 sliced - - online
T41_2 sliced - - online
- - data_dg15 data_dg failed was:T41_2
- - data_dg16 data_dg failed was:T41_1
# vxdisk -o alldgs list
DEVICE TYPE DISK GROUP STATUS
T41_1 sliced - (data_dg) online
T41_2 sliced - (data_dg) online
v nas04b_v - DISABLED ACTIVE 943718400 SELECT - fsgen
pl nas04b_v-01 nas04b_v DISABLED NODEVICE 943718400 CONCAT - RW
sd data_dg16-01 nas04b_v-01 data_dg16 0 943718400 0 - NDEV
Do :
/usr/lib/vxvm/bin/vxreattach -r T41_2
/usr/lib/vxvm/bin/vxreattach -r T41_1
Migrate from DISABLED RECOVER to ENABLED ACTIVE.
Symptom:
How to recover or start a volume that has a plex in the disabled and recover state
Solution:
The vxprint -ht output gives, among other things, the kernel state (column 4)
and the state (column 5) of a plex .
Consider the following case:
# vxprint -ht -g testdg
dg testdg default default 84000 970356463.1203.alu
dm testdg01 c1t4d0s2 sliced 2179 8920560 -
dm testdg02 c1t6d0s2 sliced 2179 8920560 -
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED RECOVER 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA
From the above output, it can be seen that the volume test has plex
test-01 in the DISABLED and RECOVER state.
To recover the volume test, use the vxmend command. This operation
applies only to volumes, or to plexes associated with a volume.
This will manually reset or change the state of a plex or volume.
The following is the procedure to recover/start this volume:
1. Bring plex test-01 to the DISABLED and OFFLINE state using the
following command: vxmend -o force off <recover_plex>
For example,
# vxmend -g testdg -o force off test-01
The below output shows plex test-01 in the DISABLED and OFFLINE state:
# vxprint -ht -g testdg
dg testdg default default 84000 970356463.1203.alu
dm testdg01 c1t4d0s2 sliced 2179 8920560 -
dm testdg02 c1t6d0s2 sliced 2179 8920560 -
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED OFFLINE 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA
2. Bring plex test-01 to the DISABLED and STALE state using the
following command: vxmend on <recover_plex>
For example,
# vxmend -g testdg on test-01
The below output shows plex test-01 in the DISABLED and STALE state:
# vxprint -ht -g testdg
dg testdg default default 84000 970356463.1203.alu
dm testdg01 c1t4d0s2 sliced 2179 8920560 -
dm testdg02 c1t6d0s2 sliced 2179 8920560 -
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED STALE 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA
3. Bring plex test-01 to the DISABLED and CLEAN state using the
following command: vxmend fix clean <recover_plex>
For example,
# vxmend -g testdg fix clean test-01
The below output shows plex test-01 in the DISABLED and CLEAN state:
# vxprint -ht -g testdg
dg testdg default default 84000 970356463.1203.alu
dm testdg01 c1t4d0s2 sliced 2179 8920560 -
dm testdg02 c1t6d0s2 sliced 2179 8920560 -
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED CLEAN 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA
4. Once plex test-01 is in the DISABLED/CLEAN state, the volume test can be
started with the following command: vxvol start <volume>
For example,
# vxvol start test
It can be seen in the below output that the volume is now ENABLED and ACTIVE:
# vxprint -ht -g testdg
dg testdg default default 84000 970356463.1203.alu
dm testdg01 c1t4d0s2 sliced 2179 8920560 -
dm testdg02 c1t6d0s2 sliced 2179 8920560 -
v test - ENABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test ENABLED ACTIVE 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA
To summarize:
# vxmend -g testdg -o force off test-01
# vxmend -g testdg on test-01
# vxmend -g testdg fix clean test-01
# vxvol -g testdg start test

2 comments:

  1. Thanks kumar your post was very informative to solve my issue.
    Thanks again.

    ReplyDelete