Wednesday, April 3, 2013

PRCR-1013 : Failed to start resource  OR 

CRS-2640: Required resource is missing


During start of DB resource we hit following error...

Error - 
PRCR-1013 : Failed to start resource ora.ASdb.db
PRCR-1064 : Failed to start resource ora.ASdb.db on node orarac1d
CRS-2640: Required resource 'ora.ASDATA03.dg' is missing.

So it says the disk group missing. When I check the ASM for the existing disk group I found that the complain was correct and I indeed cannot see the ASDATA03 disk group. After some digging I found out that disk group has been renamed from ASDATA03 to ASDATA03_05 hence the dependency was failing during start of the resource.

Diskgroup Status in ASM for ASDB -
ASDATA02_04                  MOUNTED     EXTERN             0
ASDATA03_05                  MOUNTED     EXTERN             0
ASDATA01                     MOUNTED     EXTERN             0

Disgroups registered with Cluster resouse ASDB- 
Disk Groups: ASDATA01,FRA (Missing DG ASDATA02_04 and ASDATA03_05)

Currently Datafile and archlog files are distributed on following - 
Diskgorups - ASDATA01, ASDATA03_05, ASDATA02_04

Since ASM diskgroup ASDATA02 and ASDATA03 hAS been physically dropped, We should them from CRS start/stop dependency for databASe resource 

[grid@orarac1d bin]$ crsctl stat res -p | grep ASDB
NAME=ora.FRA_ASDB.dg
DB_UNIQUE_NAME=ASDB
GEN_AUDIT_FILE_DEST=/u01/app/oracle/admin/ASDB/adump
GEN_USR_ORA_INST_NAME@SERVERNAME(orarac1c)=ASDB1
GEN_USR_ORA_INST_NAME@SERVERNAME(orarac1d)=ASDB2
SERVER_POOLS=ora.ASDB
SPFILE=+ASDATA01/ASDB/spfileASDB.ora

START_DEPENDENCIES=hard(ora.ASDATA01.dg,ora.FRA.dg, ora.ASDATA03.dg,ora.FRA_ASDB.dg,ora.ASDATA02.dg)
weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.ASDATA01.dg,ora.FRA.dg)

STOP_DEPENDENCIES=hard(intermediate:ora.ASm,shutdown:ora.ASDATA01.dg,shutdown:ora.FRA.dg,shutdown:ora.ASDATA03.dg, shutdown:ora.FRA_ASDB.dg, shutdown:ora.ASDATA02.dg)

Fix –
1. Modify ASM disk group dependency requirement
$ srvctl modify databASe -d MYDB -a “ASDATA01,ASDATA03_05,ASDATA02_04,FRA,FRA_ASDB”

2. Disable disk group if exist
$ srvctl disable diskgroup -g ASDATA03,ASDATA02 

3. Drop disk group info from OCR. Make sure you dont have any existing DB using this disk groups
$ srvctl remove diskgroup -g ASDATA03,ASDATA02 –f

Once you finish this steps and next time around when you start or stop the resource using srvctl utility, it wont complain and will start resources as expected.