Troubleshooting ASM Startup Issue in 11g Grid Infrastructure

In 11g when one places the ASM spfile, OCR or VoteDisk on ASM diskgroup, thus exists a tight interlink between these components. ASM plays very important role when starting grid infrastructure stack, let it single node or multi node i.e. RAC. 
With Single Node Grid Infrastructure, the database using ASM instance has to register it self with CSS. 

So when one cannot start ASM for some reason, the question will be how to trouble shoot the issue since everything is tightly integrated to each other. So first lets look at ASM startup sequence. 

When Grid Infrastructure starts,  Oracle will try to locate the ASM parameter file, to start CSSDAgent in following sequence. 

• First it will look into GPNP Profile to find the parameter with name "asmdiskstring" 

• Profile is usually located under <GRID_HOME>/gpnp/profiles/peer
• file - profile.xml

<orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+DATA/<host>/asmparameterfile  /registry.253.768413123"/> .

Here the issue can be if the above mentioned file is not found then ASM will fail to start. So make sure your profile reflects the correct value.

• If the above step doesn't reflect any value then, the next look up will be done in GRID_HOME/dbs folder and if located the file will start using pfile found. 

 Again, the file has to be present here and if not the ASM will fail to start as it cannot locate the parameter file. So it will be advisable to have both spfile and pfile to save some pain later. 

There is a caveat here. what if  the gpnp profile reflect the the proper file which exists on ASM but cannot be opened due to corruption. In this case again ASM will fail to start.
Solution to these issue will be to start ASM with transient parameter file as follows ( this is for 2 node RAC )

Use Case - On 2 node RAC only one node is healthy and another node is having problem with CRS stack start up. 

1- Create a new ASM pfile

ora_+ASM1.ora
+ASM1.asm_diskgroups='DATA','FRA'#Manual Mount
+ASM2.asm_diskgroups='DATA','FRA'#Manual Mount
*.asm_diskstring='/dev/oracleasm/disks/*'
*.asm_power_limit=5
*.diagnostic_dest='//u01/app/11.2.0.3/grid/log'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

2- Start up the ASM instance

on the first node

$ export $ORACLE_SID=<asm instance name>
$ export $ORACLE_HOME=<full path of the asm home>

$ sqlplus / as sysdba
sql> startup pfile=<the full pathname of ora_+ASM1.ora>

This will start ASM Instance on node 1.

3) Recreate the spfile

SQL> create spfile='+DATA' from pfile='/u01/app/11.2.0.3/grid/dbs/init+ASM1.ora'; File created.

SQL> sho parameter spfile;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/<host>/asmparameterfile

                                                      /registry.253.768413123                             

Note - Pls make sure that you check your gpnp profile reflecting its new value. 

4) Restarted the OHAS/CRS stack on node

Connect as root user:

# /u01/app/11.2.0.3/grid/bin/crsctl stop crs

Make sure that all the processes are exited and then start crs

# /u01/app/11.2.0.3/grid/bin/crsctl start crs

After some time do the health check on crs stack that it is started successfully. 

# /u01/app/11.2.0.3/grid/bin/crsctl check cluster -all