Add/Extend Nodes to Existing 11g RAC Cluster

Recently I needed to extend one of our single node cluster with one additional node. 

The host is prepared and was ready to be added to the cluster.

First of all we checked the node readiness for the addition as follows and proceed further only after those checks

[oracle@drracnode3 bin]$cluvfy stage -pre nodeadd -n  drracnode2 -verbose

[oracle@drracnode3 bin]$cluvfy stage -pre crsinst -n  drracnode2 –verbose

[oracle@drracnode3 ~]$ cd /u01/app/11.2.0.2/grid/oui/bin/

[oracle@drracnode3 bin]$ export IGNORE_PREADDNODE_CHECKS=Y

[oracle@drracnode3 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={drracnode2}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={drracnode2-vip}"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 1323 MB    Passed

Oracle Universal Installer, Version 11.2.0.2.0 Production

Copyright (C) 1999, 2010, Oracle. All rights reserved.

Performing tests to see whether nodes drracnode2 are available

............................................................... 100% Done.

-----------------------------------------------------------------------------

Cluster Node Addition Summary

Global Settings

   Source: /u01/app/11.2.0.2/grid

   New Nodes

Space Requirements

   New Nodes

      drracnode2

         /: Required 3.58GB : Available 22.01GB

Installed Products

   Product Names

      Oracle Grid Infrastructure 11.2.0.2.0

      Sun JDK 1.5.0.24.08

      Installer SDK Component 11.2.0.2.0

      Oracle One-Off Patch Installer 11.2.0.0.2

      Oracle Universal Installer 11.2.0.2.0

      Oracle Recovery Manager 11.2.0.2.0

............................................................... 100% Done.

Instantiating scripts for add node (Thursday, April 19, 2012 1:35:38 PM IST)

Instantiation of add node scripts complete

Copying to remote nodes (Thursday, April 19, 2012 1:35:41 PM IST)

.....................................................................................

Home copied to new nodes

Saving inventory on nodes (Thursday, April 19, 2012 1:50:54 PM IST)

.                                                               100% Done.

Save inventory complete

WARNING:

The following configuration scripts need to be executed as the "root" user in each cluster node.

/u01/app/11.2.0.2/grid/root.sh #On nodes drracnode2

To execute the configuration scripts:

    1. Open a terminal window

    2. Log in as "root"

    3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/11.2.0.2/grid was successful.

Please check '/tmp/silentInstall.log' for more details.

[root@drracnode2 ~]# /u01/app/11.2.0.2/grid/root.sh

Running Oracle 11g root script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /u01/app/11.2.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]: y

   Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]: y

   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

OLR initialization - successful

Adding daemon to inittab

ACFS-9200: Supported

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node drracnode3, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

After waiting for some time we checked the crsd.log file and it shows that asm system was not mounted with

ORA 1031 – Insufficient privileges

  

Here the OS user ‘oracle’ was not part of dba group hence insufficient privilege error was thrown

#usermod –g oinstall –G dba oracle

Another thing was that you don’t really need password file in this ASM instance unless you are doing password authentication. So you can remove that if needed...

Issue -

In 11.2, when logging into ASM using 'sysdba' you receive the following error:
sqlplus '/ as sysdba' 

SQL*Plus: Release 11.2.0.1.0 Production on Thu Oct 22 13:32:50 2009 

Copyright (c) 1982, 2009, Oracle. All rights reserved. 

Connected to: 
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production 
With the Automatic Storage Management option 

SQL> shutdown immediate; 
ORA-01031: insufficient privileges 
Cause


Commands that you ran using the SYSDBA privilege on ASM 11g r1 and below have now been deprecated in release 11g r2.

Starting with 11g release 2, Oracle ASM administration must be done with the SYSASM privilege. 
Solution

You will now need to connect as SYSASM to perform any administrative operations on the ASM instance or we will get an ORA-01031 error as shown below : 

ORA-01031: insufficient privileges 

Hence we tried connecting to ASM as follows and started the ASM instance manually

[oracle@drracnode3 ~]$ cd /u01/app/11.2.0.2/grid/bin/

[oracle@drracnode3 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM2

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@drracnode3 ~]$ ./sqlplus / as sysasm

SQL> startup

[oracle@drracnode3 ~]$ sqlplus  / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Thu May 10 14:37:59 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production

With the Real Application Clusters and Automatic Storage Management options

SQL> exit; 

Once the ASM is up and running we tried starting the CRS as follows

[oracle@drracnode3 ~]$ crsctl start crs

and CRS came up successfully on the node.