Tuesday, August 7, 2012

Migrating Single Node Apps/EBS R12 DB to RAC

Being running on single node for a while for EBS, it was the time to test integration / migration of EBS to HA on DB side i.e. to 11gR2 RAC. As you are aware that R12 is packed with 11.1.0.7 aka 11gR1 binaries. So you have to first migrate them to 11gR2 Binaries.

1. The migration of single node DB to 11gR2 binaries

2. Migrate the 11gR2 EBS DB to RAC Grid database aka multinode env.

This post assumes that you have installed and configured the 11gR2(11.2.0.1) Grid Home and 11gR2 RAC RDBMS Home.


Following are the high level steps for the entire process.


1) Install and configure 11.2.0.1 cluster on 2 nodes

2) Install 11gR2 RDBMS Oracle Home on 2 nodes
3) Install Application patches
4) convert non-rac database to RAC database
5) Enable autoconfig on database and application nodes
6) Startup services and verify services

For the sake of readability I have not covered the step 1 and step 2, as they are the SOP for any RAC setup. Also you will find numerous blogs and Oracle Documentation on the same, so no point in keep repeating :)
I will cover steps 3 to 6 in this write up 
On Node 1 - 
[oracle@apps_rac01 old]$ pwd
/u01/app/oracle/product/11.0/db_1/nls/data/old
[oracle@apps_rac01 old]$ perl cr9idata.pl 
Directory /u01/app/oracle/product/11.0/db_1/nls/data/9idata already exist. Overwriting...
Copying files to /u01/app/oracle/product/11.0/db_1/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u01/app/oracle/product/11.0/db_1/nls/data/9idata!
[oracle@apps_rac01 old]$ 

On Node 2- 
[oracle@apps_rac02 old]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/nls/data/old
[oracle@apps_rac02 old]$ perl ./cr9idata.pl 
Creating directory /u02/app/11.2.0/grid/nls/data/9idata ...
Copying files to /u02/app/11.2.0/grid/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u02/app/11.2.0/grid/nls/data/9idata!

Listener config - 
-- Create new listener (make sure you check on Node2 also)
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/oracle/11.2.0/db_1 -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/product/11.2.0/db_1
End points: TCP:1521

SCAN Listener - 

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02

-- Check env on both nodes 

On Node 1- 

[oracle@apps_rac01 old]$ env | grep ORACLE_HOME

ORACLE_HOME=/u01/app/oracle/product/11.0/db_1
[oracle@apps_rac01 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac01 old]$ env | grep ORACLE_BASE
ORACLE_BASE=/u01/app/oracle
[oracle@apps_rac01 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib
On Node 2- 
[oracle@apps_rac02 old]$ env | grep ORACLE_HOME
ORACLE_HOME=/u02/app/11.2.0/grid
[oracle@apps_rac02 old]$ env | grep PATH
LD_LIBRARY_PATH=/u02/app/11.2.0/grid/lib:/lib:/usr/lib
PATH=/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:/u02/app/11.2.0/grid/bin
[oracle@apps_rac02 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac02 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib

Migration - 
-- shutdown the db and create spfile on ASM
SQL> create spfile='+DATA/spfile/spfilevisr12.ora' from pfile='/u01/app/oracle/product/11.0/db_1/dbs/initvisr12.ora';
File created.
-- Take backup of existing pfile and create new pfile as follows

[oracle@apps_rac01 dbs]$ pwd
/u01/app/oracle/product/11.0/db_1/dbs
[oracle@apps_rac01 dbs]$ scp initvisr12.ora initvisr12.ora.bkp
-- - edit the local pfile in old ORACLE_HOME and provide the location of spfile.

[oracle@apps_rac01 dbs]$ cat initvisr12.ora
spfile='+DATA/spfile/spfilevisr12.ora'
-- Startup the DB, it will now use the SPFILE from ASM Disk to start.
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:44:59 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup 
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             536873088 bytes
Database Buffers          520093696 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL > Show parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/11.0/db_1/dbs/spfilevisr12.ora                                                      

Here it should have displayed the spfile from +DATA disk group but it was showing as from cooked file system.
The reason being is that, I already had an spfile under dbs on old Oracle Home. So I just renamed it and start the DB with pfile option and forcing DB to use spfile from +DATA Disk group as follows
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@apps_rac01 dbs]$ mv  spfilevisr12.ora  spfilevisr12.ora.bkp
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:51:06 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup pfile='initvisr12.ora'
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             490735744 bytes
Database Buffers          566231040 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL> sho parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/spfile/spfilevisr12.ora

So far so good. Now comes the most interesting part of the process i.e. Converting the Non RAC database to RAC database. There are several ways we can achieve this 
1. DBCA
2. rconfig
3. Oracle Enterprise Manager
4. Manually (old school way )

For this post, I am going to use rconfig, as its the more sophisticated way and less error prone to OEM. 
So first of all you have copy the convert template from default directory to your custom directory. Since I want to convert this DB as an Admin manged RAC DB, i will use ConvertToRAC_AdminManaged.xml template.
[oracle@apps_rac01 sampleXMLs]$ pwd
/u01/app/oracle/product/11.0/db_1/assistants/rconfig/sampleXMLs
[oracle@apps_rac01 sampleXMLs]$ ll
total 8

-rw-r--r-- 1 oracle dba 2591 Mar  4  2009 ConvertToRAC_AdminManaged.xml

-rw-r--r-- 1 oracle dba 2653 Mar 12  2009 ConvertToRAC_PolicyManaged.xml

[oracle@apps_rac01 sampleXMLs]$ scp ConvertToRAC_AdminManaged.xml /home/oracle/



Open the ConvertToRAC_AdminManaged.xml and make following changes based on your environment.

Before starting actual conversion, it is advisable to check the conversion process beforehand. So I will run the rconfig utility with ONLY option to check whether my conversion will work flawlessly or not. 



[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>

    <Convert>

      <Response>

        <Result code="1" >

          Got Exception

        </Result>
       <ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: SCAN Listener is not running.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_23_49.log for more details.
       </ErrorDetails>
      </Response>
    </Convert>
  </ConvertToRAC></RConfig>

Since my SCAN Listeners were down, error was thrown as above. So its time to start SCAN Listener and run the utility again.

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02
<ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: Default Listener is not configured in Grid Infrastructure Home.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_27_28.log for more details.
   </ErrorDetails>

The issue was that my listener was running from RAC ORACLE_HOME. The default listener has to running from GRID Home. 
-- First Removed the existing listener
[oracle@apps_rac01 bin]$ cd /u02/app/oracle/prodcut/11.2.0/db_1/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/prodcut/11.2.0/db_1
End points: TCP:1521
[oracle@apps_rac01 bin]$ srvctl  stop listener
[oracle@apps_rac01 bin]$ srvctl remove listener -l listener_visr12
[oracle@apps_rac01 bin]$ cd -
/u02/app/11.2.0/grid/bin

-- Create new listener from Grid Home
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/11.2.0/grid -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521
[oracle@apps_rac01 bin]$ ssh apps_rac02
Last login: Tue Jul 31 02:54:28 2012 from apps_rac01.localdomain
[oracle@apps_rac02 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac02 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521

-- However after creating listener it still failed. After googling a bit, I found workaround as follows.
[oracle@apps_rac01 bin]$ pwd
/u02/app/11.2.0/grid/bin
[oracle@apps_rac01 bin]$ ./srvctl remove listener -l listener_visr12
Connection to apps_rac02 closed.
-- Remove the listener with custom name 
[oracle@apps_rac01 bin]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u02/app/11.2.0/grid is /u01/app/oracle
-- Start the NETCA from grid home create the listener with default name "LISTENER"
[oracle@apps_rac01 bin]$ netca
Oracle Net Services Configuration:
Oracle Net Configuration Assistant is launched from Grid Infrastructure home. Ne
twork configuration will be clusterwide.
Configuring Listener:LISTENER
apps_rac01...
apps_rac02...
Listener configuration complete.
Oracle Net Listener Startup:
    Listener started successfully.
Oracle Net Services configuration successful. The exit code is 0
[oracle@apps_rac01 bin]$ ./srvctl  status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): apps_rac01,apps_rac02

With this RCONFIG went ahead, just to give another error. 
 <ErrorDetails>
             /u01/app/oracle/product/11.0/db_1/dbs/arch LOG ARCHIVE DEST does not exist on all nodes in the cluster
       </ErrorDetails>

Workaround was pretty simple this time and just created the arch directory on both nodes and there I was done.
RCONFIG didn't complain this time.
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
There is no return value for this step     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

Being said that, it's time to run RCONFIG with YES option to do the actual conversion and here I go 
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
Converting Database "visr12" to Cluster Database. Target Oracle Home: /u02/app/oracle/prodcut/11.2.0/db_1. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace

Adding UNDO tablespaces

Adding Trace files

Setting Flash Recovery Area

Updating Oratab

Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
<Oracle_Home>
         /u02/app/oracle/prodcut/11.2.0/db_1
       </Oracle_Home>
       <Database type="ADMIN_MANAGED"  >
 <InstanceList>
           <Instance SID="visr121" Node="apps_rac01"  >
           </Instance>
           <Instance SID="visr122" Node="apps_rac02"  >
           </Instance>
         </InstanceList>
       </Database>     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

[oracle@apps_rac01 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config database -d visr12
Database unique name: visr12
Database name: visr12

Oracle home: /u02/app/oracle/prodcut/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/spfilevisr12.ora

Domain: 

Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: visr12
Database instances: visr121,visr122
Disk Groups: DATA
Services: 
Database is administrator managed

-- Copy appsutil directory from source ORACLE HOME to destination ORACLE HOME (on both nodes)
[oracle@apps_rac01 bin]$ scp -r appsutil /u02/app/oracle/prodcut/11.2.0/db_1/
[oracle@apps_rac01 bin]$ scp -r appsutil apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/

-- Copy network/admin folder from source ORACLE HOME to destination ORACLE HOME  (on both nodes)
-- Change the path of ORACLE HOME from old to new (RAC) and modify the INSTANCE_NAME parameter on the respective nodes
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ /u02/app/oracle/prodcut/11.2.0/db_1/network/admin/
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/network/admin/visr12_apps_rac02/

-- You may also have to add Alias using NETCA (Run from GI Home)
Node 1 - 
[oracle@apps_rac01 bin]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 13:47:16
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 13:45:50
Uptime                    0 days 0 hr. 1 min. 25 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.71)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.72)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).

  Instance "visr121", status READY, has 1 handler(s) for this service...

The command completed successfully

Node 2 - 
[oracle@apps_rac02 visr12_apps_rac02]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 08:46:09
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 04:32:38
Uptime                    0 days 4 hr. 13 min. 33 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac02/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.73)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.74)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).
  Instance "visr122", status READY, has 1 handler(s) for this service...
The command completed successfully

-- Create context file with adbldxml.pl utility on both DB Nodes


[oracle@apps_rac01 bin]$ hostname
apps_rac01.localdomain
[oracle@apps_rac01 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac01 bin]$ perl adbldxml.pl appsuser=apps appspass=apps

The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07311354.log
AC-20010: Error: File - listener.ora could not be found at the location: 
        /listener.ora 
indicated by TNS_ADMIN. Context file can not be generated.
Could not Connect to the Database with the above parameters, Please answer the Questions below
Enter Hostname of Database server: 
Enter Hostname of Database server: apps_rac01            
Enter Port of Database server: 1521

Enter SID of Database server[visr12]:visr121
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml



[oracle@apps_rac02 bin]$ hostname 
apps_rac02.localdomain
[oracle@apps_rac02 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac02 bin]$ perl ./adbldxml.pl appsuser=apps appspass=apps
Starting context file generation for db tier..
Using JVM from /u02/app/oracle/prodcut/11.2.0/db_1/jdk/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07310853.log

AC-20010: Error: File - listener.ora could not be found at the location: 

        /listener.ora 

indicated by TNS_ADMIN. Context file can not be generated.

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: apps_rac02
Enter Port of Database server: 1521
Enter SID of Database server: visr122
Enter the value for Display Variable: 0
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml

-- Make sure your LOCAL_LISTENER are registered with DB 
Node 1 - 


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr121
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------

listener_networks                    string

local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD

                                                 DRESS=(PROTOCOL=TCP)(HOST=apps

                                                 _rac01-vip)(PORT=1521))))

remote_listener                      string      apps-scan.localdomain:1521

Node 2 - 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr122
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
listener_networks                    string
local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD
                                                 DRESS=(PROTOCOL=TCP)(HOST=apps
                                                 _rac02-vip)(PORT=1521))))
remote_listener                      string      apps-scan.localdomain:1521


-- Run AutoConfig on both nodes 
Node 1 - 
[oracle@apps_rac01 bin]$ ./adconfig.sh contextfile=../visr121_apps_rac01.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr121_apps_rac01/07311404/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

Node 2 - 
[oracle@apps_rac02 bin]$ ./adconfig.sh contextfile=../visr122_apps_rac02.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr122_apps_rac02/07310903/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml
Context Value Management will now update the Context file
  Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

-- In DB you can verify both the nodes 
SQL> select node_name from apps.fnd_nodes;
NODE_NAME
------------------------------
APPS_RAC01
APPS_RAC02

-- Check and set the DB Env for TNS_ADMIN
Node 1 - 
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
[oracle@apps_rac01 bin]$ ./srvctl setenv database -d visr12 -T TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
Node 2 - 
[oracle@apps_rac02 bin]$ cd ../../bin/
[oracle@apps_rac02 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin

-- Modify the CONTEXT FILE on apps tier & add additional node info 
[oracle@appsnode_new appl]$ vi  $CONTEXT_FILE
 <jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac01.localdomain)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac02.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=visr12))))</jdbc_url>

-- Run AutoConfig on Apps tier
[oracle@appsnode_new appl]$ adautocfg.sh 
Enter the APPS user password:
The log file for this session is located at: /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/admin/log/07312029/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
        Using CONFIG_HOME location     : /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new
Classpath                   : /u01/app/oracle/visr12/apps/apps_st/comn/java/lib/appsborg2.zip:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes
     Configuring IES_TOP.......COMPLETED
        Configuring CSD_TOP.......COMPLETED
        Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.

-- Start the Apps Tier Services 
[oracle@appsnode_new appl]$ adstrtal.sh apps/apps
You are running adstrtal.sh version 120.15.12010000.3
The logfile for this session is located at /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/admin/log/adstrtal.log
Executing service control script:

Now log-in to apps console and navigate to system administrator responsibility -> OAM -> Database status. You can see now both the nodes available and active as follows.

Set Up Load Balancing On Apps Tier

Now its time to implement load balancing for the Oracle Applications database connections...

Run the Context Editor (through the Oracle Applications Manager interface) and set the value of "Tools OH TWO_TASK" (s_tools_two_task), "iAS OH TWO_TASK" (s_weboh_twotask) and "Apps JDBC Connect Alias" (s_apps_jdbc_connect_alias). 

To load balance the forms based applications database connections, set the value of "Tools OH TWO_TASK" to point to the <database_name>_balance alias generated in the tnsnames.ora file.

To load balance the self-service applications database connections, set the value of "iAS OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in the tnsnames.ora file.


Execute AutoConfig by running the command
[oracle@appsnode_new appl]$ adautocfg.sh

Restart the Applications processes, using the new scripts generated by AutoConfig.
Ensure that value of the profile option "Application Database ID" is set to dbc file name generated in $FND_SECURE. 

Hope that this post will help you in your Apps to RAC migration successfully. Comments are welcome.