Wednesday, August 29, 2012


Error During 11.2.0.2 Rolling Upgrade

Error- 
Creating trace directory
Failed to add (property/value):('OLD_OCR_ID/'-1') for checkpoint:ROOTCRS_OLDHOMEINFO.Error code is 256
The fixes for bug 9413827 are not present in the 11.2.0.1 crs home
Apply the patches for these bugs in the 11.2.0.1 crs home and then run rootupgrade.sh
/u03/app/11.2.0.2/grid/perl/bin/perl -I/u03/app/11.2.0.2/grid/perl/lib -I/u03/app/11.2.0.2/grid/crs/install /u03/app/11.2.0.2/grid/crs/install/rootcrs.pl execution failed

Though it was indicating that it needs to have 9413827 patch, but is actually included in 655006 which turns out to be the 11.2.0.1.2 PSU. The download file is
p9655006_112010_Linux-x86-64.zip

Bug 9413827 - CRS rolling upgrade from 11.2.0.1 to 11.2.0.2 fails with OCR on ASM [ID 9413827.8]
One imp thing is worth mentioning that if  If "Patch 9655006 - 11.2.0.1.2 for Grid Infrastructure (GI) Patch Set Update" is applied on the GI home, then applying 9413827 or 9706490 (as per ID 1274629.1) will always try to rollback the 9655006. However, what I figured out is that if you already have 9655006 aka 11.2.0.1.2 PSU applied on GI and RAC then no need to applied the 9413827 or 9706490.

Before applying the 9655006 one has to upgrade the OPatch to 11.2.0.3 version as its the pre-req patch
One can apply patch p6880880 on all RAC Nodes before apply PSU.

On Node 1 - 
-- apply p9655006 patch on grid home
[oracle@apps_rac01 Oracle11g]$ export PATH=$PATH:/u02/app/11.2.0/grid/OPatch
[oracle@apps_rac01 Oracle11g]$ which opatch
/u02/app/11.2.0/grid/OPatch/opatch

[root@apps_rac01 ~]# opatch auto /home/oracle/Oracle11g -och /u02/app/11.2.0/grid
Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn Oraclers/install/crsconfig_params
opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_1
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Unable to determine if /u02/app/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u02/app/11.2.0/grid
patch /home/oracle/Oracle11g/9655006  apply successful for home  /u02/app/11.2.0/grid
patch /home/oracle/Oracle11g/9654983  apply successful for home  /u02/app/11.2.0/grid

ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.

-- apply p9655006 patch on RAC home 
[oracle@apps_rac01 Oracle11g]$ sudo su -
[root@apps_rac01 ~]#  export PATH=$PATH:/u02/app/oracle/prodcut/11.2.0/db_1/OPatch
[root@apps_rac01 ~]# which opatch
/u02/app/oracle/prodcut/11.2.0/db_1/OPatch/opatch
[root@apps_rac01 ~]# opatch version
Oracle Home is not set. OPatch cannot proceed!
OPatch returns with error code = 1
[root@apps_rac01 ~]# export ORACLE_HOME=/u02/app/oracle/prodcut/11.2.0/db_1
[root@apps_rac01 ~]# opatch version

OPatch Version: 11.2.0.3.0
OPatch succeeded.
[root@apps_rac01 Oracle11g]# opatch auto /home/oracle/Oracle11g -oh /u02/app/oracle/prodcut/11.2.0/db_1
Executing /usr/bin/perl /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn Oracle11g -oh /u02/app/oracle/prodcut/11.2.0/db_1 -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_13-23-45.log
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Unable to determine if /u02/app/oracle/prodcut/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

patch /home/oracle/Oracle11g/9655006/custom/server/9655006  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1
patch /home/oracle/Oracle11g/9654983  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1


On Node 2 - 
-- apply p9655006 patch on grid home
[root@apps_rac02 ~]# export PATH=$PATH:/u02/app/11.2.0/grid/OPatch
[root@apps_rac02 ~]# which opatch
/u02/app/11.2.0/grid/OPatch/opatch

[root@apps_rac02 stage]# opatch auto /home/oracle/stage -och /u02/app/11.2.0/grid
Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn stage -och /u02/app/11.2.0/grid -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_08-02-58.log
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp

Unable to determine if /u02/app/11.2.0/grid is shared oracle home

Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

Successfully unlock /u02/app/11.2.0/grid
patch /home/oracle/stage/9655006  apply successful for home  /u02/app/11.2.0/grid
patch /home/oracle/stage/9654983  apply successful for home  /u02/app/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
-- apply p9655006 patch on RAC home
[root@apps_rac02 stage]#  export PATH=$PATH:/u02/app/oracle/prodcut/11.2.0/db_1/OPatch
[root@apps_rac02 stage]# export ORACLE_HOME=/u02/app/oracle/prodcut/11.2.0/db_1
[root@apps_rac02 stage]# which opatch
/u02/app/11.2.0/grid/OPatch/opatch
[root@apps_rac02 stage]# /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/opatch version
OPatch Version: 11.2.0.3.0
OPatch succeeded.
[root@apps_rac02 stage]# opatch auto /home/oracle/stage -oh /u02/app/oracle/prodcut/11.2.0/db_1
Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn stage -oh /u02/app/oracle/prodcut/11.2.0/db_1 -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_08-29-26.log
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
(yes/no):yes
patch /home/oracle/stage/9655006/custom/server/9655006  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1
patch /home/oracle/stage/9654983  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1

Once this patch is applied you can now safely go ahead with your remaining part of the rolling upgrade.

Monday, August 27, 2012


Set Up Parallel Concurrent Processing in Apps/EBS R12

Edit the applications context file via Oracle Applications Manager, and set the value of the variable APPLDCP to ON.Go to - System Administrator responsibility  -> OAM Setup -> Site Map -> Auto Config -> click on Edit Parameters icon of the desired application tier node 


     Repeat the same for the second Apps Tier node i.e. appsode_new.localdomain

Execute AutoConfig by running the following command on all concurrent processing nodes:
-- Node 1
[oracle@appsnode_new ~]$ hostname
appsnode_new.localdomain
[oracle@appsnode_new ~]$ adautocfg.sh
-- Node 2 
[oracle@appsnode2_new ~]$ hostname
appsnode2_new.localdomain
[oracle@appsnode2_new ~]$ adautocfg.sh

Check the tnsnames.ora and listener.ora configuration files, located in $INST_TOP/ora/10.1.2/network/admin. Ensure that the required FNDSM and FNDFS entries are present for all other concurrent nodes.
-- Node 1
[oracle@appsnode_new forms_default_group_1]$ adalnctl.sh status
adalnctl.sh version 120.3
Checking status for listener process APPS_visr12.
LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 13-AUG-2012 14:04:15
Copyright (c) 1991, 2004, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=appsnode_new)(Port=1626))
STATUS of the LISTENER
------------------------
Alias                     APPS_visr12
Version                   TNSLSNR for Linux: Version 10.1.0.5.0 - Production
Start Date                13-AUG-2012 12:37:34
Uptime                    0 days 1 hr. 26 min. 41 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/ora/10.1.2/network/admin/listener.ora
Listener Log File         /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/ora/10.1.2/network/apps_visr12.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=appsnode_new.localdomain)(PORT=1626)))
Services Summary...
Service "FNDFS" has 1 instance(s).
  Instance "FNDFS", status UNKNOWN, has 1 handler(s) for this service...
Service "FNDSM" has 1 instance(s).
  Instance "FNDSM", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

-- Node 2
[oracle@appsnode2_new admin]$ adalnctl.sh  status
adalnctl.sh version 120.3
Checking status for listener process APPS_visr12.
LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 13-AUG-2012 11:12:13
Copyright (c) 1991, 2004, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=appsnode2_new)(Port=1626))
STATUS of the LISTENER
------------------------
Alias                     APPS_visr12
Version                   TNSLSNR for Linux: Version 10.1.0.5.0 - Production
Start Date                13-AUG-2012 09:51:00
Uptime                    0 days 1 hr. 21 min. 12 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/ora/10.1.2                          /network/admin/listener.ora
Listener Log File         /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/logs/ora/1                          0.1.2/network/apps_visr12.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=appsnode2_new.localdomain)(PORT=1626)))
Services Summary...
Service "FNDFS" has 1 instance(s).
  Instance "FNDFS", status UNKNOWN, has 1 handler(s) for this service...
Service "FNDSM" has 1 instance(s).
  Instance "FNDSM", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully


If Entries are not find then try to restart the Applications listener processes on each application tier node. 

Log on to Oracle E-Business Suite Release 12 using the SYSADMIN account, and choose the System Administrator Responsibility. Navigate to Install > Nodes screen, and ensure that each node in the cluster is registered.


Verify that the Internal Monitor for each node is defined properly, with correct primary node specification, and work shift details. For example, Internal Monitor: Host1 must have primary node as host1. Also ensure that the Internal Monitor manager is activated: this can be done from Concurrent > Manager > Administrator. 


Set the $APPLCSF environment variable on all the Concurrent Processing nodes to point to a valid log directory on both apps tier nodes. If the file systems are local then make sure it has proper privileges. If its on a shared file system, then make sure both apps nodes can write to it.

Set profile option 'Concurrent: PCP Instance Check' to ON. 
By setting it to 'ON', a concurrent manager will fail over to a secondary Application tier node if the database instance to which it is connected becomes unavailable for some reason.

Set Up Transaction Managers

Shut down the application services (servers) on all nodes

Shut down all the database instances cleanly in the Oracle RAC environment,

Edit $ORACLE_HOME/dbs/<context_name>_ifile.ora. Add the following parameters:
_lm_global_posts=TRUE
_immediate_commit_propagation=TRUE

Start the instances on all database nodes, one by one.

Start up the application services (servers) on all nodes.

Log on to Oracle E-Business Suite using the SYSADMIN account, and choose the System Administrator responsibility. Navigate to Profile > System, change the profile option ‘Concurrent: TM Transport Type' to ‘QUEUE', and verify that the transaction manager works across the Oracle RAC instance.

Navigate to Concurrent > Manager > Define screen, and set up the primary and secondary node names for transaction managers.

Do this for all needed transaction managers and Restart the concurrent managers on all nodes.
If any of the transaction managers are in deactivated status, activate them from Concurrent > Manager > Administrator.

Test - 
Now try to stop CM Processes on all nodes and start the second node. You will be able to see that for the concurrent managers for which primary node down and secondary node up, they will start on secondary node.
That's it. You just configured the Parallel Concurrent Processing for your Apps R12!

Thursday, August 16, 2012

Add Application Tier Node to Apps/EBS R12

After migrating the Apps/EBS R12 DB to RAC with two nodes it's the time to do the same for the Apps Tier. 
There are two methods available for this addition of the new node. One is using shared file system and one is traditional method, using local file system on the node. 

One can use OCFS2 or NFS as shared file system and can migrate the apps files to shared mount point which can then, in turn, can accessible by multiple nodes. However, I choose to go with the later approach for the sake of easy implementation. 

Topology info - 
apps_rac01 - Apps RAC DB Node1
apps_rac02 - Apps RAC DB Node2
DB - visr12
appsnode_new - Apps Application Tier Node1
appsnode2_new- Apps Application Tier Node2

The overall process is divided in three parts 
1. Clone the node the target system
2. Add node using adaddnode.pl script
3. Set Up DB Load Balancing On Apps Tier

Clone New Node - 

Configure Pre Clone
[oracle@appsnode2_new bin]$ perl ./adpreclone.pl appsTier

Copy the application tier file system
Log on to the source system application tier nodes and shut down the application tier server processes. Copy the "apps" from source node to the target application tier node

Start Clone 
[oracle@appsnode2_new bin]$ perl ./adcfgclone.pl appsTier
                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA
                        Oracle Applications Rapid Clone
                                 Version 12.0.0
                      adcfgclone Version 120.31.12010000.8
Enter the APPS password :
Running:
/u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/../jre/bin/java -Xmx600M -cp /u01/app/oracle/visr12/apps/apps_st/comn/clone/jlib/java:/u01/app/oracle/visr12/apps/apps_st/comn/clone/jlib/xmlparserv2.jar:/u01/app/oracle/visr12/apps/apps_st/comn/clone/jlib/ojdbc14.jar oracle.apps.ad.context.CloneContext -e /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/../context/apps/CTXORIG.xml -validate -pairsfile /tmp/adpairsfile_25768.lst -stage /u01/app/oracle/visr12/apps/apps_st/comn/clone  2> /tmp/adcfgclone_25768.err; echo $? > /tmp/adcfgclone_25768.res
Log file located at /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/CloneContext_0810092623.log
Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [appsnode2_new] : appsnode2_new.localdomain

Target System Database SID : visr12
Target System Database Server Node [appsnode2_new] : apps-scan.localdomain
Target System Base Directory : /u01/app/oracle/visr12
Target System Tools ORACLE_HOME Directory [/u01/app/oracle/visr12/apps/tech_st/10.1.2] :
Target System Web ORACLE_HOME Directory [/u01/app/oracle/visr12/apps/tech_st/10.1.3] :
Target System APPL_TOP Directory [/u01/app/oracle/visr12/apps/apps_st/appl] :
Target System COMMON_TOP Directory [/u01/app/oracle/visr12/apps/apps_st/comn] :
Target System Instance Home Directory [/u01/app/oracle/visr12/inst] :
Target System Root Service [enabled] :
Target System Web Entry Point Services [enabled] :
Target System Web Application Services [enabled] :
Target System Batch Processing Services [enabled] :
Target System Other Services [disabled] :
Do you want to preserve the Display [appsnode_new:0.0] (y/n) ? : y
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : y

Complete port information available at /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/out/visr12_appsnode2_new/portpool.lst

RC-40201: Unable to connect to Database visr12.
Creating the new APPL_TOP Context file from :
  /u01/app/oracle/visr12/apps/apps_st/appl/ad/12.0.0/admin/template/adxmlctx.tmp
The new APPL_TOP context file has been created :
  /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/appl/admin/visr12_appsnode2_new.xml
Log file located at /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/CloneContext_0810092623.log
Check Clone Context logfile /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/CloneContext_0810092623.log for details.
Running Rapid Clone with command:
perl /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/adclone.pl java=/u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/../jre mode=apply stage=/u01/app/oracle/visr12/apps/apps_st/comn/clone component=appsTier method=CUSTOM appctxtg=/u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/appl/admin/visr12_appsnode2_new.xml showProgress contextValidated=true
Do you want to preserve the Display [appsnode_new:0.0] (y/n) ? : y
Do you want the the target system to have the same port values as the source system (y/n) [y] ? : y
Complete port information available at /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/out/visr12_appsnode2_new/portpool.lst
RC-40201: Unable to connect to Database visr12.
Creating the new APPL_TOP Context file from :
  /u01/app/oracle/visr12/apps/apps_st/appl/ad/12.0.0/admin/template/adxmlctx.tmp
The new APPL_TOP context file has been created :
  /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/appl/admin/visr12_appsnode2_new.xml
Log file located at /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/CloneContext_0810092623.log
Check Clone Context logfile /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/CloneContext_0810092623.log for details.
Running Rapid Clone with command:
perl /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/adclone.pl java=/u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/../jre mode=apply stage=/u01/app/oracle/visr12/apps/apps_st/comn/clone component=appsTier method=CUSTOM appctxtg=/u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/appl/admin/visr12_appsnode2_new.xml showProgress contextValidated=true
ApplyAppsTier_08100928.log
  /     73% completed
Completed Apply...
Fri Aug 10 09:42:06 2012
Do you want to startup the Application Services for visr12? (y/n) [y] : n
Services not started

Add Cloned Node to Existing Apps Instance - 
-- Source the env
[oracle@appsnode2_new bin]$ cd /u01/app/oracle/visr12/apps/apps_st/appl/
[oracle@appsnode2_new appl]$  . ./APPSvisr12_appsnode2_new.env
-- Invoke the adaddnode.pl script
[oracle@appsnode2_new appl]$ cd /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/
[oracle@appsnode2_new bin]$ perl ./adaddnode.pl
Enter the APPS username: apps
Enter the APPS user password:

       Using CONTEXT_FILE : /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/appl/admin/visr12_appsnode2_new.xml
        Using      CTXORIG : /u01/app/oracle/visr12/apps/apps_st/comn/clone/bin/../context/apps/CTXORIG.xml
APPL_TOP name of the source system (old APPL_TOP name) : appsnode_new
APPL_TOP name of the target system (new APPL_TOP name) : appsnode2_new
Configuring snapshot information for the original (clone source) node...
Executing command:
sqlplus /nolog @/u01/app/oracle/visr12/apps/apps_st/appl/ad/12.0.0/patch/115/sql/adadmrat.sql APPS apps visr12 visr12 appsnode2_new appsnode_new /u01/app/oracle/visr12/inst/apps/visr12_appsnode2_new/temp/adadmat_sql.spool
APPL_TOP Names in AD_APPL_TOPS:
*PRESEEDED*
GLOBAL
ap6003bld
ap6004bld
ap6009bld
appsnode2_new
appsnode_new
7 rows selected.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Commit complete.
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
Completed snapshot configuration on target node.


Set Up DB Load Balancing On Apps Tier - 


Now its time to implement load balancing for the Oracle Applications database connections...

Run the Context Editor (through the Oracle Applications Manager interface) and set the value of "Tools OH TWO_TASK" (s_tools_two_task), "iAS OH TWO_TASK"(s_weboh_twotask) and "Apps JDBC Connect Alias" (s_apps_jdbc_connect_alias). 

To load balance the forms based applications database connections, set the value of "Tools OH TWO_TASK" to point to the <database_name>_balance alias generated in the tnsnames.ora file.

To load balance the self-service applications database connections, set the value of "iAS OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in the tnsnames.ora file.



Execute AutoConfig by running the command

[oracle@appsnode_new appl]$ adautocfg.sh

Restart the Applications processes, using the new scripts generated by AutoConfig.
Ensure that value of the profile option "Application Database ID" is set to dbc file name generated in $FND_SECURE. 


Once you have added the node successfully, however you can only use only one node and there is no Active-Active load balancing in place, coz one need Load Ba-lancer for that. 


Next you have to configure the Load Balancing with multiple apps tier and PCP [parallel con-current processing], which will be part of my coming post...  Stay tuned...

Ref Note - Using Load-Balancers with Oracle E-Business Suite Release 12 [ID 380489.1]

Friday, August 10, 2012

Enabling Forms Socket Mode In EBS/Apps

Oracle Applications Release12 by default configures Forms 10g stack in servlet mode, as this is the preferred and recommended deployment model for deploying forms on the web.  In this mode, there is a java servlet called the Forms Listener Servlet, manages the communication between the Forms Java Client and OracleAS Forms ServicesThe Forms Listener Servlet architecture operates through the HTTP server port alone and does not need extra ports to handle communication between the client and the application server. The servlet architecture is designed to work with industry standards, and also to be fully supported  for advanced network configurations with no additional ports or SSL configuration.

To check if forms is implemented in socket /servlet mode perform the following steps: 
Step 1: Source the middle-tier environment file from <Install_Home>/apps/apps_st/appl directory
Note:  Replace <Install_Home>  with the base directory where Oracle Application R12 is installed in your system
Step 2: Open the appsweb.cfg file pointed by environment variable "FORMS_WEB_CONFIG_FILE" and check the values for following parameters:

  • serverURL=
  • connectMode=
Note:  Refer the table below for the values that they must contain based on the implementation mode 
 

In servlet mode
In Socket mode
serverURL=/forms/lservlet
connectMode=servlet
serverURL=(should be blank)
connectMode=Socket

Although forms servlet is the preferred deployment method, there may be circumstances where customers need to switch from Forms servlet mode to socket mode, which allows desktop clients to access the Forms server directly. This may be required in the following situations:
  • Customers network topology is multinode and the Forms Services are configured on a node different from the node on which Web services(Web Entry Point and Web Applications) are configured.
  • Customers constrained by network bandwidth, or machine resources may consider socket mode as an alternative to improve performance. 
  • To reduce network traffic. The servlet mode uses http protocol on each transaction between a client and the Forms Server requiring the exchange of cookies and http headers which increases network traffic. 
  • To reduce consumption of resources use by the JVMS needed in servlet mode architecture. 
Forms10g
Comments
Listener executable name is frmsrv
Both executables are present in $ORACLE_HOME/bin.
Listener supports only non secure socketmode and servlet mode 
In Oracle Applications Release 12 customers who need http or https must use the servlet implementation.
Forms metric server configuration is obsolete in Forms10g
In Oracle Applications Release 12 customers who want  load balancing must use the servlet implementation in which forms servers can be balanced across multiple application servers by using a hardware load balancer along with the 10g Application Server.
CGI implementation is obsolete and executable is not present in 10g
Oracle Applications Release 12 the initial request that dynamically generates the html page to start the forms applet is processed by the Forms Servlet, although the servlet only receives one request per forms session.
Forms socket mode implements a default 40 bit encryption based on keys randomly generated at runtime
Oracle do not promote the default encryption as a "secure solution". Customers  who require high levels of security should use the Forms servlet configured for https.  If they need to use socket mode, they should use a secure VPN connection to provide the required level of security.

Imp Note - Forms Socket mode in Oracle Applications Release 12 is currently not supported on Windows platforms.


Process for Enabling Forms Socket Mode



Execute the following steps to switch from Forms Servlet mode to Forms Socket mode:

-- Source the environment on the application tier.
-- Stop all the application tier services:
$INST_TOP/admin/scripts/adstpall.sh

-- Run the following command to enable Forms Socket Mode:
$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \
[-contextfile=<CONTEXT_FILE>] \
-mode=socket \
[-port=<Forms port number>] \
-runautoconfig=<No or Yes> \
-appspass=<APPS password>


$FND_TOP/bin/txkrun.pl -script=ChangeFormsMode \
-contextfile=$INST_TOP/appl/admin/mycontext.xml \
-mode=socket \
-port=9095 \
-runautoconfig=No

-- If Autoconfig was not automatically executed as part of the preceding step, then run AutoConfig on the application tier:
$INST_TOP/admin/scripts/adautocfg.sh

-- Start all the application tier services:
$INST_TOP/admin/scripts/adstrtal.sh

-- Check whether Forms Server is running:
$INST_TOP/admin/scripts/adformsrvctl.sh status

-- Log in to Oracle Applications and  launch a Forms-based application.
-- Open the Sun Java Console (from Tools Menu in Internet Explorer).
-- Check whether the "mode" directive displayed in Sun Java Console when launching forms-based applications is set to socket.

-- The direct launch URL for Forms Socket Mode is:
<web_protocol>://<web_host>.<web_domain>:<web_port>/OA_HTML/frmservlet

6601055STOPPING FORM SERVER (FRMSRV) DOES NOT STOP CHILD FRMWEB PROCESSES
  • Apply patch 6329757.

Tuesday, August 7, 2012

Migrating Single Node Apps/EBS R12 DB to RAC

Being running on single node for a while for EBS, it was the time to test integration / migration of EBS to HA on DB side i.e. to 11gR2 RAC. As you are aware that R12 is packed with 11.1.0.7 aka 11gR1 binaries. So you have to first migrate them to 11gR2 Binaries.

1. The migration of single node DB to 11gR2 binaries

2. Migrate the 11gR2 EBS DB to RAC Grid database aka multinode env.

This post assumes that you have installed and configured the 11gR2(11.2.0.1) Grid Home and 11gR2 RAC RDBMS Home.


Following are the high level steps for the entire process.


1) Install and configure 11.2.0.1 cluster on 2 nodes

2) Install 11gR2 RDBMS Oracle Home on 2 nodes
3) Install Application patches
4) convert non-rac database to RAC database
5) Enable autoconfig on database and application nodes
6) Startup services and verify services

For the sake of readability I have not covered the step 1 and step 2, as they are the SOP for any RAC setup. Also you will find numerous blogs and Oracle Documentation on the same, so no point in keep repeating :)
I will cover steps 3 to 6 in this write up 
On Node 1 - 
[oracle@apps_rac01 old]$ pwd
/u01/app/oracle/product/11.0/db_1/nls/data/old
[oracle@apps_rac01 old]$ perl cr9idata.pl 
Directory /u01/app/oracle/product/11.0/db_1/nls/data/9idata already exist. Overwriting...
Copying files to /u01/app/oracle/product/11.0/db_1/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u01/app/oracle/product/11.0/db_1/nls/data/9idata!
[oracle@apps_rac01 old]$ 

On Node 2- 
[oracle@apps_rac02 old]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/nls/data/old
[oracle@apps_rac02 old]$ perl ./cr9idata.pl 
Creating directory /u02/app/11.2.0/grid/nls/data/9idata ...
Copying files to /u02/app/11.2.0/grid/nls/data/9idata...
Copy finished. 
Please reset environment variable ORA_NLS10 to /u02/app/11.2.0/grid/nls/data/9idata!

Listener config - 
-- Create new listener (make sure you check on Node2 also)
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/oracle/11.2.0/db_1 -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/product/11.2.0/db_1
End points: TCP:1521

SCAN Listener - 

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02

-- Check env on both nodes 

On Node 1- 

[oracle@apps_rac01 old]$ env | grep ORACLE_HOME

ORACLE_HOME=/u01/app/oracle/product/11.0/db_1
[oracle@apps_rac01 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac01 old]$ env | grep ORACLE_BASE
ORACLE_BASE=/u01/app/oracle
[oracle@apps_rac01 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib
On Node 2- 
[oracle@apps_rac02 old]$ env | grep ORACLE_HOME
ORACLE_HOME=/u02/app/11.2.0/grid
[oracle@apps_rac02 old]$ env | grep PATH
LD_LIBRARY_PATH=/u02/app/11.2.0/grid/lib:/lib:/usr/lib
PATH=/usr/lib/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:/u02/app/11.2.0/grid/bin
[oracle@apps_rac02 old]$ env | grep ORA_NLS10
ORA_NLS10=/u01/app/oracle/product/11.0/db_1/nls/data/9idata
[oracle@apps_rac02 old]$ env | grep LD_LIBRARY_PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.0/db_1/lib:/usr/X11R6/lib:/usr/openwin/lib:/u01/app/oracle/product/11.0/db_1/lib:/usr/dt/lib:/u01/app/oracle/product/11.0/db_1/ctx/lib

Migration - 
-- shutdown the db and create spfile on ASM
SQL> create spfile='+DATA/spfile/spfilevisr12.ora' from pfile='/u01/app/oracle/product/11.0/db_1/dbs/initvisr12.ora';
File created.
-- Take backup of existing pfile and create new pfile as follows

[oracle@apps_rac01 dbs]$ pwd
/u01/app/oracle/product/11.0/db_1/dbs
[oracle@apps_rac01 dbs]$ scp initvisr12.ora initvisr12.ora.bkp
-- - edit the local pfile in old ORACLE_HOME and provide the location of spfile.

[oracle@apps_rac01 dbs]$ cat initvisr12.ora
spfile='+DATA/spfile/spfilevisr12.ora'
-- Startup the DB, it will now use the SPFILE from ASM Disk to start.
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:44:59 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup 
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             536873088 bytes
Database Buffers          520093696 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL > Show parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/11.0/db_1/dbs/spfilevisr12.ora                                                      

Here it should have displayed the spfile from +DATA disk group but it was showing as from cooked file system.
The reason being is that, I already had an spfile under dbs on old Oracle Home. So I just renamed it and start the DB with pfile option and forcing DB to use spfile from +DATA Disk group as follows
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@apps_rac01 dbs]$ mv  spfilevisr12.ora  spfilevisr12.ora.bkp
[oracle@apps_rac01 dbs]$ !sqlp
sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 31 08:51:06 2012
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup pfile='initvisr12.ora'
ORACLE instance started.
Total System Global Area 1071333376 bytes
Fixed Size                  1341312 bytes
Variable Size             490735744 bytes
Database Buffers          566231040 bytes
Redo Buffers               13025280 bytes
Database mounted.
Database opened.
SQL> sho parameter spfile
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/spfile/spfilevisr12.ora

So far so good. Now comes the most interesting part of the process i.e. Converting the Non RAC database to RAC database. There are several ways we can achieve this 
1. DBCA
2. rconfig
3. Oracle Enterprise Manager
4. Manually (old school way )

For this post, I am going to use rconfig, as its the more sophisticated way and less error prone to OEM. 
So first of all you have copy the convert template from default directory to your custom directory. Since I want to convert this DB as an Admin manged RAC DB, i will use ConvertToRAC_AdminManaged.xml template.
[oracle@apps_rac01 sampleXMLs]$ pwd
/u01/app/oracle/product/11.0/db_1/assistants/rconfig/sampleXMLs
[oracle@apps_rac01 sampleXMLs]$ ll
total 8

-rw-r--r-- 1 oracle dba 2591 Mar  4  2009 ConvertToRAC_AdminManaged.xml

-rw-r--r-- 1 oracle dba 2653 Mar 12  2009 ConvertToRAC_PolicyManaged.xml

[oracle@apps_rac01 sampleXMLs]$ scp ConvertToRAC_AdminManaged.xml /home/oracle/



Open the ConvertToRAC_AdminManaged.xml and make following changes based on your environment.

Before starting actual conversion, it is advisable to check the conversion process beforehand. So I will run the rconfig utility with ONLY option to check whether my conversion will work flawlessly or not. 



[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>

    <Convert>

      <Response>

        <Result code="1" >

          Got Exception

        </Result>
       <ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: SCAN Listener is not running.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_23_49.log for more details.
       </ErrorDetails>
      </Response>
    </Convert>
  </ConvertToRAC></RConfig>

Since my SCAN Listeners were down, error was thrown as above. So its time to start SCAN Listener and run the utility again.

[oracle@apps_rac01 db_1]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  start scan_listener
[oracle@apps_rac01 bin]$ ./srvctl  status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node apps_rac01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node apps_rac02
<ErrorDetails>
             oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: oracle.sysman.assistants.rconfig.engine.InvalidConfigurationException: Default Listener is not configured in Grid Infrastructure Home.
Operation Failed. Refer logs at /u01/app/oracle/cfgtoollogs/rconfig/rconfig_07_31_12_09_27_28.log for more details.
   </ErrorDetails>

The issue was that my listener was running from RAC ORACLE_HOME. The default listener has to running from GRID Home. 
-- First Removed the existing listener
[oracle@apps_rac01 bin]$ cd /u02/app/oracle/prodcut/11.2.0/db_1/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: /u02/app/oracle/prodcut/11.2.0/db_1
End points: TCP:1521
[oracle@apps_rac01 bin]$ srvctl  stop listener
[oracle@apps_rac01 bin]$ srvctl remove listener -l listener_visr12
[oracle@apps_rac01 bin]$ cd -
/u02/app/11.2.0/grid/bin

-- Create new listener from Grid Home
[oracle@apps_rac01 bin]$ ./srvctl  add  listener -l listener_visr12 -o /u02/app/11.2.0/grid -p 1521
[oracle@apps_rac01 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521
[oracle@apps_rac01 bin]$ ssh apps_rac02
Last login: Tue Jul 31 02:54:28 2012 from apps_rac01.localdomain
[oracle@apps_rac02 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac02 bin]$ ./srvctl  config listener
Name: LISTENER_VISR12
Network: 1, Owner: oracle
Home: <CRS Home>
End points: TCP:1521

-- However after creating listener it still failed. After googling a bit, I found workaround as follows.
[oracle@apps_rac01 bin]$ pwd
/u02/app/11.2.0/grid/bin
[oracle@apps_rac01 bin]$ ./srvctl remove listener -l listener_visr12
Connection to apps_rac02 closed.
-- Remove the listener with custom name 
[oracle@apps_rac01 bin]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u02/app/11.2.0/grid is /u01/app/oracle
-- Start the NETCA from grid home create the listener with default name "LISTENER"
[oracle@apps_rac01 bin]$ netca
Oracle Net Services Configuration:
Oracle Net Configuration Assistant is launched from Grid Infrastructure home. Ne
twork configuration will be clusterwide.
Configuring Listener:LISTENER
apps_rac01...
apps_rac02...
Listener configuration complete.
Oracle Net Listener Startup:
    Listener started successfully.
Oracle Net Services configuration successful. The exit code is 0
[oracle@apps_rac01 bin]$ ./srvctl  status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): apps_rac01,apps_rac02

With this RCONFIG went ahead, just to give another error. 
 <ErrorDetails>
             /u01/app/oracle/product/11.0/db_1/dbs/arch LOG ARCHIVE DEST does not exist on all nodes in the cluster
       </ErrorDetails>

Workaround was pretty simple this time and just created the arch directory on both nodes and there I was done.
RCONFIG didn't complain this time.
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
There is no return value for this step     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

Being said that, it's time to run RCONFIG with YES option to do the actual conversion and here I go 
[oracle@apps_rac01 ~]$ rconfig ConvertToRAC_AdminManaged.xml 
Converting Database "visr12" to Cluster Database. Target Oracle Home: /u02/app/oracle/prodcut/11.2.0/db_1. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace

Adding UNDO tablespaces

Adding Trace files

Setting Flash Recovery Area

Updating Oratab

Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
    <Convert>
      <Response>
        <Result code="0" >
          Operation Succeeded
        </Result>
      </Response>
      <ReturnValue type="object">
<Oracle_Home>
         /u02/app/oracle/prodcut/11.2.0/db_1
       </Oracle_Home>
       <Database type="ADMIN_MANAGED"  >
 <InstanceList>
           <Instance SID="visr121" Node="apps_rac01"  >
           </Instance>
           <Instance SID="visr122" Node="apps_rac02"  >
           </Instance>
         </InstanceList>
       </Database>     </ReturnValue>
    </Convert>
  </ConvertToRAC></RConfig>

[oracle@apps_rac01 ~]$ cd /u02/app/11.2.0/grid/bin/
[oracle@apps_rac01 bin]$ ./srvctl  config database -d visr12
Database unique name: visr12
Database name: visr12

Oracle home: /u02/app/oracle/prodcut/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/spfilevisr12.ora

Domain: 

Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: visr12
Database instances: visr121,visr122
Disk Groups: DATA
Services: 
Database is administrator managed

-- Copy appsutil directory from source ORACLE HOME to destination ORACLE HOME (on both nodes)
[oracle@apps_rac01 bin]$ scp -r appsutil /u02/app/oracle/prodcut/11.2.0/db_1/
[oracle@apps_rac01 bin]$ scp -r appsutil apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/

-- Copy network/admin folder from source ORACLE HOME to destination ORACLE HOME  (on both nodes)
-- Change the path of ORACLE HOME from old to new (RAC) and modify the INSTANCE_NAME parameter on the respective nodes
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ /u02/app/oracle/prodcut/11.2.0/db_1/network/admin/
[oracle@apps_rac01 bin]$ scp -r visr12_apps_rac01/ apps_rac02:/u02/app/oracle/prodcut/11.2.0/db_1/network/admin/visr12_apps_rac02/

-- You may also have to add Alias using NETCA (Run from GI Home)
Node 1 - 
[oracle@apps_rac01 bin]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 13:47:16
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 13:45:50
Uptime                    0 days 0 hr. 1 min. 25 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.71)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.72)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).

  Instance "visr121", status READY, has 1 handler(s) for this service...

The command completed successfully

Node 2 - 
[oracle@apps_rac02 visr12_apps_rac02]$ lsnrctl status listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 31-JUL-2012 08:46:09
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                31-JUL-2012 04:32:38
Uptime                    0 days 4 hr. 13 min. 33 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/apps_rac02/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.73)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.168.0.74)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "visr12" has 1 instance(s).
  Instance "visr122", status READY, has 1 handler(s) for this service...
The command completed successfully

-- Create context file with adbldxml.pl utility on both DB Nodes


[oracle@apps_rac01 bin]$ hostname
apps_rac01.localdomain
[oracle@apps_rac01 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac01 bin]$ perl adbldxml.pl appsuser=apps appspass=apps

The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07311354.log
AC-20010: Error: File - listener.ora could not be found at the location: 
        /listener.ora 
indicated by TNS_ADMIN. Context file can not be generated.
Could not Connect to the Database with the above parameters, Please answer the Questions below
Enter Hostname of Database server: 
Enter Hostname of Database server: apps_rac01            
Enter Port of Database server: 1521

Enter SID of Database server[visr12]:visr121
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml



[oracle@apps_rac02 bin]$ hostname 
apps_rac02.localdomain
[oracle@apps_rac02 bin]$ pwd
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/bin

[oracle@apps_rac02 bin]$ perl ./adbldxml.pl appsuser=apps appspass=apps
Starting context file generation for db tier..
Using JVM from /u02/app/oracle/prodcut/11.2.0/db_1/jdk/jre/bin/java to execute java programs..
The log file for this adbldxml session is located at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/adbldxml_07310853.log

AC-20010: Error: File - listener.ora could not be found at the location: 

        /listener.ora 

indicated by TNS_ADMIN. Context file can not be generated.

Could not Connect to the Database with the above parameters, Please answer the Questions below

Enter Hostname of Database server: apps_rac02
Enter Port of Database server: 1521
Enter SID of Database server: visr122
Enter the value for Display Variable: 0
The context file has been created at:
/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml

-- Make sure your LOCAL_LISTENER are registered with DB 
Node 1 - 


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr121
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------

listener_networks                    string

local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD

                                                 DRESS=(PROTOCOL=TCP)(HOST=apps

                                                 _rac01-vip)(PORT=1521))))

remote_listener                      string      apps-scan.localdomain:1521

Node 2 - 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
instance_name                        string      visr122
SQL> sho parameter listener
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
listener_networks                    string
local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD
                                                 DRESS=(PROTOCOL=TCP)(HOST=apps
                                                 _rac02-vip)(PORT=1521))))
remote_listener                      string      apps-scan.localdomain:1521


-- Run AutoConfig on both nodes 
Node 1 - 
[oracle@apps_rac01 bin]$ ./adconfig.sh contextfile=../visr121_apps_rac01.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr121_apps_rac01/07311404/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr121_apps_rac01.xml
Context Value Management will now update the Context file
        Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

Node 2 - 
[oracle@apps_rac02 bin]$ ./adconfig.sh contextfile=../visr122_apps_rac02.xml 
Enter the APPS user password: 
The log file for this session is located at: /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/log/visr122_apps_rac02/07310903/adconfig.log
AutoConfig is configuring the Database environment...
AutoConfig will consider the custom templates if present.
        Using ORACLE_HOME location : /u02/app/oracle/prodcut/11.2.0/db_1
        Classpath                   : :/u02/app/oracle/prodcut/11.2.0/db_1/jdbc/lib/ojdbc5.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java/xmlparserv2.jar:/u02/app/oracle/prodcut/11.2.0/db_1/appsutil/java:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/netcfg.jar:/u02/app/oracle/prodcut/11.2.0/db_1/jlib/ldapjclnt11.jar
        Using Context file          : /u02/app/oracle/prodcut/11.2.0/db_1/appsutil/visr122_apps_rac02.xml
Context Value Management will now update the Context file
  Updating Context file...COMPLETED
        Attempting upload of Context file and templates to database...COMPLETED
Updating rdbms version in Context file to db112
Updating rdbms type in Context file to 32 bits
Configuring templates from ORACLE_HOME ...
AutoConfig completed successfully.

-- In DB you can verify both the nodes 
SQL> select node_name from apps.fnd_nodes;
NODE_NAME
------------------------------
APPS_RAC01
APPS_RAC02

-- Check and set the DB Env for TNS_ADMIN
Node 1 - 
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
[oracle@apps_rac01 bin]$ ./srvctl setenv database -d visr12 -T TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
[oracle@apps_rac01 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin
Node 2 - 
[oracle@apps_rac02 bin]$ cd ../../bin/
[oracle@apps_rac02 bin]$ ./srvctl  getenv database -d visr12
visr12:
TNS_ADMIN=/u02/app/oracle/prodcut/11.2.0/db_1/network/admin

-- Modify the CONTEXT FILE on apps tier & add additional node info 
[oracle@appsnode_new appl]$ vi  $CONTEXT_FILE
 <jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac01.localdomain)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=apps_rac02.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=visr12))))</jdbc_url>

-- Run AutoConfig on Apps tier
[oracle@appsnode_new appl]$ adautocfg.sh 
Enter the APPS user password:
The log file for this session is located at: /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/admin/log/07312029/adconfig.log
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
        Using CONFIG_HOME location     : /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new
Classpath                   : /u01/app/oracle/visr12/apps/apps_st/comn/java/lib/appsborg2.zip:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes
     Configuring IES_TOP.......COMPLETED
        Configuring CSD_TOP.......COMPLETED
        Configuring IGC_TOP.......COMPLETED
AutoConfig completed successfully.

-- Start the Apps Tier Services 
[oracle@appsnode_new appl]$ adstrtal.sh apps/apps
You are running adstrtal.sh version 120.15.12010000.3
The logfile for this session is located at /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/admin/log/adstrtal.log
Executing service control script:

Now log-in to apps console and navigate to system administrator responsibility -> OAM -> Database status. You can see now both the nodes available and active as follows.

Set Up Load Balancing On Apps Tier

Now its time to implement load balancing for the Oracle Applications database connections...

Run the Context Editor (through the Oracle Applications Manager interface) and set the value of "Tools OH TWO_TASK" (s_tools_two_task), "iAS OH TWO_TASK" (s_weboh_twotask) and "Apps JDBC Connect Alias" (s_apps_jdbc_connect_alias). 

To load balance the forms based applications database connections, set the value of "Tools OH TWO_TASK" to point to the <database_name>_balance alias generated in the tnsnames.ora file.

To load balance the self-service applications database connections, set the value of "iAS OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in the tnsnames.ora file.


Execute AutoConfig by running the command
[oracle@appsnode_new appl]$ adautocfg.sh

Restart the Applications processes, using the new scripts generated by AutoConfig.
Ensure that value of the profile option "Application Database ID" is set to dbc file name generated in $FND_SECURE. 

Hope that this post will help you in your Apps to RAC migration successfully. Comments are welcome.