Patch Oracle EBS/Apps R12 from 12.1.1 to 12.1.2

As promised earlier, I recently applied the patch EBS R12, 12.1.1 to 12.1.2. 
In this post, I am going to share my experience which may help you to plan your upgrade.
Preparation - 
To patch EBS/Apps from 12.1.1 to 12.1.2, the required patch is 7303033. However before one can apply this patch there is a prerequisite patch R12.AD.B.DELTA.2 i.e. 8502056,  that one needs to apply.

So ultimately it boils down to the following patches  

     1. Apply Pre-requisite patch R12.AD.B.DELTA.2 i.e. 8502056

     2. Apply 12.1.2 RUP 2 i.e. 7303033

     3.Apply the latest consolidated online help Patch 7303032(Optional)

     4. Post Patch Process

However, before we start there are two pre-checks one needs to perform as follows.

Pre-check 1 - 

The minimum required version of OracleAS 10g should be 10.1.3.4

(R12.1.1 comes with 10.1.3.4)

[root@appsnode appl]# sudo su - oracle

[oracle@appsnode ~]$ cd /u01/app/oracle/visr12/apps/apps_st/appl/

[oracle@appsnode appl]$ . ./APPSvisr12_appsnode.env

[oracle@appsnode appl]$  cat $CONTEXT_FILE  | grep ias

         <ias_version oa_var="s_ias_version">10.1.3.4.0</ias_version>

 Pre-check 2 - 

 The minimum required version of OracleAS 10g Forms and Reports should be 10.1.2.3.0 

(R12.1.1 comes with 10.1.3.3)

[oracle@appsnode appl]$  cat $CONTEXT_FILE  | grep forms_version

         <forms_version oa_var="s_forms_version">10.1.2.3.0</forms_version>

1. Apply Pre-requisite patch R12.AD.B.DELTA.2 i.e. 8502056

Execute the following command to generate your instance specific installation instructions

1. Source the Applications environment file

2. Run the Patch Application Assistant by entering "admsi.pl".

Patch Location - /home/oracle/R1212/8502056

Pre-install Tasks -

You must shut down all Application tier services before you start the patch application

Instructions for running AD Grants

Run the adgrants.sql script as a user that can connect as SYSDBA to grant privileges to selected SYS objects and create PL/SQL profiler objects.

Usage:

  1. Create $ORACLE_HOME/appsutil/admin on the database server.

  2. Copy adgrants.sql (UNIX) from this patch directory to

     $ORACLE_HOME/appsutil/admin.

  3. Set the environment to point to ORACLE_HOME on the database server.

  4. Use SQL*Plus to run the script:

     $ sqlplus /nolog

     SQL> @$ORACLE_HOME/appsutil/admin/adgrants.sql apps

-- Stop the application tier

[oracle@appsnode appl]$ . ./APPSvisr12_appsnode.env

[oracle@appsnode appl]$  cd $ADMIN_SCRIPTS_HOME

[oracle@appsnode scripts]$sh adstpall.sh apps/apps

-- Enable Maintenance Mode

[oracle@appsnode 8502056]$

adadmin defaultsfile=defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt    logfile=adadmin.log

       AD Administration Main Menu

   --------------------------------------------------

   1.    Generate Applications Files menu

   2.    Maintain Applications Files menu

   3.    Compile/Reload Applications Database Entities menu

   4.    Maintain Applications Database Entities menu

   5.    Change Maintenance Mode

   6.    Exit AD Administration                                                                                                              

Connecting to APPS......Connected successfully.

Connecting to APPS......Connected successfully.

             Change Maintenance Mode

   ----------------------------------------

Maintenance Mode is currently: [Disabled].

Maintenance mode should normally be enabled when patching

Oracle Applications and disabled when users are logged on

to the system.  See the Oracle Applications Maintenance

Utilities manual for more information about maintenance mode.

Please select an option:

   1.    Enable Maintenance Mode

   2.    Disable Maintenance Mode

   3.    Return to Main Menu

Enter your choice [3] : 1

PL/SQL procedure successfully completed.

Commit complete.

Successfully enabled Maintenance Mode.

       

-- Apply patch

This patch contains the following unified driver file to be applied with AutoPatch:

    u8502056.drv (This is the unified driver)

-- This will take 30-60 minutes

[oracle@appsnode 8502056]$

adpatch defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt  logfile=/home/oracle/patchapply_8502056.log driver=u8502056.drv  workers=8

2. Apply 12.1.2 RUP 2 i.e. 7303033

Execute the following command to generate your instance specific installation instructions

1. Source the Applications environment file

2. Run the Patch Application Assistant by entering "admsi.pl"

-- Apply patch

This patch contains the following unified driver file to be applied with AutoPatch:  u7303033.drv

-- This patch took around 30 hours for me as it failed twice and my system was running slow.

[oracle@appsnode 7303033]$

adpatch defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt  logfile=patch_apply_7303033.log driver=u7303033.drv  workers=4

Issues - 

Issue 1 - 

Patch Hung after few hours

After approx 8 hours into the patch, patch was stuck with product AR and was not progressing at all. Hence I took a look with adctrl and found that worker 1 was stalled. I decided to stop worker and restart the failed job but it didn't improve the situation so I have to manually stop the application of the patch and re-initiate it again.

Issue 2- 

Few fmx files failed to compile

When got the warning for failed execution, I choose to Yes and continued with the patch and then manually generated the plls/fmx files using adadmin to resolve the  issue.

Issue 3 - 

AutoPatch error:

The following ORACLE error:

ORA-03113: end-of-file on communication channel

occurred while executing the SQL statement:

SELECT   control_code, status FROM     fnd_install_processes ORDER BY worker_id

Telling workers to quit...

I checked the database following this error and found it went down without any apparent error. I restarted the DB and re-initiated the patch  

Post Patch Process

Update database tier nodes with the Release 12.1.2 code level.

After you have upgraded to Release 12.1.2, perform the following post-update steps to update all database tier nodes with the code level, provided by Release 12.1.2:

-- Run the admkappsutil.pl utility to create the file appsutil.zip in the <INST_TOP>/admin/out directory.

[oracle@appsnode /]$perl $AD_TOP/bin/admkappsutil.pl

[oracle@appsnode ~]$ cd /u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/out/

[oracle@appsnode out]$ ll

-rw-r--r-- 1 oracle oinstall 2745685 May 25 07:46 appsutil.zip

-- Copy this appsutil.zip file to $ORACLE_HOME/appsutil 

[oracle@appsnode out]$scp appsutil.zip  appsdbnode:/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil

[oracle@appsdbnode 11.1.0]$ mv appsutil appsutil.old

[oracle@appsdbnode 11.1.0]$ unzip -o appsutil.zip

-- Run AutoConfig on the <RDBMS ORACLE_HOME>.

 [oracle@appsdbnode visr12_appsdbnode]$ pwd

/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/scripts/visr12_appsdbnode

[oracle@appsdbnode visr12_appsdbnode]$ ./adautocfg.sh apps/apps

Database tier and application tier:

Re-run adpreclone.pl on the database tier and the application tier. 

[oracle@appsdbnode visr12_appsdbnode]$ perl ./adpreclone.pl dbTier

                     Copyright (c) 2002 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle Applications Rapid Clone

                                 Version 12.0.0

                      adpreclone Version 120.20.12010000.2

Enter the APPS User Password:

Running:

perl /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/bin/adclone.pl java=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/jre mode=stage stage=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/clone component=dbTier method=CUSTOM dbctx=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/visr12_appsdbnode.xml showProgress

APPS Password :

Beginning database tier Stage - Thu May 31 10:42:50 2012

/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=false  -Doracle.installer.oui_loc=/u01/app/oracle/visr12/db/tech_st/11.1.0/oui -classpath /u01/app/oracle/visr12/db/tech_st/11.1.0/lib/xmlparserv2.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/jdbc/lib/ojdbc6.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/java:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/ewt3.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/share.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/srvm.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/jlib/ojmisc.jar   oracle.apps.ad.clone.StageDBTier -e /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/visr12_appsdbnode.xml -stage /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/clone -tmp /tmp -method CUSTOM    -showProgress

APPS Password :

Log file located at /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/log/visr12_appsdbnode/StageDBTier_05311042.log

  \     50% completed

Completed Stage...

[oracle@appsnode scripts]$ perl ./adpreclone.pl  appsTier

                     Copyright (c) 2002 Oracle Corporation

                        Redwood Shores, California, USA

                        Oracle Applications Rapid Clone

                                Version 12.0.0

                      adpreclone Version 120.20.12010000.5

Running:

perl /u01/app/oracle/visr12/apps/apps_st/appl/ad/12.0.0/bin/adclone.pl java=/u01/app/oracle/visr12/apps/tech_st/10.1.3/appsutil/jdk mode=stage stage=/u01/app/oracle/visr12/apps/apps_st/comn/clone component=appsTier method= appctx=/u01/app/oracle/visr12/inst/apps/visr12_appsnode/appl/admin/visr12_appsnode.xml showProgress

APPS Password :

method defaulted to CUSTOM

Beginning application tier Stage - Thu May 31 05:06:42 2012

/u01/app/oracle/visr12/apps/tech_st/10.1.3/appsutil/jdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false  -Doracle.installer.oui_loc=/oui -classpath /u01/app/oracle/visr12/apps/tech_st/10.1.3/lib/xmlparserv2.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/jdbc/lib/ojdbc14.jar:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/OraInstaller.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/ewt3.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/share.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/srvm.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/jlib/ojmisc.jar  oracle.apps.ad.clone.StageAppsTier -e /u01/app/oracle/visr12/inst/apps/visr12_appsnode/appl/admin/visr12_appsnode.xml -stage /u01/app/oracle/visr12/apps/apps_st/comn/clone -tmp /tmp -method CUSTOM   -showProgress

Log file located at /u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/log/StageAppsTier_05310506.log

 /     20% completed

Completed Stage...

-- Disable the maintenance mode on Apps tier

[oracle@appsnode 8502056]$

adadmin defaultsfile=defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt    logfile=adadmin.log

       AD Administration Main Menu

   --------------------------------------------------

   1.    Generate Applications Files menu

   2.    Maintain Applications Files menu

   3.    Compile/Reload Applications Database Entities menu

   4.    Maintain Applications Database Entities menu

   5.    Change Maintenance Mode

   6.    Exit AD Administration

                                                                                                              

Connecting to APPS......Connected successfully.

Connecting to APPS......Connected successfully.

             Change Maintenance Mode

   ----------------------------------------

Maintenance Mode is currently: [Enabled].

Maintenance mode should normally be enabled when patching Oracle Applications and disabled when users are logged on to the system.  See the Oracle Applications Maintenance Utilities manual for more information about maintenance mode.

Please select an option:

   1.    Enable Maintenance Mode

   2.    Disable Maintenance Mode

   3.    Return to Main Menu

Enter your choice [3] : 2

PL/SQL procedure successfully completed.

Commit complete.

Successfully disabled Maintenance Mode.

Start the Application Tier - 

-- Start the services on Apps tier - 

[oracle@appsnode scripts]$ pwd

/u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/scripts

[oracle@appsnode scripts]$ ./adstrtal.sh apps/apps

[oracle@appsnode scripts]$ ./adopmnctl.sh  status

You are running adopmnctl.sh version 120.6.12010000.3

Checking status of OPMN managed processes...

Processes in Instance: visr12_appsnode.appsnode.localdomain

---------------------------------+--------------------+---------+---------

ias-component                              | process-type       |     pid   | status

---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm          |   16264 | Alive

OC4JGroup:default_group          | OC4J:forms         |   16176 | Alive

OC4JGroup:default_group          | OC4J:oacore        |   16095 | Alive

HTTP_Server                                   | HTTP_Server        |   16043 | Alive

Check the Version from Back End - 

[oracle@appsdbnode ~]$ cd /u01/app/oracle/visr12/db/tech_st/11.1.0/

[oracle@appsdbnode 11.1.0]$ . ./visr12_appsdbnode.env

[oracle@appsdbnode 11.1.0]$ sqlplus apps/apps

SQL> select RELEASE_NAME from fnd_product_groups;

RELEASE_NAME

--------------------------------------------------

12.1.2

You just have patch Oracle EBS/Apps R12 to new code level.

Next Step - Patch 12.1.2 Apps to 12.1.3