Wednesday, May 30, 2012


Apply PSU 5 for 11gR2 Grid and RAC (11.2.0.2)

Oracle released the Patch Set Update 5 for 11g Grid Infrastructure and RAC recently. This PSU contains some imp bug fixes and new enhancements so I decided to apply the latest Grid Infra Patch Set Update on 11.2.0.2( Both CRS and RAC are on same version) in my test env.

During the analysis, the important facet that catches me was, "Do Patchset Updates (PSU's) Change the Oracle Release Version/Fifth Digit?"
Answer was Big NO.

As per MOS -
The PSU does NOT change the database version . That means PSU  10.2.0.4.x /10.2.0.5.x, 11.1.0.7.x, 11.2.0.1.x /11.2.0.2.x ( where x is the fifth digit ) does NOT change the 5th digit of the Oracle Database Server version.
After applying the PSU "opatch lsinventory" still shows the version as 10.2.0.4.0/10.2.0.5.0 for Oracle 10g R2, 11.1.0.7 for Oracle 11g R1  and 11.2.0.1 /11.2.0.2 for Oracle 11g R2
PSU's also will not change the version of oracle binaries (like sqlplus, exp/imp etc.)
Note : It is also applicable for the Oracle Enterprise Manager Grid Control and EM Agent 


In-fact it was the supposed to be my last note in this post but turned out to be the first one!!


Its time, I run you thru the actual process step by step


Noes - racnode2, racnode2
GI Home - 112020
RAC Home - 11202
ASM - Yes


Another imp note is that when you apply PSU, you can choose to either keep HAS Stack up or down, I actually choose to keep the stack down to avoid any unpleasant situation that may arise...

[root@racnode1 ~]# cd /u01/app/11.2.0.2/grid/bin/
[root@racnode1 bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'

[root@racnode2 ~]# cd /u01/app/11.2.0.2/grid/bin/
[root@racnode2 bin]#  ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode2'


Pre-requisite : 


Apply latest opatch patch on RAC nodes (OPatch utility version should be 11.2.0.1.5 or later or otherwise PSU application will fail)
Patch number - p6880880_112000_LINUX.zip


ON NODE1 - 


[root@racnode1 grid]# pwd
/u01/app/11.2.0.2/grid
[root@racnode1 grid]# sudo su - oracle
[oracle@racnode1 ~]$ cd /u01/app/11.2.0.2/grid
[oracle@racnode1 OPatch]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/11.2.0.2/grid is /u01/app/oracle

[oracle@racnode1 OPatch]$ ./opatch lsinventory
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/11.2.0.2/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1                                -- Version Before OPatch patch was applied 
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/11.2.0.2/grid/oui
Log file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch2012-05-22_12-09-22PM.log
Patch history file: /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-05-22_12-09-22PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure                                           11.2.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
  Local node = racnode1
  Remote node = racnode2
--------------------------------------------------------------------------------
OPatch succeeded.



ON NODE2 - 


[root@racnode2 bin]# sudo su - oracle
[oracle@racnode2 ~]$ cd /u01/app/11.2.0.2/grid/OPatch/
[oracle@racnode2 OPatch]$ . oraenv
ORACLE_SID = [wiustg2] ? +ASM2
The Oracle base for ORACLE_HOME=/u01/app/11.2.0.2/grid is /u01/app/oracle

[oracle@racnode2 OPatch]$ ./opatch lsinventory
Invoking OPatch 11.2.0.1.1
Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/11.2.0.2/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1                           -- Version Before OPatch patch was applied 
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/11.2.0.2/grid/oui
Log file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch2012-05-22_12-10-59PM.log
Patch history file: /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch_history.txt
Lsinventory Output file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-05-22_12-10-59PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure                                           11.2.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
  Local node = racnode2
  Remote node = racnode1
--------------------------------------------------------------------------------
OPatch succeeded.


ON NODE1 - 

[oracle@racnode1 OPatch]$ echo $ORACLE_HOME
/u01/app/11.2.0.2/grid
[oracle@racnode1 OPatch]$ unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip  -d /u01/app/11.2.0.2/grid
Archive:  /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
   creating: /u01/app/11.2.0.2/grid/OPatch/oplan/
  inflating: /u01/app/11.2.0.2/grid/OPatch/oplan/README.html
  inflating: /u01/app/11.2.0.2/grid/OPatch/oplan/README.txt
replace /u01/app/11.2.0.2/grid/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
  inflating: /u01/app/11.2.0.2/grid/OPatch/docs/FAQ
[oracle@racnode1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/11.2.0.2/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0.2/grid/oraInst.loc
OPatch version    : 11.2.0.3.0                   -- Version after OPatch patch applied
OUI version       : 11.2.0.2.0



[root@racnode1 ~]# unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip  -d /u01/app/oracle/product/11.2.0.2/dbhome_1
Archive:  /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
   creating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/
  inflating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/README.html
replace /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A


ON NODE2 - 
[oracle@racnode2 OPatch]$ echo $ORACLE_HOME
/u01/app/11.2.0.2/grid
[oracle@racnode2 OPatch]$ unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip  -d /u01/app/11.2.0.2/grid
Archive:  /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
   creating: /u01/app/11.2.0.2/grid/OPatch/oplan/

[oracle@racnode2 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/11.2.0.2/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0.2/grid/oraInst.loc
OPatch version    : 11.2.0.3.0                 -- Version after OPatch patch applied
OUI version       : 11.2.0.2.0
One-off Patch Conflict Detection and Resolution.
opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./13653086
export PATH=$PATH:<GI_HOME>/OPatch
opatch auto <UNZIPPED_PATCH_LOCATION>
opatch auto <UNZIPPED_PATCH_LOCATION> -oh <GI_HOME>
opatch auto <UNZIPPED_PATCH_LOCATION> -oh <ORACLE_HOME>, <ORACLE_HOME2>

Patch  13653086     : applied on Tue May 22 14:15:01 IST 2012
Unique Patch ID:  14538982
Patch description:  "Grid Infrastructure Patch Set Update 11.2.0.2.5 (includes DB PSU 11.2.0.2.5) #2"
   Created on 2 Feb 2012, 02:06:04 hrs PST8PDT


[root@racnode2 ~]# unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip  -d /u01/app/oracle/product/11.2.0.2/dbhome_1
Archive:  /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
   creating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/
  inflating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/README.html
replace /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A



NOTE - 
The fastest and easiest way to determine whether you have one-off patches in the Oracle home that conflict with the PSU, and to get the necessary conflict resolution patches, is to use the Patch Recommendations and Patch Plans features on the Patches & Updates tab in My Oracle Support. 

Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

Generate ocm.rsp file - 


Since we apply the patch in silent mode, one needs to create ocm.rsp file which will be required during patch application. Generate the ocm.rsp as follows

[root@racnode1 11.2.0.2]# export PATH=$PATH:/u01/app/11.2.0.2/grid/OPatch

-- To generate ocm.rsp file
#$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /home/oracle/ocm.rsp
The OCM configuration response file (/home/oracle/ocm.rsp) was successfully created.

-- To Patch GI Home
The utility must be executed by an operating system (OS) user with root privileges (usually the user root), and it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage. 
The utility should not be run in parallel on the cluster nodes.


Depending on command line options specified, one invocation of Opatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC or Oracle RAC One Node database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.

Add the directory containing the opatch to the $PATH environment variable.
You can choose following methods to apply patches selectively to different Homes at same time or separately.


1. To patch GI home and all Oracle RAC  database homes of the same version at same time
2. To patch only the GI home
3. To patch one or more Oracle RAC  database homes


I choose to apply patch to GI Home and RAC Home separately to have better control and easy of troubleshooting.


Applying Patch on GI Home - 


ON NODE1  -

[root@racnode1 112025_GI_PSUPatch]# opatch auto /home/oracle/112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid


Executing /usr/bin/perl /u01/app/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/app/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_14-01-14.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Successfully unlock /u01/app/11.2.0.2/grid
patch /home/oracle/112025_GI_PSUPatch/13653086  apply successful for home  /u01/app/11.2.0.2/grid
patch /home/oracle/112025_GI_PSUPatch/13343424  apply successful for home  /u01/app/11.2.0.2/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.


ON NODE2 - 


[root@racnode2 112025_GI_PSUPatch]# cd /u01/app/11.2.0.2/grid/OPatch/                         
[root@racnode2 OPatch]# opatch auto /home/oracle/112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid                                                                                               


Executing /usr/bin/perl /u01/app/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params                                                                           opatch auto log file location is /u01/app/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_14-33-40.log                                                                        
Detected Oracle Clusterware install                                                           
 Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params        
OPatch  is bundled with OCM, Enter the absolute OCM response file path: /home/oracle/ocm.rsp                                                        
Successfully unlock /u01/app/11.2.0.2/grid                                                                                                                                                    patch /home/oracle/112025_GI_PSUPatch/13653086  apply successful for home  /u01/app/11.2.0.2/grid                                                                                             patch /home/oracle/112025_GI_PSUPatch/13343424  apply successful for home  /u01/app/11.2.0.2/grid                                                                                             ACFS-9300: ADVM/ACFS distribution files found.                                                 
ACFS-9312: Existing ADVM/ACFS installation detected.                                           
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.


Applying Patch on RAC Home - 


ON NODE1 - 
[root@racnode1 ~]# which opatch
/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/opatch


[root@racnode1]# opatch auto /home/oracle/112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1

Executing /usr/bin/perl ./crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1 -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_16-13-57.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
patch /home/oracle/112025_GI_PSUPatch/13653086/custom/server/13653086  apply successful for home  /u01/app/oracle/product/11.2.0.2/dbhome_1
patch /home/oracle/112025_GI_PSUPatch/13343424  apply successful for home  /u01/app/oracle/product/11.2.0.2/dbhome_1

ON NODE2 - 
[root@racnode2 ~]# export PATH=$PATH:/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch
[root@racnode2 ~]# which opatch
/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/opatch

[root@racnode2]# opatch auto /home/oracle/112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1

Executing /usr/bin/perl /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1 -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_15-38-25.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
patch /home/oracle/112025_GI_PSUPatch/13653086/custom/server/13653086  apply successful for home  /u01/app/oracle/product/11.2.0.2/dbhome_1

Check Inventory on Both Nodes - 

[oracle@racnode1 OPatch]$ ./opatch lsinventory -detail 
Oracle Home       : /u01/app/11.2.0.2/grid
Central Inventory : /u01/app/oraInventory
Interim patches (2) :
Patch  13343424     : applied on Tue May 22 14:20:00 IST 2012
Unique Patch ID:  14538982
Patch description:  "Database Patch Set Update : 11.2.0.2.5 (13343424)"


[oracle@racnode2 OPatch]$ ./opatch lsinventory -bugs_fixed | egrep 'PSU|PATCH SET UPDATE'
13343447   13653086  Tue May 22 14:47:17 IST 2012   GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU 11.2.0.2.
13396479   13653086  Tue May 22 14:47:17 IST 2012   11.2.0.2 GI PSU 5 HAS-CRS TRACKING BUG
13396487   13653086  Tue May 22 14:47:17 IST 2012   11.2.0.2 GI PSU5 USM TRACKING BUG
                                                    JAN 2012 PSU
13653086   13653086  Tue May 22 14:47:17 IST 2012   GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.2.5
                                                    (INCLUDES DB PSU 11.2.0.2.5)



As you can now see that the PSU is reflecting in inventory properly. 


Post Patch Steps - 


Start your database/instances/services and check that they are running fine...