Error During 11.2.0.2 Rolling Upgrade

Error- 
Creating trace directory
Failed to add (property/value):('OLD_OCR_ID/'-1') for checkpoint:ROOTCRS_OLDHOMEINFO.Error code is 256
The fixes for bug 9413827 are not present in the 11.2.0.1 crs home
Apply the patches for these bugs in the 11.2.0.1 crs home and then run rootupgrade.sh
/u03/app/11.2.0.2/grid/perl/bin/perl -I/u03/app/11.2.0.2/grid/perl/lib -I/u03/app/11.2.0.2/grid/crs/install /u03/app/11.2.0.2/grid/crs/install/rootcrs.pl execution failed

Though it was indicating that it needs to have 9413827 patch, but is actually included in 655006 which turns out to be the 11.2.0.1.2 PSU. The download file is

p9655006_112010_Linux-x86-64.zip

Bug 9413827 - CRS rolling upgrade from 11.2.0.1 to 11.2.0.2 fails with OCR on ASM [ID 9413827.8]

One imp thing is worth mentioning that if  If "Patch 9655006 - 11.2.0.1.2 for Grid Infrastructure (GI) Patch Set Update" is applied on the GI home, then applying 9413827 or 9706490 (as per ID 1274629.1) will always try to rollback the 9655006. However, what I figured out is that if you already have 9655006 aka 11.2.0.1.2 PSU applied on GI and RAC then no need to applied the 9413827 or 9706490.

Before applying the 9655006 one has to upgrade the OPatch to 11.2.0.3 version as its the pre-req patch
One can apply patch p6880880 on all RAC Nodes before apply PSU.

On Node 1 - 
-- apply p9655006 patch on grid home
[oracle@apps_rac01 Oracle11g]$ export PATH=$PATH:/u02/app/11.2.0/grid/OPatch
[oracle@apps_rac01 Oracle11g]$ which opatch
/u02/app/11.2.0/grid/OPatch/opatch

[root@apps_rac01 ~]# opatch auto /home/oracle/Oracle11g -och /u02/app/11.2.0/grid
Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn Oraclers/install/crsconfig_params
opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_1
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Unable to determine if /u02/app/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u02/app/11.2.0/grid
patch /home/oracle/Oracle11g/9655006  apply successful for home  /u02/app/11.2.0/grid
patch /home/oracle/Oracle11g/9654983  apply successful for home  /u02/app/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
-- apply p9655006 patch on RAC home 
[oracle@apps_rac01 Oracle11g]$ sudo su -
[root@apps_rac01 ~]#  export PATH=$PATH:/u02/app/oracle/prodcut/11.2.0/db_1/OPatch
[root@apps_rac01 ~]# which opatch/u02/app/oracle/prodcut/11.2.0/db_1/OPatch/opatch
[root@apps_rac01 ~]# opatch version
Oracle Home is not set. OPatch cannot proceed!
OPatch returns with error code = 1
[root@apps_rac01 ~]# export ORACLE_HOME=/u02/app/oracle/prodcut/11.2.0/db_1
[root@apps_rac01 ~]# opatch version
OPatch Version: 11.2.0.3.0
OPatch succeeded.
[root@apps_rac01 Oracle11g]# opatch auto /home/oracle/Oracle11g -oh /u02/app/oracle/prodcut/11.2.0/db_1
Executing /usr/bin/perl /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn Oracle11g -oh /u02/app/oracle/prodcut/11.2.0/db_1 -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_13-23-45.log
Detected Oracle Clusterware install
Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Unable to determine if /u02/app/oracle/prodcut/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch /home/oracle/Oracle11g/9655006/custom/server/9655006  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1
patch /home/oracle/Oracle11g/9654983  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1

On Node 2 - 

-- apply p9655006 patch on grid home

[root@apps_rac02 ~]# export PATH=$PATH:/u02/app/11.2.0/grid/OPatch

[root@apps_rac02 ~]# which opatch

/u02/app/11.2.0/grid/OPatch/opatch

[root@apps_rac02 stage]# opatch auto /home/oracle/stage -och /u02/app/11.2.0/grid

Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn stage -och /u02/app/11.2.0/grid -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params

opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_08-02-58.log

Detected Oracle Clusterware install

Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params

OPatch  is bundled with OCM, Enter the absolute OCM response file path:

/home/oracle/ocm.rsp

Unable to determine if /u02/app/11.2.0/grid is shared oracle home

Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes

Successfully unlock /u02/app/11.2.0/grid

patch /home/oracle/stage/9655006  apply successful for home  /u02/app/11.2.0/grid

patch /home/oracle/stage/9654983  apply successful for home  /u02/app/11.2.0/grid

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

-- apply p9655006 patch on RAC home

[root@apps_rac02 stage]#  export PATH=$PATH:/u02/app/oracle/prodcut/11.2.0/db_1/OPatch

[root@apps_rac02 stage]# export ORACLE_HOME=/u02/app/oracle/prodcut/11.2.0/db_1

[root@apps_rac02 stage]# which opatch

/u02/app/11.2.0/grid/OPatch/opatch

[root@apps_rac02 stage]# /u02/app/oracle/prodcut/11.2.0/db_1/OPatch/opatch version

OPatch Version: 11.2.0.3.0

OPatch succeeded.

[root@apps_rac02 stage]# opatch auto /home/oracle/stage -oh /u02/app/oracle/prodcut/11.2.0/db_1

Executing /usr/bin/perl /u02/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn stage -oh /u02/app/oracle/prodcut/11.2.0/db_1 -paramfile /u02/app/11.2.0/grid/crs/install/crsconfig_params

opatch auto log file location is /u02/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-08-27_08-29-26.log

Detected Oracle Clusterware install

Using configuration parameter file: /u02/app/11.2.0/grid/crs/install/crsconfig_params

OPatch  is bundled with OCM, Enter the absolute OCM response file path:

/home/oracle/ocm.rsp

(yes/no):yes

patch /home/oracle/stage/9655006/custom/server/9655006  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1

patch /home/oracle/stage/9654983  apply successful for home  /u02/app/oracle/prodcut/11.2.0/db_1

Once this patch is applied you can now safely go ahead with your remaining part of the rolling upgrade.