ORA-12547: TNS:lost contact / Process W000 died /

Process J000 died / kkjcre1p: unable to spawn jobq slave process

There are days when you can’t seem to do a thing wrong and some are days when you can’t do a thing right. 26 March 13 was also one of those later kind of days.

DBA got the page that production DB was not accessible for users to run batch jobs and we jumped on issue. The error reported was as follows

ORA-12547: TNS:lost contact

This is very generic error. After spending a bit time DBA figured out that even they cannot connect to DB. Things were not looking right.  General suspect in this kind situation is the process limits. So the alert file was scanned for similar error (though users never reported similar error). Surprisingly there was no such error that indicates that process parameter in DB has not maxed out nor the session’s parameter was!!!

Though Error which we got was following…

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Errors in file /db/archive/app/oracle/ product/diag/rdbms/mmprod/MMPROD/trace/MMPROD_cjq0_778436.trc:

Process W000 died, see its trace file

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Hmm.. Unable to spawn process… May be JOB_QUEUE_PROCESS parameter reached limit. But then it will not allow the background job but will not block user connections, even as sysdba.

Something  amiss. We need to look at different angle now. Probably something wrong with OS hence SA were called in and they declared (as always) that nothing seem wrong with OS. We also asked them to check profile limits and they seemed to be fine.  So the ball is back in out court.

MOS was consulted and we came across few notes suggesting that this might be due to permissions issues. But looking at ORACLE_HOME binaries, things were looking fine!!!

So the decision was taken to bounce the DB and bring the DB back in business as much valuable time was lost in trouble shooting and all SLA in shambles. DB was bounce and now we find ourselves in more serious shit than we can imagine as DB was not ready to start.

Things were getting interesting by minute. Suddenly we noticed that when we were trying to set ORACLE_HOME, we got PERMISSIONS DENIED error. One of the DBA came across following stack trace…

skgpgcmdout: read() for cmd /usr/bin/procstack 1425470 2>&1 timed out after 24.378 seconds

Now something is really messed up and one of our applications Primary DBA (very experienced guy, whose opinion cannot be over ruled!) again asked to SA to check OS more thoroughly.

And Guess what surprise, surprise. SA came back few minutes later and declared that permission on special device /dev/null was changed by an unknown entity and that brought the whole house down.

Once, the permissions were fixed the DB came up nicely.

So the important lesson learned was for the connections issue, not always look at DB but also beyond it.

ORA-01110: data file xx MRP0: Background Media Recovery process shutdown 

whilst doing a simple admin task we ran into issue on our production DG setup. On primary DB we ran out space on one Table space and as routine task one of our DBA, added data file on primary. Usually this kind of changes were routine in our env having DG setup, but in this particular setup we ran into the issue. 


Log file shows - 

MRP0: Background Media Recovery terminated with error 1111
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_pr00_22450.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
Slave exiting with ORA-1111 exception
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_pr00_22450.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
Recovery Slave PR00 previously exited with exception 1111
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_mrp0_22432.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
MRP0: Background Media Recovery process shutdown (xxxxxxx)

In the database - 

SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA/p2psfdom/datafile/domstab3.264.804084437
+DATA/p2psfdom/datafile/hadoopuser_tab.272.804085249
/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025

The database is using ASM hence it should add DBF to ASM and not on disk. This behavior was controlled by DG parameter STANDBY_FILE_MANAGEMENT. 
This parameter was set to MANUAL in this env, hence the file was not propagated to ASM, hence needs a fix.

Following is the fix that one can use to overcome this sort of issue

-- Set the STANDBY_FILE_MANAGEMENT initialization to MANUAL on the standby database if its --- not 
SQL>   alter system set standby_file_management='manual';

-- Next rename the datafile: 

SQL > alter database create datafile ‘/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025’ as '+DG_DATA’ size 8192m;

-- Next we re-enable auto standby file management.
SQL> alter system set standby_file_management='auto';

-- The last step on the standby database side
SQL> recover managed standby database disconnect from session;

Managed recovery should now be able to proceed normally.

OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' OR OPatch failed to locate Central Inventory

This happend during application of the CPU aka SPU i.e. SPU Jan 2013. While running the OPatch on the oracle home we hit the wall.
As a standard practice we run the pre-requisite checks on any Oracle Home / DB which needs to be patched and we ran into folliwng errors.

Output Log
Step is being run by operating system user : 'oracle'
Run privilege of the step is : Normal 
This is Provisioning Executor Script

Input PropertyFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/properties
Input directiveScriptFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl
Input directiveTypeProperty is NAME_Executor
Input componentPath is null
Input directivePath is Directive
Input componentFilename is null
Directive Type is SUB_Perl
Dir to CD is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42
Final command line to execute is $PERL_HOME/perl "/u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl" -debug "false" -oh "/u01/app/oracle/product/11.2.3/db_1" -stageLoc "/u01/app/oracle/agent11g/agent11g/EMStage" -entryPoint "oracle.opatch.em.apply" -patches "14841409" -usePrereqComp "false" -isRac "false" -racNodes "${racNodes}" ${cleanPatches} -patchOpts "''" -applyBundle "${applyBundle}" -entryPointForPatchset "oracle.patchset.db_Custom" -isPatchset "false" -sqlScript ""defaultSqlScript""
The output of the directive is:
Mon Feb 11 03:54:03 2013 - Arguments: -debug false -oh /u01/app/oracle/product/11.2.3/db_1 -stageLoc /u01/app/oracle/agent11g/agent11g/EMStage -entryPoint oracle.opatch.em.apply -patches 14841409 -usePrereqComp false -isRac false -racNodes  -patchOpts '' -applyBundle  -entryPointForPatchset oracle.patchset.db_Custom -isPatchset false -sqlScript defaultSqlScript
Oracle Home:/u01/app/oracle/product/11.2.3/db_1
Stage Location:/u01/app/oracle/agent11g/agent11g/EMStage
Entry Point:oracle.opatch.em.apply
Is RAC :false
Entry Point For Patchset:oracle.patchset.db_Custom
Is PATCHSET :false
OUI and OPatch present in the ORACLE_HOME:/u01/app/oracle/product/11.2.3/db_1 will be used for running the prerequisite checks.
Mon Feb 11 03:54:03 2013 - Checking if the patch has sqls and the required metadata...

Mon Feb 11 03:54:03 2013 - Path is: /u01/app/oracle/agent11g/agent11g/EMStage/14841409/14841409
Mon Feb 11 03:54:03 2013 - STATUS: Patch '14841409' has required metadata to apply sql.

Checking for the latest OPatch ...
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
OPatch version command:/u01/app/oracle/product/11.2.3/db_1/OPatch/opatch version -oh "/u01/app/oracle/product/11.2.3/db_1".
OPatch version:11.2.0.3.3.

Starting execution of Prerequisites...
Total No of checks: 21
Performing check for CheckUserAdminPrivilege
Is the user ADMIN?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOracleHome
Is Oracle Home valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckIfOHLockedForPatching
Is the Oracle Home locked?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckJDK
Is JDK present in the Oracle Home?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOraInstLocation
Is given oraInst.loc file valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckCentralInventoryLocation
Is Central Inventory location valid?
Check complete. T
he overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' from ruleset 'OPatchChecks'. The underlying test 'oracle.opatch.opatchprereq.OPatchChecks.checkCentralInventoryLocation' has thrown an unhandled exception [null]. If the exception results in a failure of the test and the exception needs to be propagated, then encapsulate the exception in the result object and return the result.
Recommendation: Check if the oraInst.loc specified is correct.
========================================================
Performing check for CheckCentralInventoryForOH
Is ORACLE_HOME registered with the Central Inventory?
Check complete. The overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory.
========================================================

Check complete: Failed <<<<
Problem: Read-Write Session could not be created for this Central Inventory.
Recommendation: Check if there are proper permissions for the Central Inventory. Also, check if some other instance has locked the given Central Inventory.
]
Check complete. The overall result of this check is: Failed <<<<

So, clearly the issue was with the central inventory location. So I decided to point to central inventory for OPatch. and ran the check with following flag

#./opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir /home/oracle/14841409

This also failed with following error
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory

So something is amiss and decided to look into the inventory.xml. However, the inventory looks fine and still something not right. Now if you read the above error carefully it also tells you, how to fix it!!

Now I had two ways, possibly, to fix the problem.
1. Either to re-create the central inventoy (which looks preeety dicy to me as this was shared server with other application groups)
2. OR to simpley try to reattach the home and see if that fix the problem.

I decided to go with option 2 and again I have two choices to fix it.

1. Either run the attachHome script from Problem ORACLE_HOME 
OR
2.  run the following command to fix it.
Command -

./runInstaller -silent -invPtrLoc=/u01/app/oracle/product/11.2.3/db_1/oraInst.loc ORACLE_HOME="/u01/app/oracle/product/11.2.3/db_1" ORACLE_HOME_NAME="OraDb11g_home1"

However there is a problem with above command. To use it I need to have the name of ORACLE_HOME. Hence in the case where my inventory is corrupted, it is very difficult to find out the correct name for ORACLE_HOME.
So I simply decided to run the attachHome from ORACLE_HOME
#$ORACLE_HOME/oui/bin/attachHome.sh

And when I ran the opatch again the issue was resolved.

OEM Agent Not Starting Up On Oracle RAC After Restart

 Not sure what went wrong, but after installation of agent on RAC Cluster, the agent was crashing out.

Log shows...

emagent.trc 
FATAL_ERROR::500|ORA-20603: The timezone of the multiagent target
(ODB,rac_database)is not consistent with the timezone (America/Detroit) reported by other agents.
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: number of fatal error exceeds the limit 3
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: agent will shutdown now

emagent.log 
 Unsuccessful Upload attempts for XML file exceeds specified limit=3, Agent will shutdown (00851)
2013-02-04 18:47:57,213 Thread-1383669120 EMAgent abnormal terminating (00704)

So far looks like something wrong with Time zone between OMS and agent. But upon reviewing following file, the thought immediately went away.

emdctl.log
 2013-02-05 04:18:11,186 Thread-2402549120 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:18:11,186 Thread-2402549120 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:22:05,803 Thread-3880828288 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:22:05,803 Thread-3880828288 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:23:06,873 Thread-1536675200 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:23:06,874 Thread-1536675200 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1

So looks like the root.sh on hostb was not ran!!! So its the time to run it and see how it goes.

Latest after root.sh execution

2013-02-05 06:29:19,293 Thread-1116105024 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://oms:7799/em/upload/: retStatus=-1
2013-02-05 06:29:19,298 Thread-1116105024 WARN  http: nmehl_readAgentKey: File access failure
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Open wallet failed, ret = 28759
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: nmehlenv_openWallet failed
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Error initializing SSL
2013-02-05 06:29:19,298 Thread-1116105024 ERROR http: 11: Unable to initialize ssl connection with server, aborting connection attempt: ret -1

Upon careful review of this, I realized the agent was trying to upload data on 7799 port , which is my management port and upload port is 4900. So thats is there. I need to fix the port to upload it on 4900

On HostA Agent URL         : https://hosta:3872/emd/main
Repository URL    : https://oms:4900/em/upload

On HostB
Agent URL         : http://hostb:3872/emd/main
Repository URL    : https://oms:7799/em/upload/

So one has to modify the $AGENT_HOME/sysman/config/emd.properties

There is an entry REPOSITORY_URL=https://oms:7799/em/upload

I have to modify this to reflect the port as follows
REPOSITORY_URL=https://oms:4900/em/upload

Once done I have to clear out the pending upload files...
Files to remove -
$AGENT_HOME/sysman/emd/state/*
$AGENT_HOME/sysman/emd/upload/*

Once done do clearstate agent
$AGENT_HOME/bin/emctl clearstate agent

Try to start the agent now..
$AGENT_HOME/bin/emctl start agent and now, the agent comes back up fine..

OMS hung with warnings <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>

Recently our prod OMS server was hung and was not accessible. We tried to bounce, as it was showing OMS down and tried to being it up.But OMS didnt come up and only WebTier came up. We tried twice and both the times it failed miserably.

Error During Startup - Error during start oms. Please check error and log files

We checked the emctl.trc and log files to see whats goin on we found following.
-- Here it was showing the connection reset. But why to reset the conection which was already established ?? something was not right so need to see other logs.

2013-01-15 01:22:06,139 [EMUI_01_22_02_/console/database/instance/sitemap] ERROR svlt.PageHandler handleRequest.640 - javax.servlet.ServletException: javax.servlet.jsp.JspException: java.net.SocketException: Connection reset
javax.servlet.ServletException: javax.servlet.jsp.JspException: java.net.SocketException: Connection reset
        at weblogic.servlet.jsp.PageContextImpl.handlePageException(PageContextImpl.java:417)
        at jsp_servlet._database._instance._sitemap.__sitemap._jspService(__sitemap.java:2620)

-- emctl.trc shows following, still no clue whats happening !!
java.sql.SQLException: ORA-01403: no data found
ORA-06512: at "SYSMAN.EM_TARGET", line 3503
ORA-06512: at line 1

-- Weblogic logs following error with code. At last some error code to dig around and see what the real issue is.
WARN  jdbc.ConnectionCache _getConnection.354 - Got a fatal exeption when getting a connection; Error code = 17002; Cleaning up cache and retrying
<Jan 16, 2013 1:18:02 AM CST> <Warning> <Socket> <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>
which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        com.bea.logging.RotatingFileOutputStream.write(RotatingFileOutputStream.java:227)
        weblogic.servlet.logging.CLFLogger.log(CLFLogger.java:108)
        weblogic.servlet.logging.LogManagerHttp.log(LogManagerHttp.java:121)
        weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1510)
        weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1441)
        weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
        weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
####<Jan 16, 2013 12:57:52 AM CST> <Error> <WebLogicServer> <EMGC_OMS1> <[STANDBY] ExecuteThread: '72' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1358319472827> <BEA-000337> <[STUCK] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "632" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@385fca37[

-- However the error that really threw me off was this from EMGC_OMS log file
Exception in thread "OMSHeartbeatThread" java.lang.OutOfMemoryError: Java heap space

So, Now I know the reason the story why the connections were reset and cache was being cleanned up. Hmm. It was running low on memory. Meanwhile we made the matters worse by trying to restart the OMS twice and stoping it. How ? well, when we tried to stop the OMS after failed restart, the stop action also errored out. Interestingly that left the JAVA process intact, kicking and alive!!

fix - To fix the issue first we have to clean up the mess and that to in crude way. So we did following.
-- Identify the java processes running and then kill them# ps -ef | grep java
# kill -9 <pid>
-- Identify and EMGC processes running and kill them # ps -ef | grep -V | grep EMGC
# kill -9 <pid>

Now when we tried to start OMS, if came up nice and clean. However we have just fixed the issue temporarily. The root cause was lack of memory. Grid Control 11g OMS for Linux 64 bit software was shipped with a default maximum OMS JVM heap size of 512MB. This value can be a little low for larger Grid Control deployments and should be raised to 1024MB.
So, the following is the way to do it...

#cd /u01/app/Oracle/GC11g/gc_inst/user_projects/domains/GCDomain/bin
-- take backup of file before making changes#cp startEMServer.sh startEMServer.sh.bkp
-- Add the following lines to the file startEMServer.sh before the invocation of startWeblogic.sh command:

if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then

USER_MEM_ARGS="-Xms256m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=512m"
export USER_MEM_ARGS
fi

Save and exit. Start the OMS once again and check if its running with new modifed parameters!! Now onwards your OMS may less likely to encounter this issue.

Applying DB PSU 11.2.0.1.6 (12419378 ) Patch

To install the PSU 11.2.0.1.6 patch, the Oracle home must have the 11.2.0.1.0 Database installed. Subsequent PSU patches can be installed on Oracle Database 11.2.0.1.0 or any PSU with a lower 5th numeral version than the one being installed.
The actual patch needs to be downloaded from MOS and the one I used was for Linux - 
Patch Name - p12419378_112010_LINUX_11.2.0.1.6(PSU)

You must use the Opatch utility version 11.2.0.1.3 or later to apply this patch. Oracle recommends that you use the latest released OPatch 11.2, which is available for download from My Oracle Support patch 6880880 by selecting the 11.2.0.0.0 release.
For updating OPatch to latest version pls check the following note -
<http://handsonoracle.blogspot.in/2012/05/apply-psu-5-for-11gr2-grid-and-rac-11.html>

If you are installing the PSU to an environment that has a Grid Infrastructure home, note the following:

-- Database PSU 11.2.0.1.6 should only be applied to the Database home using the instructions contained in the patch readme file

-- 11.2.0.1.2 Grid Infrastructure PSU <<Patch 9655006>> should be applied to the Grid Infrastructure home using the readme instructions provided with that patch.

To apply GI PSU 9655006 check the following note -

<http://handsonoracle.blogspot.in/2012/11/applying-gi-psu-11.html>

Once you have all the pre-requisite checked you can install this patch in rolling mode. I'm implementing this on single node cluster for the demonstration purpose. However the same method can be used for multi-nodes as well.

To do the rolling upgrade, you have to leave the CRS Stack running on the node to be patched and OPatch will automatically shut down that stack during patching and will restart it after patch.

Since this PSU is meant for RAC Binaries you have to shutdown all instances and services running out of this home.

Detailed instructions are as follows..

[root@appsractest oracle]# sudo su - oracle

[oracle@appsractest ~]$ export PATH=$PATH:/u01/app/oracle/product/11.0/db_1/OPatch

[oracle@appsractest ~]$ which opatch

/u01/app/oracle/product/11.0/db_1/OPatch/opatch

[oracle@appsractest ~]$ cd /u01/app/grid/11.0/12419378/

[oracle@appsractest 12419378]$ ll

total 60

drwxr-xr-x  3 oracle oinstall  4096 Jul  8  2011 custom

drwxr-xr-x  4 oracle oinstall  4096 Jul  8  2011 etc

drwxr-xr-x 12 oracle oinstall  4096 Jul  8  2011 files

-rwxr-xr-x  1 oracle oinstall  2871 Jul  8  2011 patchmd.xml

-rw-rw-r--  1 oracle oinstall 40790 Jul 18  2011 README.html

-rw-r--r--  1 oracle oinstall    21 Jul  8  2011 README.txt

[oracle@appsractest 12419378]$ opatch apply

Oracle Interim Patch Installer version 11.2.0.3.0

Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.0/db_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/11.0/db_1/oraInst.loc

OPatch version    : 11.2.0.3.0

OUI version       : 11.2.0.1.0

Log file location : /u01/app/oracle/product/11.0/db_1/cfgtoollogs/opatch/12419378_Nov_07_2012_10_28_01/apply2012-11-07_10-28-00AM_1.log

Applying interim patch '12419378' to OH '/u01/app/oracle/product/11.0/db_1'

Verifying environment and performing prerequisite checks...

Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ]

Interim patch 12419378 is a superset of the patch(es) [  9654983 ] in the Oracle Home

OPatch will roll back the subset patches and apply the given patch.

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/oracle/product/11.0/db_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Backing up files...

Rolling back interim patch '9654983' from OH '/u01/app/oracle/product/11.0/db_1'

Patching component oracle.rdbms.rsf, 11.2.0.1.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.1.0...

Patching component oracle.rdbms, 11.2.0.1.0...

Patching component oracle.oraolap, 11.2.0.1.0...

Patching component oracle.rdbms.deconfig, 11.2.0.1.0...

Patching component oracle.javavm.server, 11.2.0.1.0...

Patching component oracle.precomp.common, 11.2.0.1.0...

Patching component oracle.network.rsf, 11.2.0.1.0...

Patching component oracle.network.listener, 11.2.0.1.0...

RollbackSession removing interim patch '9654983' from inventory

OPatch back to application of the patch '12419378' after auto-rollback.

Patching component oracle.rdbms.rsf, 11.2.0.1.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.1.0...

Patching component oracle.rdbms, 11.2.0.1.0...

Patching component oracle.oraolap, 11.2.0.1.0...

Patching component oracle.rdbms.deconfig, 11.2.0.1.0...

Patching component oracle.javavm.server, 11.2.0.1.0...

Patching component oracle.precomp.common, 11.2.0.1.0...

Patching component oracle.network.rsf, 11.2.0.1.0...

Patching component oracle.network.listener, 11.2.0.1.0...

Patching component oracle.rdbms.dv.oc4j, 11.2.0.1.0...

Patching component oracle.sdo.locator, 11.2.0.1.0...

Patching component oracle.sysman.console.db, 11.2.0.1.0...

Patching component oracle.sysman.oms.core, 10.2.0.4.2...

Patching component oracle.rdbms.dv, 11.2.0.1.0...

Patching component oracle.rdbms.dv, 11.2.0.1.0...

Patching component oracle.xdk.rsf, 11.2.0.1.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.1.0...

Patching component oracle.ldap.rsf, 11.2.0.1.0...

Patching component oracle.sysman.plugin.db.main.repository, 11.2.0.1.0...

Verifying the update...

atch 12419378 successfully applied

Log file location: /u01/app/oracle/product/11.0/db_1/cfgtoollogs/opatch/12419378_Nov_07_2012_10_28_01/apply2012-11-07_10-28-00AM_1.log

OPatch succeeded.

-- Now if you check the OPatch for the applied PSU's you will see something like following.

9352237    12419378  Wed Nov 07 10:46:19 IST 2012   DATABASE PSU 11.2.0.1.1

9654983    12419378  Wed Nov 07 10:46:19 IST 2012   DATABASE PSU 11.2.0.1.2 (INCLUDES CPUJUL2010)

9952216    12419378  Wed Nov 07 10:46:19 IST 2012   DATABASE PSU 11.2.0.1.3 (INCLUDES CPUOCT2010)

12419378   12419378  Wed Nov 07 10:46:19 IST 2012   DATABASE PSU 11.2.0.1.6 (INCLUDES CPUJUL2011)

PNow you just need to load the modified sql in the DB the way you normally load during your CPU apply process and you are done. You jsut applied DB PSU successfully to your Database.

Applying GI PSU 11.2.0.1.2 (9655006) on 11.2.0.1 fails with 

"The opatch Component check failed"

Application of the GI PSU 11.2.0.1.2 failed with following error.. Though the pre-requisite of having OPatch version was clearly satisfied after upgrading OPatch to 11.2.0.3 using patch p6880880 as per MOS note.

Error - 

[root@appsractest 11.0]# opatch auto /u01/app/grid/11.0/stage  -oh /u01/app/grid/11.0
Executing /usr/bin/perl /u01/app/grid/11.0/OPatch/crs/patch112.pl -patchdir /u01/app/grid/11.0 -patchn stage -oh /u01/app/grid/11.0 -paramfile /u01/app/grid/11.0/crs/install/crsconfig_params
opatch auto log file location is /u01/app/grid/11.0/OPatch/crs/../../cfgtoollogs/opatchauto2012-11-07_06-24-58.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/grid/11.0/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
The opatch minimum version  check for patch /u01/app/grid/11.0/stage/9655006 failed  for /u01/app/grid/11.0
The opatch minimum version  check for patch /u01/app/grid/11.0/stage/9654983 failed  for /u01/app/grid/11.0
Opatch version check failed for oracle home  /u01/app/grid/11.0
Opatch version  check failed

Fix - 
There are mainly two possible reasons for this...

1. Multiple patches in the directory. 
Make sure you have only one patch unzipped in the directory specified as patch top. If there are multiple patches then the OPatch will fail.

2. Permission issues when trying to write to hidden patch_storage directory under GRID_HOME. When opatch is run it will try to update the info in patch_storage directory so that rollback of the patch is possible.
When OPatch auto is run (even as root), internally it will execute opatch utility with napply using Oracle user. So if the oracle user doesn't have the permissions to write to .patch_storage the opatch auto application will fail.
[root@appsractest 11.0]# cd /u01/app/grid/11.0
[root@appsractest 11.0]# ll .patch_storage/
total 4
-rw-r--r-- 1 root root 796 Nov  5 10:15 LsInventory__11-05-2012_10-15-08.log
[root@appsractest 11.0]# chown  oracle:oinstall .patch_storage/ -R
[root@appsractest 11.0]# ll .patch_storage/
total 4
-rw-r--r-- 1 oracle oinstall 796 Nov  5 10:15 LsInventory__11-05-2012_10-15-08.log
[root@appsractest 11.0]# chmod 777 .patch_storage/ -R
[root@appsractest 11.0]# ll .patch_storage/
total 4
-rwxrwxrwx 1 oracle oinstall 796 Nov  5 10:15 LsInventory__11-05-2012_10-15-08.log

Once done above two things, when you run the opatch again, it will go thru fine 
-- To update the GI home
[root@appsractest 11.0]# opatch auto /u01/app/grid/11.0/stage  -oh /u01/app/grid/11.0
Executing /usr/bin/perl /u01/app/grid/11.0/OPatch/crs/patch112.pl -patchdir /u01/app/grid/11.0 -patchn stage -oh /u01/app/grid/11.0 -paramfile /u01/app/grid/11.0/crs/install/crsconfig_params
opatch auto log file location is /u01/app/grid/11.0/OPatch/crs/../../cfgtoollogs/opatchauto2012-11-07_06-27-26.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/grid/11.0/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
Successfully unlock /u01/app/grid/11.0
patch /u01/app/grid/11.0/stage/9655006  apply successful for home  /u01/app/grid/11.0
patch /u01/app/grid/11.0/stage/9654983  apply successful for home  /u01/app/grid/11.0
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.

-- To Update RAC home

[root@appsractest ~]# export PATH=$PATH:/u01/app/oracle/product/11.0/db_1/OPatch/

[root@appsractest ~]# which opatch

/u01/app/oracle/product/11.0/db_1/OPatch/opatch

[root@appsractest ~]# opatch auto /u01/app/grid/11.0/stage -oh /u01/app/oracle/product/11.0/db_1

Executing /usr/bin/perl /u01/app/oracle/product/11.0/db_1/OPatch/crs/patch112.pl -patchdir /u01/app/grid/11.0 -patchn stage -oh /u01/app/oracle/product/11.0/db_1 -paramfile /u01/app/grid/11.0/crs/install/crsconfig_params

opatch auto log file location is /u01/app/oracle/product/11.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-11-07_07-03-02.log

Detected Oracle Clusterware install

Using configuration parameter file: /u01/app/grid/11.0/crs/install/crsconfig_params

OPatch  is bundled with OCM, Enter the absolute OCM response file path:

/home/oracle/ocm.rsp

patch /u01/app/grid/11.0/stage/9655006/custom/server/9655006  apply successful for home  /u01/app/oracle/product/11.0/db_1

patch /u01/app/grid/11.0/stage/9654983  apply successful for home  /u01/app/oracle/product/11.0/db_1

-- Check the status of the resources for RAC Home and you will find that resources are started automatically after application of patch

[oracle@appsractest ~]$ srvctl  status database -d iedge

Instance iedge1 is running on node appsractest

-- Now when you try to see the applied patch history from OPatch, you will see that patch is applied...

9655006    9655006   Wed Nov 07 07:06:11 IST 2012   GI PSU 11.2.0.1.2 (INCLUDES DATABASE PSU 11.2.0.1.2)