Cluster Agent Install fails with OUI-35000 & PRKC-1044

While installing the cluster OEM agent on RAC, we hit following error.

[oracle@rac1a bin]$agentDownload.linux_x64 -b /u01/app/oracle/product -m oemgc1 -r 7799 -n XXXPRD -c "rac1a,rac1b" -y 

ERROR: OUI-35000: Fatal cluster error encountered (PRKC-1044 : Failed to check remote command execution setup for node rac1b using shells /usr/bin/ssh and /usr/bin/rsh

rac1b: Connection refusedPRKC-1044 : Failed to check remote command execution setup for node rac1a using shells /usr/bin/ssh and /usr/bin/rsh

rac1a: Connection refused). Correct the problem and try the operation again.

Completed with Status=1

Primarily the issue seems to be with rsh setup on cluster nodes. Initially it was not setup hence we got it done with help of SA's. So RSH was in place but again it failed with same error. 

So I decided to run it from another node in cluster and After running from it, hit the following err at end...

ERROR: Remote 'AttachHome' failed on nodes: 'rac1a'. Refer to '/u01/app/oraInventory/logs/installActions2013-04-12_05-47-02AM.log' for details.

You can manually re-run the following command on the failed nodes after the installation:

 /u01/app/oracle/product/agent11g/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/agent11g ORACLE_HOME_NAME=agent11g1 CLUSTER_NODES=rac1a,rac1b "INVENTORY_LOCATION=/u01/app/oraInventory" LOCAL_NODE=<node on which command is to be run>.

That looks like install went fine but agents were not configured at all. Also when you check inventory the agent home was not registered with it, hence above message made some sense. 

So I went ahead and ran following command on both nodes one by one. 

runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/agent11g ORACLE_HOME_NAME=agent11g1 CLUSTER_NODES=rac1a,rac1b "INVENTORY_LOCATION=/u01/app/oracle/oraInventory" LOCAL_NODE=rac1a

On Node1 -

[oracle@rac1a bin]$ ./runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/u01/app/oracle/product/agent11g ORACLE_HOME_NAME=agent11g1 CLUSTER_NODES=rac1a,rac1b "INVENTORY_LOCATION=/u01/app/oraInventory" LOCAL_NODE=rac1a

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 19077 MB    Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-04-12_06-46-16AM. Please wait ...[oracle@rac1a bin]$ The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

Please execute the 'null' script at the end of the session.

'AttachHome' was successful.

[oracle@rac1a bin]$ ./agentca -f -n XXXPRD -c rac1a,rac1b

CLUSTER_NAME environment variable is set to XXXPRD

The ORACLE_HOME=/u01/app/oracle/product/agent11g doesn't exist in the oraInventory specified in /oracletemp/PS1/oraInventory, Please specify the correct oraInventory location using -i option

Since there are multiple ORACLE_HOME on this host, the inventory pointer was wrong in /etc/oraInst.loc. 

[oracle@rac1a bin]$ cat /etc/oraInst.loc

#inventory_loc=/oracletemp/oraInventory

inst_group=dba

So inventory was wrong hence needs to fix it to point at right location after adding following entry

inventory_loc=/u01/app/oraInventory

[oracle@rac1a bin]$ cat /etc/oraInst.loc

#inventory_loc=/oracletemp/oraInventory

inventory_loc=/u01/app/oraInventory

inst_group=dba

[oracle@rac1a bin]$ ./agentca -f -n XXXPRD -c rac1a,rac1b -i /etc/oraInst.loc

CLUSTER_NAME environment variable is set to XXXPRD

Stopping the agent using /u01/app/oracle/product/agent11g/bin/emctl  stop agent

EM Configuration issue. /u01/app/oracle/product/agent11g/rac1a not found.

Running agentca using /u01/app/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent11g ACTION=Configure MODE=Perform RESPONSE_FILE=/u01/app/oracle/product/agent11g/response_file RERUN=TRUE INV_PTR_LOC=/etc/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}

Perform - mode is starting for action: Configure

Perform - mode finished for action: Configure

You can see the log file: /u01/app/oracle/product/agent11g/cfgtoollogs/oui/configActions2013-04-12_07-14-18-AM.log

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 19077 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

Starting the agent using /u01/app/oracle/product/agent11g/bin/emctl  start agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0

Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

Starting agent ......... started.

Stopping the agent using /u01/app/oracle/product/agent11g/bin/emctl  stop agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0

Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

Stopping agent ... stopped.

Running Agent Addon Configuration using /u01/app/oracle/product/agent11g/perl/bin/perl /u01/app/oracle/product/agent11g/sysman/install/AddonConfig.pl

Arguments passed

Configuring Addon from xml : oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml

Running Command : /u01/app/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent11g ACTION=configure MODE=perform RERUN=true  RESPONSE_FILE=/u01/app/oracle/product/agent11g/vt_responsefile COMPONENT_XML={oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml}

 Setting the invPtrLoc to /u01/app/oracle/product/agent11g/oraInst.loc

perform - mode is starting for action: configure

perform - mode finished for action: configure

You can see the log file: /u01/app/oracle/product/agent11g/cfgtoollogs/oui/configActions2013-04-12_07-15-12-AM.log

Agent Addon Configuration done

Starting the agent using /u01/app/oracle/product/agent11g/bin/emctl  start agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0

Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

Agent is already running

On Node2 - 

oracle@rac1b /u01/app/oracle/product/agent11g/bin> ./agentca -f -n XXXPRD -c rac1a,rac1b

CLUSTER_NAME environment variable is set to XXXPRD

Stopping the agent using /u01/app/oracle/product/agent11g/bin/emctl  stop agent

EM Configuration issue. /u01/app/oracle/product/agent11g/rac1b not found.

Running agentca using /u01/app/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent11g ACTION=Configure MODE=Perform RESPONSE_FILE=/u01/app/oracle/product/agent11g/response_file RERUN=TRUE INV_PTR_LOC=/u01/app/oracle/product/agent11g/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}

Perform - mode is starting for action: Configure

Running Agent Addon Configuration using /u01/app/oracle/product/agent11g/perl/bin/perl /u01/app/oracle/product/agent11g/sysman/install/AddonConfig.pl

Arguments passed

Configuring Addon from xml : oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml

Running Command : /u01/app/oracle/product/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent11g ACTION=configure MODE=perform RERUN=true  RESPONSE_FILE=/u01/app/oracle/product/agent11g/vt_responsefile COMPONENT_XML={oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml}

 Setting the invPtrLoc to /u01/app/oracle/product/agent11g/oraInst.loc

perform - mode is starting for action: configure

perform - mode finished for action: configure

You can see the log file: /u01/app/oracle/product/agent11g/cfgtoollogs/oui/configActions2013-04-12_07-21-35-AM.log

 Agent Addon Configuration done

Starting the agent using /u01/app/oracle/product/agent11g/bin/emctl  start agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0

Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

Agent is already running

Once done, cluster was properly discovered by OEM GC and agent was able to upload data successfully....

PRCR-1013 : Failed to start resource  OR 

CRS-2640: Required resource is missing

During start of DB resource we hit following error...

Error - 
PRCR-1013 : Failed to start resource ora.ASdb.db
PRCR-1064 : Failed to start resource ora.ASdb.db on node orarac1d
CRS-2640: Required resource 'ora.ASDATA03.dg' is missing.

So it says the disk group missing. When I check the ASM for the existing disk group I found that the complain was correct and I indeed cannot see the ASDATA03 disk group. After some digging I found out that disk group has been renamed from ASDATA03 to ASDATA03_05 hence the dependency was failing during start of the resource.

Diskgroup Status in ASM for ASDB -
ASDATA02_04                  MOUNTED     EXTERN             0
ASDATA03_05                  MOUNTED     EXTERN             0
ASDATA01                     MOUNTED     EXTERN             0

Disgroups registered with Cluster resouse ASDB- 
Disk Groups: ASDATA01,FRA (Missing DG ASDATA02_04 and ASDATA03_05)

Currently Datafile and archlog files are distributed on following - 
Diskgorups - ASDATA01, ASDATA03_05, ASDATA02_04

Since ASM diskgroup ASDATA02 and ASDATA03 hAS been physically dropped, We should them from CRS start/stop dependency for databASe resource 

[grid@orarac1d bin]$ crsctl stat res -p | grep ASDB
NAME=ora.FRA_ASDB.dg
DB_UNIQUE_NAME=ASDB
GEN_AUDIT_FILE_DEST=/u01/app/oracle/admin/ASDB/adump
GEN_USR_ORA_INST_NAME@SERVERNAME(orarac1c)=ASDB1
GEN_USR_ORA_INST_NAME@SERVERNAME(orarac1d)=ASDB2
SERVER_POOLS=ora.ASDB
SPFILE=+ASDATA01/ASDB/spfileASDB.ora

START_DEPENDENCIES=hard(ora.ASDATA01.dg,ora.FRA.dg, ora.ASDATA03.dg,ora.FRA_ASDB.dg,ora.ASDATA02.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.ASDATA01.dg,ora.FRA.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.ASm,shutdown:ora.ASDATA01.dg,shutdown:ora.FRA.dg,shutdown:ora.ASDATA03.dg, shutdown:ora.FRA_ASDB.dg, shutdown:ora.ASDATA02.dg)

Fix –
1. Modify ASM disk group dependency requirement
$ srvctl modify databASe -d MYDB -a “ASDATA01,ASDATA03_05,ASDATA02_04,FRA,FRA_ASDB”

2. Disable disk group if exist
$ srvctl disable diskgroup -g ASDATA03,ASDATA02 

3. Drop disk group info from OCR. Make sure you dont have any existing DB using this disk groups
$ srvctl remove diskgroup -g ASDATA03,ASDATA02 –f

Once you finish this steps and next time around when you start or stop the resource using srvctl utility, it wont complain and will start resources as expected.

ORA-12547: TNS:lost contact / Process W000 died /

Process J000 died / kkjcre1p: unable to spawn jobq slave process

There are days when you can’t seem to do a thing wrong and some are days when you can’t do a thing right. 26 March 13 was also one of those later kind of days.

DBA got the page that production DB was not accessible for users to run batch jobs and we jumped on issue. The error reported was as follows

ORA-12547: TNS:lost contact

This is very generic error. After spending a bit time DBA figured out that even they cannot connect to DB. Things were not looking right.  General suspect in this kind situation is the process limits. So the alert file was scanned for similar error (though users never reported similar error). Surprisingly there was no such error that indicates that process parameter in DB has not maxed out nor the session’s parameter was!!!

Though Error which we got was following…

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Errors in file /db/archive/app/oracle/ product/diag/rdbms/mmprod/MMPROD/trace/MMPROD_cjq0_778436.trc:

Process W000 died, see its trace file

Process J000 died, see its trace file

kkjcre1p: unable to spawn jobq slave process

Hmm.. Unable to spawn process… May be JOB_QUEUE_PROCESS parameter reached limit. But then it will not allow the background job but will not block user connections, even as sysdba.

Something  amiss. We need to look at different angle now. Probably something wrong with OS hence SA were called in and they declared (as always) that nothing seem wrong with OS. We also asked them to check profile limits and they seemed to be fine.  So the ball is back in out court.

MOS was consulted and we came across few notes suggesting that this might be due to permissions issues. But looking at ORACLE_HOME binaries, things were looking fine!!!

So the decision was taken to bounce the DB and bring the DB back in business as much valuable time was lost in trouble shooting and all SLA in shambles. DB was bounce and now we find ourselves in more serious shit than we can imagine as DB was not ready to start.

Things were getting interesting by minute. Suddenly we noticed that when we were trying to set ORACLE_HOME, we got PERMISSIONS DENIED error. One of the DBA came across following stack trace…

skgpgcmdout: read() for cmd /usr/bin/procstack 1425470 2>&1 timed out after 24.378 seconds

Now something is really messed up and one of our applications Primary DBA (very experienced guy, whose opinion cannot be over ruled!) again asked to SA to check OS more thoroughly.

And Guess what surprise, surprise. SA came back few minutes later and declared that permission on special device /dev/null was changed by an unknown entity and that brought the whole house down.

Once, the permissions were fixed the DB came up nicely.

So the important lesson learned was for the connections issue, not always look at DB but also beyond it.

ORA-01110: data file xx MRP0: Background Media Recovery process shutdown 

whilst doing a simple admin task we ran into issue on our production DG setup. On primary DB we ran out space on one Table space and as routine task one of our DBA, added data file on primary. Usually this kind of changes were routine in our env having DG setup, but in this particular setup we ran into the issue. 


Log file shows - 

MRP0: Background Media Recovery terminated with error 1111
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_pr00_22450.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
Slave exiting with ORA-1111 exception
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_pr00_22450.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
Recovery Slave PR00 previously exited with exception 1111
Errors in file /u01/app/oracle/diag/rdbms/xxxxxxx/xxxxxxx/trace/xxxxxxx_mrp0_22432.trc:
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
ORA-01157: cannot identify/lock data file 25 - see DBWR trace file
ORA-01111: name for data file 25 is unknown - rename to correct file
ORA-01110: data file 25: '/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025'
MRP0: Background Media Recovery process shutdown (xxxxxxx)

In the database - 

SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA/p2psfdom/datafile/domstab3.264.804084437
+DATA/p2psfdom/datafile/hadoopuser_tab.272.804085249
/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025

The database is using ASM hence it should add DBF to ASM and not on disk. This behavior was controlled by DG parameter STANDBY_FILE_MANAGEMENT. 
This parameter was set to MANUAL in this env, hence the file was not propagated to ASM, hence needs a fix.

Following is the fix that one can use to overcome this sort of issue

-- Set the STANDBY_FILE_MANAGEMENT initialization to MANUAL on the standby database if its --- not 
SQL>   alter system set standby_file_management='manual';

-- Next rename the datafile: 

SQL > alter database create datafile ‘/u01/app/oracle/product/11.2.0/db_1/dbs/UNNAMED00025’ as '+DG_DATA’ size 8192m;

-- Next we re-enable auto standby file management.
SQL> alter system set standby_file_management='auto';

-- The last step on the standby database side
SQL> recover managed standby database disconnect from session;

Managed recovery should now be able to proceed normally.

OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' OR OPatch failed to locate Central Inventory

This happend during application of the CPU aka SPU i.e. SPU Jan 2013. While running the OPatch on the oracle home we hit the wall.
As a standard practice we run the pre-requisite checks on any Oracle Home / DB which needs to be patched and we ran into folliwng errors.

Output Log
Step is being run by operating system user : 'oracle'
Run privilege of the step is : Normal 
This is Provisioning Executor Script

Input PropertyFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/properties
Input directiveScriptFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl
Input directiveTypeProperty is NAME_Executor
Input componentPath is null
Input directivePath is Directive
Input componentFilename is null
Directive Type is SUB_Perl
Dir to CD is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42
Final command line to execute is $PERL_HOME/perl "/u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl" -debug "false" -oh "/u01/app/oracle/product/11.2.3/db_1" -stageLoc "/u01/app/oracle/agent11g/agent11g/EMStage" -entryPoint "oracle.opatch.em.apply" -patches "14841409" -usePrereqComp "false" -isRac "false" -racNodes "${racNodes}" ${cleanPatches} -patchOpts "''" -applyBundle "${applyBundle}" -entryPointForPatchset "oracle.patchset.db_Custom" -isPatchset "false" -sqlScript ""defaultSqlScript""
The output of the directive is:
Mon Feb 11 03:54:03 2013 - Arguments: -debug false -oh /u01/app/oracle/product/11.2.3/db_1 -stageLoc /u01/app/oracle/agent11g/agent11g/EMStage -entryPoint oracle.opatch.em.apply -patches 14841409 -usePrereqComp false -isRac false -racNodes  -patchOpts '' -applyBundle  -entryPointForPatchset oracle.patchset.db_Custom -isPatchset false -sqlScript defaultSqlScript
Oracle Home:/u01/app/oracle/product/11.2.3/db_1
Stage Location:/u01/app/oracle/agent11g/agent11g/EMStage
Entry Point:oracle.opatch.em.apply
Is RAC :false
Entry Point For Patchset:oracle.patchset.db_Custom
Is PATCHSET :false
OUI and OPatch present in the ORACLE_HOME:/u01/app/oracle/product/11.2.3/db_1 will be used for running the prerequisite checks.
Mon Feb 11 03:54:03 2013 - Checking if the patch has sqls and the required metadata...

Mon Feb 11 03:54:03 2013 - Path is: /u01/app/oracle/agent11g/agent11g/EMStage/14841409/14841409
Mon Feb 11 03:54:03 2013 - STATUS: Patch '14841409' has required metadata to apply sql.

Checking for the latest OPatch ...
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
OPatch version command:/u01/app/oracle/product/11.2.3/db_1/OPatch/opatch version -oh "/u01/app/oracle/product/11.2.3/db_1".
OPatch version:11.2.0.3.3.

Starting execution of Prerequisites...
Total No of checks: 21
Performing check for CheckUserAdminPrivilege
Is the user ADMIN?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOracleHome
Is Oracle Home valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckIfOHLockedForPatching
Is the Oracle Home locked?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckJDK
Is JDK present in the Oracle Home?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOraInstLocation
Is given oraInst.loc file valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckCentralInventoryLocation
Is Central Inventory location valid?
Check complete. T
he overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' from ruleset 'OPatchChecks'. The underlying test 'oracle.opatch.opatchprereq.OPatchChecks.checkCentralInventoryLocation' has thrown an unhandled exception [null]. If the exception results in a failure of the test and the exception needs to be propagated, then encapsulate the exception in the result object and return the result.
Recommendation: Check if the oraInst.loc specified is correct.
========================================================
Performing check for CheckCentralInventoryForOH
Is ORACLE_HOME registered with the Central Inventory?
Check complete. The overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory.
========================================================

Check complete: Failed <<<<
Problem: Read-Write Session could not be created for this Central Inventory.
Recommendation: Check if there are proper permissions for the Central Inventory. Also, check if some other instance has locked the given Central Inventory.
]
Check complete. The overall result of this check is: Failed <<<<

So, clearly the issue was with the central inventory location. So I decided to point to central inventory for OPatch. and ran the check with following flag

#./opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir /home/oracle/14841409

This also failed with following error
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory

So something is amiss and decided to look into the inventory.xml. However, the inventory looks fine and still something not right. Now if you read the above error carefully it also tells you, how to fix it!!

Now I had two ways, possibly, to fix the problem.
1. Either to re-create the central inventoy (which looks preeety dicy to me as this was shared server with other application groups)
2. OR to simpley try to reattach the home and see if that fix the problem.

I decided to go with option 2 and again I have two choices to fix it.

1. Either run the attachHome script from Problem ORACLE_HOME 
OR
2.  run the following command to fix it.
Command -

./runInstaller -silent -invPtrLoc=/u01/app/oracle/product/11.2.3/db_1/oraInst.loc ORACLE_HOME="/u01/app/oracle/product/11.2.3/db_1" ORACLE_HOME_NAME="OraDb11g_home1"

However there is a problem with above command. To use it I need to have the name of ORACLE_HOME. Hence in the case where my inventory is corrupted, it is very difficult to find out the correct name for ORACLE_HOME.
So I simply decided to run the attachHome from ORACLE_HOME
#$ORACLE_HOME/oui/bin/attachHome.sh

And when I ran the opatch again the issue was resolved.

OEM Agent Not Starting Up On Oracle RAC After Restart

 Not sure what went wrong, but after installation of agent on RAC Cluster, the agent was crashing out.

Log shows...

emagent.trc 
FATAL_ERROR::500|ORA-20603: The timezone of the multiagent target
(ODB,rac_database)is not consistent with the timezone (America/Detroit) reported by other agents.
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: number of fatal error exceeds the limit 3
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: agent will shutdown now

emagent.log 
 Unsuccessful Upload attempts for XML file exceeds specified limit=3, Agent will shutdown (00851)
2013-02-04 18:47:57,213 Thread-1383669120 EMAgent abnormal terminating (00704)

So far looks like something wrong with Time zone between OMS and agent. But upon reviewing following file, the thought immediately went away.

emdctl.log
 2013-02-05 04:18:11,186 Thread-2402549120 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:18:11,186 Thread-2402549120 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:22:05,803 Thread-3880828288 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:22:05,803 Thread-3880828288 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:23:06,873 Thread-1536675200 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:23:06,874 Thread-1536675200 ERROR main: nmectla_agentctl: Error connecting to https://hostb:3872/emd/main. Returning status code 1

So looks like the root.sh on hostb was not ran!!! So its the time to run it and see how it goes.

Latest after root.sh execution

2013-02-05 06:29:19,293 Thread-1116105024 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://oms:7799/em/upload/: retStatus=-1
2013-02-05 06:29:19,298 Thread-1116105024 WARN  http: nmehl_readAgentKey: File access failure
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Open wallet failed, ret = 28759
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: nmehlenv_openWallet failed
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Error initializing SSL
2013-02-05 06:29:19,298 Thread-1116105024 ERROR http: 11: Unable to initialize ssl connection with server, aborting connection attempt: ret -1

Upon careful review of this, I realized the agent was trying to upload data on 7799 port , which is my management port and upload port is 4900. So thats is there. I need to fix the port to upload it on 4900

On HostA Agent URL         : https://hosta:3872/emd/main
Repository URL    : https://oms:4900/em/upload

On HostB
Agent URL         : http://hostb:3872/emd/main
Repository URL    : https://oms:7799/em/upload/

So one has to modify the $AGENT_HOME/sysman/config/emd.properties

There is an entry REPOSITORY_URL=https://oms:7799/em/upload

I have to modify this to reflect the port as follows
REPOSITORY_URL=https://oms:4900/em/upload

Once done I have to clear out the pending upload files...
Files to remove -
$AGENT_HOME/sysman/emd/state/*
$AGENT_HOME/sysman/emd/upload/*

Once done do clearstate agent
$AGENT_HOME/bin/emctl clearstate agent

Try to start the agent now..
$AGENT_HOME/bin/emctl start agent and now, the agent comes back up fine..

OMS hung with warnings <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>

Recently our prod OMS server was hung and was not accessible. We tried to bounce, as it was showing OMS down and tried to being it up.But OMS didnt come up and only WebTier came up. We tried twice and both the times it failed miserably.

Error During Startup - Error during start oms. Please check error and log files

We checked the emctl.trc and log files to see whats goin on we found following.
-- Here it was showing the connection reset. But why to reset the conection which was already established ?? something was not right so need to see other logs.

2013-01-15 01:22:06,139 [EMUI_01_22_02_/console/database/instance/sitemap] ERROR svlt.PageHandler handleRequest.640 - javax.servlet.ServletException: javax.servlet.jsp.JspException: java.net.SocketException: Connection reset
javax.servlet.ServletException: javax.servlet.jsp.JspException: java.net.SocketException: Connection reset
        at weblogic.servlet.jsp.PageContextImpl.handlePageException(PageContextImpl.java:417)
        at jsp_servlet._database._instance._sitemap.__sitemap._jspService(__sitemap.java:2620)

-- emctl.trc shows following, still no clue whats happening !!
java.sql.SQLException: ORA-01403: no data found
ORA-06512: at "SYSMAN.EM_TARGET", line 3503
ORA-06512: at line 1

-- Weblogic logs following error with code. At last some error code to dig around and see what the real issue is.
WARN  jdbc.ConnectionCache _getConnection.354 - Got a fatal exeption when getting a connection; Error code = 17002; Cleaning up cache and retrying
<Jan 16, 2013 1:18:02 AM CST> <Warning> <Socket> <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>
which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        com.bea.logging.RotatingFileOutputStream.write(RotatingFileOutputStream.java:227)
        weblogic.servlet.logging.CLFLogger.log(CLFLogger.java:108)
        weblogic.servlet.logging.LogManagerHttp.log(LogManagerHttp.java:121)
        weblogic.servlet.internal.ServletResponseImpl.send(ServletResponseImpl.java:1510)
        weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1441)
        weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
        weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
####<Jan 16, 2013 12:57:52 AM CST> <Error> <WebLogicServer> <EMGC_OMS1> <[STANDBY] ExecuteThread: '72' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1358319472827> <BEA-000337> <[STUCK] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "632" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@385fca37[

-- However the error that really threw me off was this from EMGC_OMS log file
Exception in thread "OMSHeartbeatThread" java.lang.OutOfMemoryError: Java heap space

So, Now I know the reason the story why the connections were reset and cache was being cleanned up. Hmm. It was running low on memory. Meanwhile we made the matters worse by trying to restart the OMS twice and stoping it. How ? well, when we tried to stop the OMS after failed restart, the stop action also errored out. Interestingly that left the JAVA process intact, kicking and alive!!

fix - To fix the issue first we have to clean up the mess and that to in crude way. So we did following.
-- Identify the java processes running and then kill them# ps -ef | grep java
# kill -9 <pid>
-- Identify and EMGC processes running and kill them # ps -ef | grep -V | grep EMGC
# kill -9 <pid>

Now when we tried to start OMS, if came up nice and clean. However we have just fixed the issue temporarily. The root cause was lack of memory. Grid Control 11g OMS for Linux 64 bit software was shipped with a default maximum OMS JVM heap size of 512MB. This value can be a little low for larger Grid Control deployments and should be raised to 1024MB.
So, the following is the way to do it...

#cd /u01/app/Oracle/GC11g/gc_inst/user_projects/domains/GCDomain/bin
-- take backup of file before making changes#cp startEMServer.sh startEMServer.sh.bkp
-- Add the following lines to the file startEMServer.sh before the invocation of startWeblogic.sh command:

if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then

USER_MEM_ARGS="-Xms256m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=512m"
export USER_MEM_ARGS
fi

Save and exit. Start the OMS once again and check if its running with new modifed parameters!! Now onwards your OMS may less likely to encounter this issue.