Tuesday, January 14, 2014


Upgrade OEM 11g to 12C - One System Approach


There are two different ways to upgrade OEM Grid Control environment.

1.      One system upgrade
2.      Two system upgrade.

One system upgrade is nothing but the upgrading the existing environment. In this note, I will describe  how to do a 1-System Upgrade Approach to upgrade an existing 11.1.0.1 Grid Control environment to OEM 12c. 
We will then see how to upgrade one of the existing 11g Management Agents to 12c.

Note that the OMS (Oracle Management Service) 12c communicates only with Oracle Management Agent 12c.

1.  The first step here is to apply the PSU on existing 11g Environment.
 At time of this application I have applied PSU 8. However currently PSU 10 is also available so you can also apply PSU 10.

Running the PSU pre-requisite

a)[oracle@Host11G software]$ export ORACLE_HOME=/u01/app/oracle/OEM11g/Middleware/oms11g
Shutdown OMS before applying the patch

b)[oracle@Host11G software]$ /u01/app/oracle/OEM11g/Middleware/oms11g/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /orabkup/oracle/software/p10065631_111010_Generic/10065631
Invoking OPatch 11.1.0.8.0

Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Patch history file: /u01/app/oracle/OEM11g/Middleware/oms11g/cfgtoollogs/opatch/opatch_history.txt
OPatch detects the Middleware Home as "/u01/app/oracle/OEM11g/Middleware"
…………………..
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.


c) Applying the PSU
Shutdown OMS before applying the patch
[oracle@Host11G bin]$ ./emctl stop oms

[oracle@Host11G 10065631]$ $ORACLE_HOME/OPatch/opatch apply

Invoking OPatch 11.1.0.8.0
Oracle Home       : /u01/app/oracle/OEM11g/Middleware/oms11g
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /u01/app/oracle/OEM11g/Middleware/oms11g/oui
Log file location : /u01/app/oracle/OEM11g/Middleware/oms11g/cfgtoollogs/opatch/opatch2013-06-28_05-32-20AM.log

Running prerequisite checks...
Patching component oracle.sysman.oms.core, 11.1.0.1.0...
Updating jar file "/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/jlib/emclikit.jar" with "/sysman/jlib/emclikit.jar/emCLI.jar/oracle/sysman/emCLI/CLILogger.class"
Copying file to "/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/plugins/connector/install_remedycntr.pl"
Copying file to "/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/omsca/scripts/wls/create_domain.py"
Copying file to "/u01/app/oracle/OEM11g/Middleware/oms11g/bin/SecureOMSCmds.pm"
Copying file to "/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/emdrep/scripts/SecureAgent_oms.pl"
ApplySession adding interim patch '10065631' to inventory

Verifying the update...
Inventory check OK: Patch ID 10065631 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 10065631 are present in Oracle Home.
Execution of 'sh /orabkup/oracle/software/p10065631_111010_Generic/10065631/custom/scripts/post -apply 10065631 ':
Return Code = 0
OPatch succeeded.
[oracle@Host11G 10065631]$


d) [oracle@Host11G 10065631]$ echo $PWD
/orabkup/oracle/software/p10065631_111010_Generic/10065631


$ORACLE_HOME/bin/rcuJDBCEngine sys/xxxx@<REPO-HOST>:1521:omsupg JDBC_SCRIPT post_install_script.sql $PWD $ORACLE_HOME

Statement Type: 'BEGIN/END Anonymous Block'
Completed SQL script execution normally.
1 scripts were processed

[oracle@Host11G 10065631]$ $ORACLE_HOME/bin/rcuJDBCEngine sys/n4u2ck@<REPO-HOST>:1521:omsupg JDBC_SCRIPT post_install_patch.sql $PWD $ORACLE_HOME

output:
Extracting Statement from File Name: 'post_install_patch.sql' Line Number: 316
Extracted SQL Statement: [truncate table bundle_component_files]
 Statement Type: 'DDL Statement'
Executing SQL statement: truncate table bundle_component_files
Extracting Statement from File Name: 'post_install_patch.sql' Line Number: 317
Extracted SQL Statement: [drop table bundle_component_files]
 Statement Type: 'DROP Statement'
Completed SQL script execution normally.
8 scripts were processed


e) [oracle@Host11G bin]$ ./emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
WARNING: Limit of open file descriptors is found to be 1024.
The OMS has been started but it may run out of descriptors under heavy usage.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.

Login to the grid control console to check whether it is running fine.


f) Then again stop the OMS

[oracle@Host11G bin]$ ./emctl stop oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

g.) Installing the Oracle Grid Control 12c Pre-upgrade Patch 

Once PSU is applied the next step is to install 12c pre-upgrade patch. This patch introduces the grid upgrade console in 11g console.


[oracle@Host11G 14375077]$ $ORACLE_HOME/OPatch/opatch apply
Invoking OPatch 11.1.0.8.0

Oracle Interim Patch Installer version 11.1.0.8.0
Copyright (c) 2009, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/OEM11g/Middleware/oms11g
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.8.0
OUI version       : 11.1.0.8.0
OUI location      : /u01/app/oracle/OEM11g/Middleware/oms11g/oui
Log file location : /u01/app/oracle/OEM11g/Middleware/oms11g/cfgtoollogs/opatch/opatch2013-06-28_05-58-07AM.log

Patch history file: /u01/app/oracle/OEM11g/Middleware/oms11g/cfgtoollogs/opatch/opatch_history.txt


OPatch detects the Middleware Home as "/u01/app/oracle/OEM11g/Middleware"

ApplySession applying interim patch '14375077' to OH '/u01/app/oracle/OEM11g/Middleware/oms11g'
Running prerequisite checks...
OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

Backing up files and inventory (not for auto-rollback) for the Oracle Home
Backing up files affected by the patch '14375077' for restore. This might take a while...
Backing up files affected by the patch '14375077' for rollback. This might take a while...


………………………………………
Verifying the update...
Inventory check OK: Patch ID 14375077 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 14375077 are present in Oracle Home.

OPatch succeeded.

g.) Run Post Upgrade scripts

[oracle@Host11G bin]$rcuJDBCEngine sys/XXX@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgdef.sql

Example -
[oracle@Host11G bin]$ rcuJDBCEngine sys/XXXX@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgdef.sql
Extracting Statement from File Name: '/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgdef.sql' Line Number: 20
………………….
 Statement Type: 'PL/SQL Statement - Object Type:PACKAGE Object Name:PRE_UPG_DB_LINKS'
Completed SQL script execution normally.
1 scripts were processed
[oracle@Host11G bin]$


[oracle@Host11G bin]$rcuJDBCEngine sys/xxxxxx@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgbody.sql

Example -
[oracle@Host11G bin]$ ./rcuJDBCEngine sys/xxxxxx@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgbody.sql
Extracting Statement from File Name: '/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/puc_dblink_pkgbody.sql' Line Number: 20
Extracted SQL Statement: [CREATE OR REPLACE PACKAGE BODY PRE_UPG_DB_LINKS
AS

Statement Type: 'PL/SQL Statement - Object Type:PACKAGE BODY Object Name:PRE_UPG_DB_LINKS'
Completed SQL script execution normally.
1 scripts were processed
[oracle@Host11G bin]$


[oracle@Host11G 14375077]$rcuJDBCEngine sys/xxxx@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/pre_upg_console.sql
.sql

Error:
>>>A SQL Error occurred!: Error creating PL/SQL Object EM_PLUGIN_UPG_SUPPORT:PACKAGE BODY
oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Error creating PL/SQL Object EM_PLUGIN_UPG_SUPPORT:PACKAGE BODY
        at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.parseNexecuteScript(JDBCEngine.java:1139)
        at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:1992)

Workaround: Use sysman account instead of sys account;

[oracle@Host11G bin]$./rcuJDBCEngine sysman/XXXXX@<REPO-HOST>:1521:omsupg JDBC_SCRIPT /u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/pre_upg_console.sql

Output:

Extracting Statement from File Name: '/u01/app/oracle/OEM11g/Middleware/oms11g/sysman/preupgc/RefreshAndValidateJob.sql' Line Number: 346
Extracted SQL Statement: [COMMIT;]
………….
Skipping Unsupported Statement
 Statement Type: 'Oracle RCU NotSupported SQLPlus Statement'
Completed SQL script execution normally.
23 scripts were processed
[oracle@Host11G bin]$


h.) Start the OMS  and check for the upgrade console

[oracle@Host11G bin]$ ./emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
WARNING: Limit of open file descriptors is found to be 1024.
The OMS has been started but it may run out of descriptors under heavy usage.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.
[oracle@Host11G bin]$

i.) [oracle@Host11G bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : (unknown)
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/OEM11g/Middleware/agent12g/agent_inst
Agent Binaries    : /u01/app/oracle/OEM11g/Middleware/agent12g/core/12.1.0.2.0
Agent Process ID  : 27785
Parent Process ID : 27620
Agent URL         : https://Host11G.searshc.com:1830/emd/main/
Repository URL    : https://Host11G.searshc.com:4900/empbs/upload
Started at        : 2013-06-28 09:34:36
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 95
Size of XML files pending upload(MB)         : 0.08
Available disk space on upload filesystem    : 36.57%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2013-06-28 09:38:45
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2013-06-28 09:39:15

---------------------------------------------------------------
Agent is Running and Ready
[oracle@Host11G bin]$

ONCE THIS IS DONE YOUR PRE-UPGRADE PART IS DONE.
NOW ONE NEEDS TO DOWNLOAD THE PLUGINS REQUIRED FOR 12c FROM OTN AND MAKE THEM AVAILABLE TO 11G OMS HOST SO THAT THEY CAN BE VALIDATED.

ONCE THEY ARE VALIDATED YOU NEED TO CHECK UPGRADE CONSOLE LINK FROM 11G OMS AND MAKE SURE IT LOOKS LIKE FOLLOWING SCREEN. THERE SHOULD BE NO YELLOW OR RED ONES.






UPGRADE TO OEM12c USING ONE SYSTEM APPROACH

HERE THE EXISTING OMS IS GOING TO GET UPDATED HENCE ENTERPRISE MONITORING WILL BE COMPLETELY DOWN DURING UPGRADE PERIOD WHICH DIRECTLY DEPENDS ON THE SIZE OF YOUR ENVIRONMENT I.E. NO OF MONITORED TARGETS.

[oracle@Host11G em12c]$ ./runInstaller
Starting Oracle Universal Installer...















Error -

The below specified Management Plug-Ins (in the format "PluginID:PluginVersion:PluginUpdate") are registered with the 12c Upgrade Console, but are not present for installation of the current Management Service. In order to proceed, you need to install the following plugins :

oracle.sysman.db:12.1.0.3.0:20130402
Ensure that you download and make the software for those Management Plug-Ins available. Once they are available, for installing these required plugins, exit the installer and invoke the installer by passing the  "-pluginLocation <absolute_path_to_plugin_archives_location>"  to the runInstaller or setup.exe
Fix – download disk1 also and unzip it 


MAKE SURE YOU ATTEND ALL THE WARNINGS.
ANY AVOIDED WARNING MAY RESULT IN FAILED ISNTALLATION.


[oracle@ushofsvpracbl18 dbs]$ !sqlp
sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Fri Jun 28 12:09:47 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> grant execute on dbms_random to dbsnmp;
Grant succeeded.

SQL> grant execute on dbms_random to sysman;
Grant succeeded.

SQL> alter user oracle_ocm account unlock identified by oracle_ocm;
User altered.
  






[root@Host11G ~]# cd /u01/app/oracle/OEM12c/Middleware/oms/
[root@Host11G oms]# ./allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/OEM12c/Middleware/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/OEM12c/Middleware/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist
/u01/app/oracle/OEM12c/Middleware/oms
Finished execution of  /u01/app/oracle/OEM12c/Middleware/oms/root.sh ......


This information is also available at:

                /u01/app/oracle/OEM12c/Middleware/oms/install/setupinfo.txt

See below for information pertaining to your Enterprise Manager installation:


Use the following URL to access:

                1. Enterprise Manager Cloud Control URL: https://Host11G.searshc.com:7799/em
                2. Admin Server URL: https://Host11G.searshc.com:7101/console

The following details need to be provided during the additional OMS install:

                1. Admin Server Hostname: Host11G.searshc.com
                2. Admin Server Port: 7101

 NOTE:
 An encryption key has been generated to encrypt sensitive data in the Management Repository. If this key is lost, all encrypted data in the Repository becomes unusable. Please run following command to backup the Management Service configuration including the emkey and keep the backup on another system:
 emctl exportconfig oms -dir <backup location>

You can either set the "oracle.sysman.prov.agentpush.enablePty" property to true in the "/u01/app/oracle/OEM12c/Middleware/oms/sysman/prov/agentpush/agentpush.properties" file, pass -S argument to the sudo command, or enable the "visiblepw" flag in the sudoers file.


[oracle@Host11G agentpush]$ pwd
/u01/app/oracle/OEM12c/Middleware/oms/sysman/prov/agentpush

[oracle@Host11G agentpush]$ vi agentpush.properties
#The value of this property decides whether a pseudo terminal needs to be allocated while executing the pdp utility over ssh
oracle.sysman.prov.agentpush.enablePty=true

THIS WILL COMPLETE YOUR ONE SYSTEM UPGRADE. 
In next post I will describe how to perform 2-system upgrade...

Thursday, January 9, 2014

Block Change Tracking Not Used Though Enabled

During normal checks of the backups, we observed the backups are taking longer than usual. These are QA databases which are not backed up thru schedules. We use Ad hoc backups for these hosts to conserve space. Also we usually take monthly full and rest of the days L1 incremental backups.  Also the change i.e. block change tracking was implemented recently on these hosts. We double checked that earlier runs were finishing with BCT in around couple of hours which has now gone back to previous run times of 7-8 hours. Strange that why this happened??!!

Following confirms what we were observing.
 BCT is enabled and alive.
Then we checked whether datafile backup is using BCT or not.

select completion_time, datafile_blocks, blocks_read, blocks, used_change_tracking
from v$backup_datafile where file#=8 order by 1;

 




We also checked the OS process, responsible for writing changed blocks to BCT file and found it running on both nodes of cluster .

[oracle@  rman]$ ps -ef | grep ctwr

oracle    6284     1  0 Jul26 ?        00:00:00 ora_ctwr_xxxxx1

Alert log also shows that process is active as well–
CTWR started with pid=65, OS id=2653

Block change tracking service is active.

So, we have everything looking healthy but why the BCT is still not used. We researched a bit on MOS and we got the reason why it was not used.

Cause –
If L1 backups are more than 8 then you need to increase the parameter _bct_bitmaps_per_file (init.or or spfile setting) from its default of 8 to the number of incremental that one take between full backups.   Also, if you have taken more than 8 incremental backups since the last full back up, you need to take another full back up before the change will kick in, because the historical bitmaps have already been deleted.

When I checked the number of L1 taken since last full backup they were in excess of 8!!  So we decided to take full backup and after that when we checked incremental, the issue was resolved.

Lesson learned –
If there are 8 or more than 8 L1 backups since last L0, the BCT will not be used for backups.


OEM Targets are in Pending State, Not Monitored

This was really wierd problem that we faced. Usually when we configure the new target in Grid Monitoring Console, it automatically detects the status post first upload from agent. Sometime it also takes more time to get the correct status reflected. 
It might also happen that one has to clear the previous state of the agent and discard the old data, which yet to uploaded to OEM repository, due to either host/agent crash or comunication link failure. One needs to shut down the agent and issue following command to get it in working order.

#<AGENT_HOME>/bin/emctl stop agent
#<AGENT_HOME>/bin/emctl clearstate agent
#<AGENT_HOME>/bin/emctl start agent

However, in this particular case it was giving us all sorts of headache. We tried following solutions to get it in working state and none of them worked. 

1. tried removing and readding targets back
2. Agent resync
3. agent clearstate
4. Adding targets with new name
5. Agent stop, wait for 10 mins and start again (This is work around suggested for bug faced in 11g OEM)

One of the solution could be completely remove the agent, reinstall and re-configure the targets, which is obviously long and tedious process. Also there is a risk of leaving production environment without monitoring, which every change management and review guys find difficult to digest. 
This happens to one of node in 8 node cluster. So the instance running on other nodes are detected and showing correct state, but on one node it always shows in pending state. Also on this particular node there are some targets in pre-prod env, hence under black out to avoid false alarms. 

So in order to identify the root cause for this, first we need to understand the reason. 
We checked the following table under sysman schema in repository database. 

select TARGET_NAME,owner,discovered_name,display_name,dynamic_property_status,broken_reason,broken_str from mgmt_targets where target_type='oracle_database' and target_name=''xxxx'

If you don't have the access on OEM repository database then, you can use following workaround, to get the data..
http(s)://host.domain:port/emd/browser/main

Another query one can use is as follows...

[oracle@/oracle/ent11gr2/agent11g/bin]: ./emctl status agent target xxxPRD,oracle_database

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Target Name : xxxPRD
Target Type : oracle_database
Current severity state
----------------------
Metric          Column name             Key             State           Timestamp
--------------------------------------------------------------------------------
DeferredTrans   errortrans_count        n/a             CLEAR           2014-01-21 03:30:04
DeferredTrans   deftrans_count          n/a             CLEAR           2014-01-21 03:30:04
UserLocks       maxBlockedSess          UL              CLEAR           2014-01-21 03:29:51
UserLocks       maxBlockedDBTime        UL              CLEAR           2014-01-21 03:29:51
UserLocks       maxBlockedSess          TX              CLEAR           2014-01-21 03:29:51
UserLocks       maxBlockedDBTime        TX              CLEAR           2014-01-21 03:29:51
UserLocks       maxBlockedSess          TM              CLEAR           2014-01-21 03:29:51
UserLocks       maxBlockedDBTime        TM              CLEAR           2014-01-21 03:29:51
Response        Status                  n/a             CLEAR           2014-01-21 03:29:39
Response        State                   n/a             CLEAR           2014-01-21 03:29:39
health_check    Status                  n/a             CLEAR           2014-01-21 03:32:12
health_check    Unmounted               n/a             CLEAR           2014-01-21 03:32:12


Once we have data about the target status, we figured out the targets were in broken state as they cannot compute properties
Hence, one needs to execute the below commands from the Agent home where the target database resides

#<AGENT_HOME>/bin/emctl reload agent dynamicproperties -upload_timeout 240 <Name of the problematic database>:rac_database

[oracle@ bin]$ ./emctl reload agent dynamicproperties -upload_timeout 240 xxxPRD:rac_database
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Skipping -upload_timeout, not in required <tgtName>:<tgtType> formSkipping 240, not in required <tgtName>:<tgtType> formEMD recompute dynprops completed successfully

Once can see that it completed successfully. One can also try following

[oracle@ bin]$ ./emctl reload agent dynamicproperties  XXXPRD:rac_database
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Now try to collect the data for RESPONSE metric.

[oracle@tryprorarac2f bin]$ ./emctl control agent runCollection xxxPRD:rac_database Response
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection error:
Response::Couldn't find collection item

So there was an error. I decided to go ahead and see, if that resolves the issue.

3). Follow the below steps from console

a) . make sure test connection in the target database page working fine.

click on "Targets/Databases"
select the database which is showing 'status pending'
click on "configure"
click on "test connection"
Make sure that the message "The connection test was successful" is received.

b). All Repository jobs are working fine
Click on setup
management services and repository
Repository Operations
check that the "Repository Scheduler Jobs Status" jobs are all running on schedule.

Even after doing this, targets were not in monitored state. Meanwhile, researching on MOS, I came across one note suggesting that if there are targets on same node in black out that may also cause issue during collection of data. 
Taking a hint from this I ended black out on all the targets. Next did the upload from agent post clearstate and Bingo ...... I see that targets are now in green i.e. monitored..

Note - Even after this, if the issue still persists, then try to remove the target and add them back manually.  I will work after that. 

Thursday, January 2, 2014



OEM Startup Issue Due to Wrong Repository

Recently our post upgrade to 12c our repository failed to start with following error.

[oracle@ushofsvpracbl3 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
WARNING: Limit of open file descriptors is found to be 1024.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.

Upon looking at , we figured out the DB was down and we started it up. However even after the DB was up the OMS was not coming up. We tried everything from listenr reconfig to reregistration. but all in vain. 

So now we suspect that what DB is needed to connect housing the repository. Following command will give you all those details. 

[oracle@ushofsvpracbl3 bin]$ ./emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
<Dec 30, 2013 6:54:42 AM CST> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>

Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=hofsvdorarac2e.intra.searshc.com)(PORT=1525)))(CONNECT_DATA=(SID=OMSUPG)))
Repository User : SYSMAN

This info is fetched from one of the property file aka emoms.properties.

[oracle@ushofsvpracbl3 gc_inst]$ pwd
/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/ocm/

[oracle@ushofsvpracbl3 gc_inst]$ vi emoms.properties


oracle.sysman.emSDK.svlt.ConsoleServerName=ushofsvpracbl3.searshc.com\:4889_Management_Service
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=TCP)(HOST\=hofsvdorarac2e.intra.searshc.com)(PORT\=1525)))(CONNECT_DATA\=(SID\=OMSUPG)))

Upon looking at this, that we figured out that there are 2 databases with same name across the different clusters!! We starting right database in wrong cluster. So once identified, we started the database correctly and OMS came up nicely. 

[oracle@ushofsvpracbl3 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Down

[oracle@ushofsvpracbl3 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Already Started
Oracle Management Server is Up
WARNING: Limit of open file descriptors is found to be 1024.
The OMS has been started but it may run out of descriptors under heavy usage.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.

[oracle@ushofsvpracbl3 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up