OEM Startup Issue Due to Wrong Repository

Recently our post upgrade to 12c our repository failed to start with following error.

[oracle@ushofsvpracbl3 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Failed to connect to repository database. OMS will be automatically restarted once it identifies that database and listener are up.
WARNING: Limit of open file descriptors is found to be 1024.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.

Upon looking at , we figured out the DB was down and we started it up. However even after the DB was up the OMS was not coming up. We tried everything from listenr reconfig to reregistration. but all in vain. 

So now we suspect that what DB is needed to connect housing the repository. Following command will give you all those details. 

[oracle@ushofsvpracbl3 bin]$ ./emctl config oms -list_repos_details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
<Dec 30, 2013 6:54:42 AM CST> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>

Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=hofsvdorarac2e.intra.searshc.com)(PORT=1525)))(CONNECT_DATA=(SID=OMSUPG)))
Repository User : SYSMAN

This info is fetched from one of the property file aka emoms.properties.

[oracle@ushofsvpracbl3 gc_inst]$ pwd
/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/ocm/

[oracle@ushofsvpracbl3 gc_inst]$ vi emoms.properties

oracle.sysman.emSDK.svlt.ConsoleServerName=ushofsvpracbl3.searshc.com\:4889_Management_Service
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(ADDRESS\=(PROTOCOL\=TCP)(HOST\=hofsvdorarac2e.intra.searshc.com)(PORT\=1525)))(CONNECT_DATA\=(SID\=OMSUPG)))

Upon looking at this, that we figured out that there are 2 databases with same name across the different clusters!! We starting right database in wrong cluster. So once identified, we started the database correctly and OMS came up nicely. 

[oracle@ushofsvpracbl3 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Down

[oracle@ushofsvpracbl3 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Already Started
Oracle Management Server is Up
WARNING: Limit of open file descriptors is found to be 1024.
The OMS has been started but it may run out of descriptors under heavy usage.
For proper functioning of OMS, please set "ulimit -n" to be at least 4096.

[oracle@ushofsvpracbl3 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up