elcarOnOsdnaH

Thursday, October 18, 2012

Cloning Oracle 11g Grid Home

Cloning oracle RAC binaries has become common practice but same is not always true with the Grid!! We recently patched one of our RAC env and need to apply the same patches to the QA.

One way was to repeat the process on env to be patched. Second was to clone env and use it for RAC DBs.
Since the Cloning Grid was not very common and was looking challenging, i decided to give it a try.

On the source node, create a copy of the Oracle grid infrastructure home. To keep the installed Oracle grid infrastructure home as a working home, make a full copy of the source Oracle grid infrastructure home and remove the unnecessary files from the copy. For example, as root on Linux systems, run the cp command:

[oracle@appsractest bin]$ cp -prf Grid_home location_of_the_copy_of_Grid_home

Delete unnecessary files from the copy.

The Oracle grid infrastructure home contains files that are relevant only to the source node, so you can remove the unnecessary files from the copy in the log, crs/init, and cdata directories. Check following to remove the unnecessary files from the copy of the Oracle grid infrastructure home:

[oracle@appsractest bin]$ cd /u01/app/11.2.0.2/grid

[oracle@appsractest grid]$ rm -rf /u01/app/11.2.0.2/grid/log/host_name
[oracle@appsractest grid]$ rm -rf crs/init
[oracle@appsractest grid]$ rm -rf cdata
[oracle@appsractest grid]$ rm -rf gpnp/*
[oracle@appsractest grid]$ rm -rf network/admin/*.ora
[oracle@appsractest grid]$ find . -name '*.ouibak' -exec rm {} \;
[oracle@appsractest grid]$ find . -name '*.ouibak.1' -exec rm {} \;
[oracle@appsractest grid]$ rm -rf root.sh*
[oracle@appsractest grid]$ rm -rf /u01/app/11.2.0.2/grid/inventory/ContentsXML/oraclehomeproperties.xml
[oracle@appsractest grid]$ cd cfgtoollogs
[root@node1 cfgtoollogs]# find . -type f -exec rm -f {} \;

Create a compressed copy of the previously copied Oracle grid infrastructure home using tar or gzip on Linux and UNIX systems. Ensure that the tool you use preserves the permissions and file time-stamps.

The steps to create a cluster through cloning are as follows:

Prepare the new cluster nodes

Deploy Oracle Clusterware on the destination nodes

Run the clone.pl script on each destination node

Prepare crsconfig_params file on all nodes

Run the orainstRoot.sh script on each node

Run the Grid_home/root.sh script

Run the configuration assistants and CVU

Step 1: Prepare the New Cluster Nodes

On each destination node, perform the following preinstallation steps:

Specify the kernel parameters

Configure block devices for Oracle Clusterware devices

Ensure that you have set the block device permissions correctly

Use short, nondomain-qualified names for all of the names in the Hosts file

Test whether the interconnect interfaces are reachable using the ping command

Verify that the VIP addresses are not active at the start of the cloning process by using the ping command (the ping command of the VIP address must fail)

Delete all files in the Grid_home/gpnp folder

Note:

If the Grid_home/gpnp folder contains any files, then cluster creation fails. All resources are added to the existing cluster, instead.

Run CVU to verify your hardware and operating system environment

-- Create the grid home directory on destination home

[root@appsractest ]## mkdir -p /u01/app/11.2.0/grid

[root@appsractest ]# cd /u01/app/11.2.0/grid

[root@appsractest ]# tar -zxvf /home/oracle/gridHome.tgz

In this example, path_name represents the directory structure in which you want to install the Oracle grid infrastructure home. Note that you can change the Grid Home location as part of the clone process

Change the ownership of all of the files to belong to the oracle:oinstall group, and create a directory for the Oracle Inventory. The following example shows the commands to do this on a Linux system:

[root@appsractest grid]# chown -R oracle:oinstall /u01/app/11.2.0/grid

[root@appsractest grid]# mkdir -p /u01/app/oraInventory

[root@appsractest grid]# chown oracle:oinstall /u01/app/oracle/oraInventory

It is important to remove any Oracle network files from the /u01/app/11.2.0.2/grid/network/admin directory on both nodes before continuing. For example, remove any tnsnames.ora, listener.ora or sqlnet.ora files.

[oracle@appsractest bin]$ /home/oracle/clone.sh

./runInstaller -clone -waitForCompletion "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/11.2.0.2/grid" "ORACLE_HOME_NAME=OraHome1Grid" "INVENTORY_LOCATION=/u01/app/oraInventory" "CLUSTER_NODES={appsractest}" "LOCAL_NODE=appsractest" -silent -noConfig -nowait

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 2047 MB Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-10-16_05-26-26AM. Please wait ...Oracle Universal Installer, Version 11.2.0.2.0 Production

You can find the log of this install session at:

/u01/app/oraInventory/logs/cloneActions2012-10-16_05-26-26AM.log

...

Performing tests to see whether nodes are available

............................................................... 100% Done.

Installation in progress (Tuesday, October 16, 2012 5:27:02 AM IST)

......................................................................... 73% Done.

Install successful

Linking in progress (Tuesday, October 16, 2012 5:27:20 AM IST)

Link successful

Setup in progress (Tuesday, October 16, 2012 5:29:49 AM IST)

................ 100% Done.

Setup successful

End of install phases.(Tuesday, October 16, 2012 5:30:10 AM IST)

WARNING:

The following configuration scripts need to be executed as the "root" user in each cluster node.

/u01/app/11.2.0.2/grid/rootupgrade.sh #On nodes appsractest

To execute the configuration scripts:

1. Open a terminal window

2. Log in as "root"

3. Run the scripts in each cluster node

The cloning of OraHome1Grid was successful.

Please check '/u01/app/oraInventory/logs/cloneActions2012-10-16_05-26-26AM.log' for more details.

-- Repeat the above steps for all the nodes in cluster

Run the script on the local node first. After successful completion, you can run the script in parallel on all the other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.

Execution of rootupgrade.sh log file -

Running Oracle 11g root script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/11.2.0.2/grid

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster execute the following command:

/u01/app/11.2.0.2/grid/crs/config/config.sh

This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

-- Run orainstRoot.sh script as follows

[root@appsractest ~]# cd /u01/app/oraInventory/

[root@appsractest oraInventory]# ./orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@appsractest oraInventory]#

[root@appsractest oraInventory]# echo Then run the following command:

Then run the following command:

-- For our process we need to prepare the following crsconfig file and then execute rootcrs.pl script as follows

-- prepare crsconfig_params file /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

[root@appsractest grid]# /u01/app/11.2.0.2/grid/perl/bin/perl -I/u01/app/11.2.0.2/grid/perl/lib -I/u01/app/11.2.0.2/grid/crs/install /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
-- so its using the file crsconfig file we created earlier...

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

OLR initialization - successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding daemon to inittab

ACFS-9200: Supported

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-2672: Attempting to start 'ora.mdnsd' on 'appsractest'

CRS-2676: Start of 'ora.mdnsd' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'appsractest'

CRS-2676: Start of 'ora.gpnpd' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'appsractest'

CRS-2672: Attempting to start 'ora.gipcd' on 'appsractest'

CRS-2676: Start of 'ora.gipcd' on 'appsractest' succeeded

CRS-2676: Start of 'ora.cssdmonitor' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'appsractest'

CRS-2672: Attempting to start 'ora.diskmon' on 'appsractest'

CRS-2676: Start of 'ora.diskmon' on 'appsractest' succeeded

CRS-2676: Start of 'ora.cssd' on 'appsractest' succeeded

Disk Group OCR_DISK creation failed with the following message:

ORA-15018: diskgroup cannot be created

ORA-15031: disk specification '/dev/hdd2' matches no disks

ORA-15025: could not open disk "/dev/hdd2"

ORA-15056: additional error message

Configuration of ASM ... failed

see asmca logs at /u01/app/oracle/cfgtoollogs/asmca for details

Did not succssfully configure and start ASM at /u01/app/11.2.0.2/grid/crs/install/crsconfig_lib.pm line 6464.

The issue here is that oracleasm_disks parameter having invalid value.
fix - modified parameter for oracleasm_disks -
from /dev/hdd1 to /dev/oracleasm/OCR_DISK disks

-- next run

[root@appsractest grid]# /u01/app/11.2.0.2/grid/perl/bin/perl -I/u01/app/11.2.0.2/grid/perl/lib -I/u01/app/11.2.0.2/grid/crs/install /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'appsractest'

CRS-2676: Start of 'ora.cssdmonitor' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'appsractest'

CRS-2672: Attempting to start 'ora.diskmon' on 'appsractest'

CRS-2676: Start of 'ora.diskmon' on 'appsractest' succeeded

CRS-2676: Start of 'ora.cssd' on 'appsractest' succeeded

You need disks from at least two different failure groups, excluding quorum disks and quorum failure groups, to create a Disk Group with normal redundancy

Configuration of ASM ... failed

-- Since I'm using external redundancy that parameter needs to get modify from NORMAL to EXTERNAL
fix - modified ASM_REDUNDANCY=EXTERNAL

-- Next run

Disk Group OCR_DISK creation failed with the following message:

ORA-15018: diskgroup cannot be created

ORA-15031: disk specification '/dev/oracleasm/disks' matches no disks

ORA-15014: path '/dev/oracleasm/disks/OCR_DISK' is not in the discovery set

ORA-15014: path '/dev/oracleasm/disks/DATA' is not in the discovery set

-- Disk discovery string needs to be fully qualified path as follows
fix - modified ASM_DISCOVERY_STRING=/dev/oracleasm/* to ASM_DISCOVERY_STRING=/dev/oracleasm/disks/*

[root@appsractest grid]# /u01/app/11.2.0.2/grid/perl/bin/perl -I/u01/app/11.2.0.2/grid/perl/lib -I/u01/app/11.2.0.2/grid/crs/install /u01/app/11.2.0.2/grid/crs/install/rootcrs.pl

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'appsractest'

CRS-2676: Start of 'ora.cssdmonitor' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'appsractest'

CRS-2672: Attempting to start 'ora.diskmon' on 'appsractest'

CRS-2676: Start of 'ora.diskmon' on 'appsractest' succeeded

CRS-2676: Start of 'ora.cssd' on 'appsractest' succeeded

ASM created and started successfully.

Disk Group OCR_DISK created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 6b227d8c9a314fe5bf3ce025c322a476.

Successfully replaced voting disk group with +OCR_DISK.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 6b227d8c9a314fe5bf3ce025c322a476 (/dev/oracleasm/disks/OCR_DISK) [OCR_DISK]

Located 1 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'appsractest'

CRS-2676: Start of 'ora.asm' on 'appsractest' succeeded

CRS-2672: Attempting to start 'ora.OCR_DISK.dg' on 'appsractest'

CRS-2676: Start of 'ora.OCR_DISK.dg' on 'appsractest' succeeded

ACFS-9200: Supported

CRS-2672: Attempting to start 'ora.registry.acfs' on 'appsractest'

CRS-2676: Start of 'ora.registry.acfs' on 'appsractest' succeeded

Preparing packages for installation...

cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

At the end of the Oracle Clusterware installation, on each new node, manually run the configuration assistants and CVU. The following commands should be run from the first node only.

[oracle] $ /u01/app/11.2.0/grid/bin/netca \

/orahome /u01/app/11.2.0/grid \

/orahnam OraGridHome1 \

/instype typical \

/inscomp client,oraclenet,javavm,server\

/insprtcl tcp \

/cfg local \

/authadp NO_VALUE \

/responseFile /u01/app/11.2.0/grid/network/install/netca_typ.rsp \

/silent

[oracle@appsractest bin]$ /u01/app/11.2.0.2/grid/bin/netca /orahome /u01/app/11.2.0.2/grid /orahnam OraHome1Grid /instype typical /inscomp client,oraclenet,javavm,server /insprtcl tcp /cfg local /authadp NO_VALUE /responseFile /u01/app/11.2.0.2/grid/network/install/netca_typ.rsp /silent

Parsing command line arguments:

Parameter "orahome" = /u01/app/11.2.0.2/grid

Parameter "orahnam" = OraHome1Grid

Parameter "instype" = typical

Parameter "inscomp" = client,oraclenet,javavm,server

Parameter "insprtcl" = tcp

Parameter "cfg" = local

Parameter "authadp" = NO_VALUE

Parameter "responsefile" = /u01/app/11.2.0.2/grid/network/install/netca_typ.rsp

Parameter "silent" = true

Done parsing command line arguments.

Oracle Net Services Configuration:

Oracle Net Listener Startup:

Listener started successfully.

Listener configuration complete.

Oracle Net Services configuration successful. The exit code is 0

-- If you use a supported cluster file system, then you do not need to execute this command. If you are using Oracle ASM, then run the following command:

[oracle] $ /u01/app/11.2.0/grid/bin/asmca -silent -postConfigureASM -sysAsmPassword oracle -asmsnmpPassword oracle

-- Setup Oracle ASM password..

[oracle@appsractest bin]$ /u01/app/11.2.0.2/grid/bin/asmca -silent -postConfigureASM -sysAsmPassword oracle -asmsnmpPassword oracle

PostConfiguration completed successfully

Now if you check your Grid with CRS is up and running. Next step will be to clone Oracle RAC Binaries or Install them as per your need. I will put that topic in my next post.
One last thing is that you have to repeat this process for all the nodes on your destination cluster, so make sure you follow all the above mentioned steps. You can do some of this steps at a time on multiple nodes but doing them one by one will give you better control and ease of trouble shooting.

Note - If you a plan to run a pre-11g release 2 (11.2) database on this cluster, then you should run oifcfg as described in the Oracle Database 11g release 2 (11.2) documentation

Friday, October 12, 2012

PRKH-1010 : Unable to Communicate With CRS Services During Listener Configuration

After installing the Oracle Binaries for stand alone server when I tried to manually create the listener after Net configuration assistant failed during installation, I hit following error.
Though it was surprising that it was showing error of CRS on stand alone server!!!

This was due to the bug with 11201 binaries which was causing this issue.
[oracle@oem ~]$ cd /u01/app/oracle/product/11.0/db_1/bin/
[oracle@oem bin]$ ./netca
oracle.ops.mgmt.cluster.ClusterException: PRKH-1010 : Unable to communicate with CRS services.
[Communications Error(Native: prsr_initCLSS:[3])]
at oracle.ops.mgmt.cluster.GetActiveNodes.<init>(GetActiveNodes.java:179)
at oracle.ops.mgmt.cluster.GetActiveNodes.create(GetActiveNodes.java:221)
at oracle.ops.mgmt.cluster.Cluster.getNodes(Cluster.java:313)
at oracle.sysman.assistants.util.hasi.HAUtils.getNodes(HAUtils.java:1620)
at oracle.sysman.assistants.util.hasi.HAUtils.getAccessibleNodes(HAUtils.java:1638)
at oracle.net.ca.InitialSetup.configureOPS(NetCA.java:4526)
at oracle.net.ca.InitialSetup.<init>(NetCA.java:4024)
at oracle.net.ca.NetCA.main(NetCA.java:405)

Fix -
This was due to ocr.loc file pointing to Shared device or ASM which is not hosted locally.
so the ideas is to modify this behavior with local_only parameter set to TRUE.

[root@oem ~]# vi /etc/oracle/ocr.loc

ocrconfig_loc=+DATA

#local_only=FALSE
local_only=TRUE

Once this done, when you invoke the configuration assistance this time around it works fine...

[oracle@oem bin]$ ./netca

Oracle Net Services Configuration:

Configuring Listener:LISTENER

Listener configuration complete.

Oracle Net Services configuration successful. The exit code is 0

[oracle@oem bin]$ ps -ef | grep lsnr

oracle 10886 1 0 15:16 ? 00:00:00 /u01/app/oracle/product/11.0/db_1/bin/tnslsnr LISTENER -inherit

oracle 12089 5982 0 15:24 pts/3 00:00:00 grep lsnr

[oracle@oem bin]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 11-OCT-2012 15:24:30

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oem.localdomain)(PORT=1521)))

STATUS of the LISTENER

------------------------

Alias LISTENER

Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production

Start Date 11-OCT-2012 15:16:36

Uptime 0 days 0 hr. 7 min. 54 sec

Trace Level off

Security ON: Local OS Authentication

SNMP OFF

Listener Log File /u01/app/oracle/diag/tnslsnr/oem/listener/alert/log.xml

Listening Endpoints Summary...

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem.localdomain)(PORT=1521)))

Services Summary...

Service "oemdb" has 1 instance(s).

Instance "oemdb", status READY, has 1 handler(s) for this service...

Service "oemdbXDB" has 1 instance(s).

Instance "oemdb", status READY, has 1 handler(s) for this service...

The command completed successfully

Tuesday, October 9, 2012

Forms Run-time Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications/EBS

Release 12 (12.1x and 12.2) now includes Oracle Application Server (OracleAS) 10g 10.1.2 as the foundation for Forms, Reports, and C code. This replaces the 8.0.6-based ORACLE_HOME that 9iAS 1.0.2.2 provides today in Release 11i.

The Forms Runtime Diagnostic (FRD) is a method for capturing all events that occur in a form session and writing a file with the details of those events. There is overhead in writing this file, so you should only use this for the purposes of development and debugging

When a form is run with FRD enabled, a combination of external user-application interactions and internal Forms processing events are written in chronological order to a log on the file system.

Activating Forms Runtime Diagnostics (FRD)

Forms Runtime Diagnostics (FRD) can be enabled /activated in one of the following ways:

Option 1: Obtain FRD Trace Using Profile Options
Option 2: Obtain FRD Trace Using Appsweb.cfg

Option 1: Obtain FRD Trace Using Profile Options

When accessing Applications through the 'Home Page' (the page resultant after successful login) there is no way to pass command line parameters such as record=all, etc. for the duration of a single session. This method enables users to run FRD tracing through the 'Home Page' by using the profile option 'ICX: Forms Launcher'.

To start tracing in this way, modify Profile Option 'ICX: Forms Launcher' at USER level to include the Forms parameters required for FRD logging:

Note:
* If the 'ICX: Forms Launcher' profile option is not updatable at the USER level, then you may need to access 'Application Developer' to set the option 'Updatable' for the profile 'ICX_FORMS_LAUNCHER'
Step 1: Set the profile option 'ICX: Forms Launcher' at user level to same as site value
E.g. copy the site level value for profile option 'ICX: Forms Launcher' and paste it in user level field so as to set it at the user level

Step 2: Append the user value of ICX: Forms Launcher with the Forms parameters for FRD i.e '?record=collect'

http://hostname.domain:port/forms/frmservlet?record=collect

Step 3: Save this change at the USER level.
Step 4: For trace file details and location
The FRD log file will be written in the directory pointed by environment variable $FORMS_TRACE_DIR. By default, the trace file with name collect_<pid> gets written in $FORMS_TRACE_DIR , where <pid> is the process identifier.This is a simple text file and can be viewed directly

Option 2: Obtain FRD Trace Using Appsweb.cfg

When accessing Applications through the 'Home Page' (the page resultant after successful login) there is no way to pass command line parameters such as record=forms/all, etc. for the duration of a single session. It is also possible to obtain FRD trace by using Appsweb.cfg file.

To start tracing in this way, please refer to the steps below:

Step 1: Edit the appsweb.cfg file pointed by environment variable "FORMS_WEB_CONFIG_FILE"

Step 2: Forms parameters for FRD can be set in appsweb.cfg file in one of the following ways :

Look for "record=" parameter after the 'ENVIRONMENT SPECIFIC PARAMETERS' section and set it to collect as shown below.
Once done login into Oracle Applications and launch forms via self-service.

Extracts from appsweb.cfg file :

# Sub argument for otherparams
record=collect

In R12, the FRD log file will be written in the directory pointed by environment variable $FORMS_TRACE_DIR and one can specify the log filename with '.log'extension in one of the following ways:

1. In appsweb.cfg:

Look for "log=" parameter after the 'ENVIRONMENT SPECIFIC PARAMETERS' section and set it with name that you want the trace file to be created with.

E.g

log=user1.log

2. Using +log directive in the URL used to set the frd trace via profile options or in ad hoc way

E.g

http://hostname.domain:port/forms/frmservlet?record=collect+log=user1.log

Thursday, October 4, 2012

Forms EAR Redeployment with txkrun.pl -script=CfgOC4JApp Failed

During OPMN startup the oaforms deployment failed and found the note suggesting the redeployment of the forms as follows

[oracle@appsnode_new config]$ txkrun.pl -script=CfgOC4JApp
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/rgf/TXK/txkCfgOC4JApp_Fri_Aug_3_15_44_56_2012.log
Program : /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl started @ Fri Aug 3 15:44:56 2012
*** Log File = /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/rgf/TXK/txkCfgOC4JApp_Fri_Aug_3_15_44_56_2012.log
Enter Application name for re-deployment ? forms
Enter Oc4j Instance password for re-deployment ?
Run Autoconfig <Yes/No> ? No
*****************************************************
Required values for starting OC4J instance "forms":
====================================================
s_formsstatus = enabled
s_forms_nprocs = 1 (value should be greater than 0)

Existing values from the context file:
======================================
s_formsstatus = enabled
s_forms_nprocs = 1
----------------------------------------------
*** Values for context variables are VALID ***
----------------------------------------------
*****************************************************
Stopping all OPMN processes.
OPMN stopped.
Errors encountered running /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl

*******FATAL ERROR*******
PROGRAM : /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl(/u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/bin/txkrun.pl)
TIME : Fri Aug 3 15:45:11 2012
FUNCTION: TXK::Process::run [ Level 3 ]
MESSAGES:
Command error: <rc> = 32512, <command> = /u01/app/oracle/visr12/apps/tech_st/10.1.3/opmn/bin/opmnctl start

STACK TRACE shows following...
TXK::Error::abort('TXK::Error','HASH(0x9971160)') called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 299
TXK::Common::doError('TXK::Process=HASH(0xa8f2998)','Command error: <rc> = 32512, <command> = /u01/app/oracle/visr...','undef') called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/Common.pm line 314
TXK::Common::setError('TXK::Process=HASH(0xa8f2998)','Command error: <rc> = 32512, <command> = /u01/app/oracle/visr...') called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/Process.pm line 449
TXK::Process::run('TXK::Process=HASH(0xa8f2998)','HASH(0x9eb33f0)') called at /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 1587
TXK::RunScript::execOPMNControl('HASH(0xa17e4ac)') called at /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl line 599
require /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/RunScript.pm line 105
TXK::RunScript::require('TXK::RunScript','/u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115...') called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177
eval {...} called at /u01/app/oracle/visr12/apps/apps_st/appl/au/12.0.0/perl/TXK/Script.pm line 177
TXK::Script::run('TXK::Script=HASH(0xa260a00)','/u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/app...','/u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115...') called at /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/bin/txkrun.pl line 174

Workaround :
This happens due to the patch mismatch or ambiguity in txkCfgOC4JApp.pl in Perl script

1. Backup file $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl .
[oracle@appsnode_new bin]$ cp $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl $FND_TOP/patch/115/bin/txkCfgOC4JApp.pl.bk
2. Replace the following line as follows :
$proc ->run ( { command => { text => "$oh/opmn/bin/opmnctl",
To
$proc ->run ( { command => { text => "/<BASE_PATH>/<SID>/inst/apps/<SID>_<hostname>/admin/scripts/adopmnctl.sh",
i.e.
As per in my env..
$proc ->run ( { command => { text => "/u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/admin/scripts/adopmnctl.sh",
3. Rerun Rerun command
[oracle@appsnode_new bin]$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp

[oracle@appsnode_new bin]$ $FND_TOP/bin/txkrun.pl -script=CfgOC4JApp

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS

*** Log File = /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/rgf/TXK/txkCfgOC4JApp_Fri_Aug_3_16_08_01_2012.log

Program : /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl started @ Fri Aug 3 16:08:01 2012

*** Log File = /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/logs/appl/rgf/TXK/txkCfgOC4JApp_Fri_Aug_3_16_08_01_2012.log

Enter Application name for re-deployment ? forms

Enter Oc4j Instance password for re-deployment ?

Run Autoconfig <Yes/No> ? No

*****************************************************

Required values for starting OC4J instance "forms":

====================================================

s_formsstatus = enabled

s_forms_nprocs = 1 (value should be greater than 0)

Existing values from the context file:

======================================

s_formsstatus = enabled

s_forms_nprocs = 1

----------------------------------------------

** Values for context variables are VALID ***

*****************************************************

Stopping all OPMN processes.

OPMN stopped.

OPMN started.

Deplolying Application : "forms" onto OC4J instance: "forms"

Application deployed successfully.

Stopping and starting OC4J instances.

Started OC4J instances.

Binding webApp : "forms" with webmodule : "formsweb" for OC4J instance: "forms"

Web application bound successfully.

Stopping OPMN.

OPMN stopped.

Program : /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl completed @ Fri Aug 3 16:08:41 2012

End of /u01/app/oracle/visr12/apps/apps_st/appl/fnd/12.0.0/patch/115/bin/txkCfgOC4JApp.pl : No Errors encountered

[oracle@appsnode_new bin]$ adautocfg.sh

Enter the APPS user password:

The log file for this session is located at: /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/admin/log/08031615/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.

Using CONFIG_HOME location : /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new

Classpath : /u01/app/oracle/visr12/apps/apps_st/comn/java/lib/appsborg2.zip:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes

Using Context file : /u01/app/oracle/visr12/inst/apps/visr12_appsnode_new/appl/admin/visr12_appsnode_new.xml

Context Value Management will now update the Context file.

Since Auto Config ran successfully this the opmn will start fine too....

Sunday, September 23, 2012

Node Reboot or Shutdown Skipped to Stop 11gR2 Grid Infrastructure

During maintenance in one of our RAC env , the node was rebooted without bringing down the grid manually (as GI automatically stops all its processes automatically when it detects the node shutting down).

Upon start up we had issues of starting up ASM instance and due to this our CSS and then on CRS was not coming up hence grid was un-operational.

During research it was revealed that this was due to the unpublished bug 8740030. Due to this bug, while rebooting a node, command K19ohasd in /etc, which suppose to stop Grid Infrastructure, will be skipped as /var/lock/subsys/ohasd* doesn't exist:

# ls -l /var/lock/subsys/ohasd* | wc -l

output: 0

Looking at the logs reveals following...

CSS Logs -

[ CSSD][4105858816]clssscProcessKillShutdown: Initiating shutdown due to process kill

[ CSSD][4105858816]###################################

[ CSSD][1145833792]clssgmSendShutdown: Aborting client (0x2aaaac01c850) proc (0x90f66c0), iocapables 1.

ASM Logs -

ORA-29746: Cluster Synchronization Service is being shut down.

ORA-29702: error occurred in Cluster Group Service operation

GMON (ospid: 6595): terminating the instance due to error 29746

Instance terminated by GMON, pid = 6595

Fix -

To fix the issue, one has to modify /etc/init.d/ohasd

1. From:

Linux)

LOGMSG="$LOGGER -puser.err"

LOGERR="$LOGGER -puser.alert"

;;

To:

Linux)

LOGMSG="$LOGGER -puser.err"

LOGERR="$LOGGER -puser.alert"

SUBSYSFILE="/var/lock/subsys/ohasd"

;;

2. From:

start()

{

$ECHO -n $"Starting $PROG: "

To:

start()

{

case `/bin/uname` in

Linux)

/bin/touch $SUBSYSFILE

;;

esac

$ECHO -n $"Starting $PROG: "

3. From:

stop()

{

$ECHO -n "Stopping Oracle Clusterware stack"

}

To:

stop()

{

case `/bin/uname` in

Linux)

$RMF $SUBSYSFILE

;;

esac

$ECHO -n "Stopping Oracle Clusterware stack"

}

Once /etc/init.d/ohasd is modified, please execute following command before reboot the node:

#/bin/touch /var/lock/subsys/ohasd

Once this is done, try to reboot the node again without shutting down the GI and see if it stops gracefully or not. I tested it and it worked fine this time.

Friday, September 21, 2012

Apps/EBS R12: How to Diagnose Start-up Problems for Apache

When using the adapcctl.sh script to start Apache in Release 12 this fails with an error like:
12/07/12-15:32:56 :: adapcctl.sh: starting OPMN managed OHS instance
opmnctl: starting opmn managed processes...
================================================================================
opmn id=xxx.yyy.com:6202
0 of 1 processes started.
ias-instance id=DEV_dev.dev.xxx.yyy
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
HTTP_Server/HTTP_Server/HTTP_Server/
Error
--> Process (index=1,uid=1561342258,pid=14927)
failed to start a managed process after the maximum retry limit
Log:
/oracle/dev/inst/apps/DEV_dev/logs/ora/10.1.3/opmn/HTTP_Server~1.log
12/07/11-15:33:00 :: adapcctl.sh: exiting with status 204
================================================================================
The log file HTTP_Server~1.log referenced does not exactly reports a root-cause for the failure, so further analysis is required to identify what prevents Apache from being started.
The adapcctl.sh is essentially a wrapper script calling the native Apache starting command. To identify what is preventing the Apache from being started the approach is run this command direct. For this some additional actions are needed.

1. Run the <SID>_<host>.env location in $INST_TOP/ora/10.1.3 directory
# . ./$INST_TOP/ora/10.1.3/<SID>_<host>.env
This sets the $ORACLE_HOME to the AS10G 10.1.3 HOME (instead of AS10G 10.1.2 HOME), so relevant settings picked from the right AS10G HOME.

2. Run the following command:
# $INST_TOP/ora/10.1.3/Apache/Apache/bin/apachectl configtest -f $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf
This validates the httpd.conf configuration file used by Apache. If there are errors raised for this step it appears the httpd.conf may be corrupted/misconfigured and this prevents Apache from being started. Resolve any problems reported (e.g. by running Autoconfig to have the configuration being recreated) and retest. If the commands responds with an OK proceed with the next step.

3. Run the following command:
# $INST_TOP/ora/10.1.3/Apache/Apache/bin/apachectl startsll -f $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf
This starts the Apache server direct instead of using OPMN. This could expose errors not observed easily when Apache is started as OPMN service, so can assist in finding out why Apache can not be started.
After this command completes it's expected to see number of httpd processes while running:
# ps -ef | grep httpd
If this still does not show any obvious errors the next step is to run the same command and run strace/truss/tusc to see what OS calls are executed.
The below example uses strace command available on Linux platform. Check OS documentation for exact parameters to be used for the utility on the platform used.

4. Run the following command:
# strace -o startapache.trc -ff -t $INST_TOP/ora/10.1.3/Apache/Apache/bin/apachectl startsll -f $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf &

This command saves the output in startapache.trc and on Linux the -ff makes that each child process started is logged in separate log file where the <PID> is added to the file name.
Review the trace files for errors reported. If useful it may be an option to collect the same from similar instance not having the problem so trace files can be compared. The OS calls logged in the trace file may expose problems in areas like:
Opening files required for Apache to run (missing, privileges)
Creating or updating (log/pid) files (privileges, size of log file hitting 2GB limit)
Memory issues

5. After root-cause has been identified and issue is resolved so direct start works fine run the following command to stop Apache service
# $INST_TOP/ora/10.1.3/Apache/Apache/bin/apachectl stop -f $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf
Then use the adapcctl.sh script to confirm that Apache now also starts using the recommended way and confirm this also works fine:
# $INST_TOP/admin/scripts/adapcctl.sh start

Friday, September 14, 2012

Issues during recreating DB Control Console in 11gR2

Error -
[oracle@devtstdbsrv bin]$ ./emca -config dbcontrol db -repos recreate
INFO: Stopping Database Control (this may take a while) ...
Sep 13, 2012 12:00:22 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Listener is not up or database service is not registered with it. Start the Listener and register database service and run EM Configuration Assistant again .
Refer to the log file at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_2012_09_12_23_59_03.log for more details.
Could not complete the configuration. Refer to the log file at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_2012_09_12_23_59_03.log for more details.

Since, I cannot drop the re
[oracle@devtstdbsrv bin]$ hostname
[oracle@devtstdbsrv bin]$ emca -deconfig dbcontrol db
STARTED EMCA at Sep 13, 2012 12:04:04 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:
Database SID: orcl
Do you wish to continue? [yes(Y)/no(N)]: yes
Sep 13, 2012 12:04:11 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_2012_09_13_00_04_04.log.
Sep 13, 2012 12:04:11 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Enterprise Manager configuration completed successfully
FINISHED EMCA at Sep 13, 2012 12:05:07 AM

-- However drop repos operation failed with following error.
[oracle@devtstdbsrv bin]$ emca repos drop
....................................
SEVERE: Error creating the repository
Sep 13, 2012 12:35:19 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Refer to the log file at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_repos_create_<date>.log for more details.

Looking at the log file reveals following error...

CONFIG: ORA-20001: SYSMAN already EXISTS..
ORA-06512: at line 17
oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-20001: SYSMAN already EXISTS..
ORA-06512: at line 17

Work around -

Perform following actions in your DB
SQL> drop user sysman cascade;
User dropped.
SQL> DROP PUBLIC SYNONYM setemviewusercontext;
Synonym dropped.
SQL> DROP ROLE mgmt_user;
Role dropped.
SQL> DROP PUBLIC SYNONYM mgmt_target_blackouts;
Synonym dropped.
SQL> DROP USER mgmt_view;
User dropped.

There is one more thing to add here is that since my repos drop operation failed earlier I have to manually drop the repos using following method..

[oracle@devtstdbsrv bin]$ pwd
/u01/app/oracle/product/11.0/db_1/sysman/admin/emdrep/bin
[oracle@devtstdbsrv bin]$ ./RepManager devtstdbsrv 1521 orcl -action drop
Enter SYS user's password :
Enter repository user name : sysman
Getting temporary tablespace from database...
Found temporary tablespace: TEMP
Checking SYS Credentials ... rem error switch
OK.
rem error switch
Dropping the repository..
Checking for Repos User ... Exists.
Repos User exists..
Clearing EM Contexts ... OK.
Dropping EM users ...
Done.
Dropping Repos User ...

Dropping Roles/Synonymns/Tablespaces ... Done.

Dropped Repository Successfully.

If this is not done then later during create repos operation you will face following error..

CONFIG: ORA-00955: name is already used by an existing object
oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-00955: name is already used by an existing object

Now try to re-create the repository once again. We will try to create first repository and then db-control in separate steps as follows.

[oracle@devtstdbsrv dbs]$ emca -repos create
STARTED EMCA at Sep 13, 2012 12:53:10 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: orcl
Listener port number: 1531
Password for SYS user:
Password for SYSMAN user:
Do you wish to continue? [yes(Y)/no(N)]: yes
Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_2012_09_13_00_53_10.log.
Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...

Sep 13, 2012 2:05:44 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Enterprise Manager configuration completed successfully
FINISHED EMCA at Sep 13, 2012 2:05:45 AM

--If you check the logfile during this time, you will see following messages

Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngine
CONFIG: isLocalConnectionRequired: true. Connecting to database instance locally.
Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngineLoacly
CONFIG: SQLEngine connecting with SID: orcl, oracleHome: /u01/app/oracle/product/11.0/db_1, and user: SYS
Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngineLoacly
CONFIG: SQLEngine created successfully and connected
Sep 13, 2012 12:53:21 AM oracle.sysman.emcp.EMReposConfig createRepository

So the listener issue as described earlier during drop operation was not really a issue. It was more like a place holder error rather than actual error!!!

Now one needs to config the db-control as follows
[oracle@devtstdbsrv dbs]$ emca -config dbcontrol db

STARTED EMCA at Sep 13, 2012 2:40:50 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: orcl
The database orcl is already being monitored by central agent(s)
Database Control will monitor the exisiting targets
Do you wish to continue? [yes(Y)/no(N)]: yes
Listener ORACLE_HOME [ /u01/app/oracle/product/11.0/db_1 ]:
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------
You have specified the following settings
Database ORACLE_HOME ................ /u01/app/oracle/product/11.0/db_1
Local hostname ................ devtstdbsrv.localdomain
Listener ORACLE_HOME ................ /u01/app/oracle/product/11.0/db_1
Listener port number ................ 1521
Database SID ................ orcl
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: yes
Sep 13, 2012 2:41:05 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/orcl/emca_2012_09_13_02_40_49.log.
Sep 13, 2012 2:41:13 AM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...

Sep 13, 2012 2:42:39 AM oracle.sysman.emcp.EMReposConfig invoke

INFO: Uploaded configuration data successfully

Sep 13, 2012 2:42:46 AM oracle.sysman.emcp.util.DBControlUtil configureSoftwareLib

INFO: Software library configured successfully.

Sep 13, 2012 2:42:46 AM oracle.sysman.emcp.EMDBPostConfig configureSoftwareLibrary

INFO: Deploying Provisioning archives ...

INFO: Provisioning archives deployed successfully.

Sep 13, 2012 2:44:33 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole

INFO: Securing Database Control (this may take a while) ...

Sep 13, 2012 2:44:48 AM oracle.sysman.emcp.util.PlatformInterface executeCommand

WARNING: Error executing /u01/app/oracle/product/11.0/db_1/bin/emctl secure dbconsole -host devtstdbsrv.localdomain -sid orcl

Sep 13, 2012 2:44:48 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration

WARNING: Error securing Database control.

Sep 13, 2012 2:44:48 AM oracle.sysman.emcp.EMDBPostConfig setWarnMsg

INFO: Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):

1) Set the environment variable ORACLE_SID to orcl

2) /u01/app/oracle/product/11.0/db_1/bin/emctl stop dbconsole

3) /u01/app/oracle/product/11.0/db_1/bin/emctl config emkey -repos -sysman_pwd < Password for SYSMAN user >

4) /u01/app/oracle/product/11.0/db_1/bin/emctl secure dbconsole -sysman_pwd < Password for SYSMAN user >

5) /u01/app/oracle/product/11.0/db_1/bin/emctl start dbconsole

To secure Em Key, run /u01/app/oracle/product/11.0/db_1/bin/emctl config emkey -remove_from_repos -sysman_pwd < Password for SYSMAN user >

Sep 13, 2012 2:44:48 AM oracle.sysman.emcp.util.DBControlUtil startOMS

INFO: Starting Database Control (this may take a while) ...

Sep 13, 2012 2:47:55 AM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /u01/app/oracle/product/11.0/db_1/bin/emctl start dbconsole
Sep 13, 2012 2:47:55 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://devtstdbsrv.localdomain:1158/em <<<<<<<<<<<
Sep 13, 2012 2:47:55 AM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING: Error starting Database Control.Please execute the following command(s).

1) Set the environment variable ORACLE_UNQNAME to Database unique name
2) /u01/app/oracle/product/11.0/db_1/bin/emctl start dbconsole

Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):

1) Set the environment variable ORACLE_SID to orcl
2) /u01/app/oracle/product/11.0/db_1/bin/emctl stop dbconsole
3) /u01/app/oracle/product/11.0/db_1/bin/emctl config emkey -repos -sysman_pwd < Password for SYSMAN user >
4) /u01/app/oracle/product/11.0/db_1/bin/emctl secure dbconsole -sysman_pwd < Password for SYSMAN user >
5) /u01/app/oracle/product/11.0/db_1/bin/emctl start dbconsole

To secure Em Key, run /u01/app/oracle/product/11.0/db_1/bin/emctl config emkey -remove_from_repos -sysman_pwd < Password for SYSMAN user >
Error starting Database Control.Please execute the following command(s).
1) Set the environment variable ORACLE_UNQNAME to Database unique name
2) /u01/app/oracle/product/11.0/db_1/bin/emctl start dbconsole

I ran the commands from 1 thru 5 and my dbconsole was up and running...