Tuesday, June 16, 2015

Apply PSU6 Manually


In this post I am describing how to manually apply the PSU patch on both GI home and RAC Home respectively. 

when one uses opatch auto the whole process is automated which we are going to do manually. Opatch will check the applicability of the each component of the PSU patch and apply it on GI OR RAC home accordingly. 


##
## APPLY PSU 6 i.e. patch 20834621
##
PSU 6 has following patch components. One has to apply this components both on GI home and RAC Home. 
Before starting the patch process, make sure you dont have any conflicts with prior applied patches.


GI PSU - 
GI PSU component         - 20485808
DB PSU - 20299013
OCW Component - 20420937
ACFS Component - 20299019 

Process - 

root@host301[/ora00/app/11.2.0.4/grid/crs/install]# ./roothas.pl -unlock
Using configuration parameter file: ./crsconfig_params

Successfully unlock /ora00/app/11.2.0.4/grid


Run the pre root script. If this is a GI Home, as the root user execute:
# /ora00/app/11.2.0.4/grid/crs/install/roothas.pl -unlock



##
## RUN SLIBCLEAN AS ROOT( only for AIX )
##
root@host401[/ora00/app/11.2.0.4/grid]# /usr/sbin/slibclean


PATCH - 20834621

Apply the CRS patch using grid user. As the GI home owner execute:

$ /ora00/app/11.2.0.4/grid/OPatch/opatch apply -oh /ora00/app/11.2.0.4/grid -local /11204_PSU6/20834621/20485808/20420937

Patching component oracle.crs, 11.2.0.4.0...

Verifying the update...
Patch 20420937 successfully applied
Log file location: /ora00/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-05-01_18-12-49PM_1.log

OPatch succeeded.

##

As the GI home owner execute:
$ /ora00/app/11.2.0.4/grid/OPatch/opatch apply -oh /ora00/app/11.2.0.4/grid -local /11204_PSU6/20834621/20485808/20299019

Patching component oracle.usm, 11.2.0.4.0...

Verifying the update...
Patch 20299019 successfully applied
Log file location: /ora00/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-05-01_18-23-01PM_1.log

OPatch succeeded.

##


As the GI home owner execute:
$ /ora00/app/11.2.0.4/grid/OPatch/opatch apply -oh /ora00/app/11.2.0.4/grid -local /11204_PSU6/20834621/20485808/20299013

Composite patch 20299013 successfully applied.
OPatch Session completed with warnings.
Log file location: /ora00/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-05-01_18-24-59PM_1.log

OPatch completed with warnings.

Composite patch 20299013 successfully applied.
OPatch Session completed with warnings.
Log file location: /ora00/app/11.2.0.4/grid/cfgtoollogs/opatch/opatch2015-05-01_18-24-59PM_1.log

OPatch completed with warnings.


AS ORACLE USER - 

Run the pre script for DB component of the patch. As the database home owner execute:
$ /11204_PSU6/11204_PSU6/20834621/20485808/20420937/custom/scripts/prepatch.sh -dbhome /ora01/app/oracle/product/11.2.0.4/db_1

Apply the DB patch. As the database home owner execute:

$ /ora01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch apply -oh /ora01/app/oracle/product/11.2.0.4/db_1 -local /11204_PSU6/11204_PSU6/20834621/20485808/20420937/custom/server/20420937

Patching component oracle.rdbms, 11.2.0.4.0...

Verifying the update...
Patch 20420937 successfully applied
Log file location: /ora01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2015-05-01_19-33-25PM_1.log

OPatch succeeded.

$ /ora01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch apply -oh /ora01/app/oracle/product/11.2.0.4/db_1 -local /11204_PSU6/20834621/20485808/20299013

Composite patch 20299013 successfully applied.
OPatch Session completed with warnings.
Log file location: /ora01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2015-05-01_19-35-30PM_1.log

OPatch completed with warnings.



Run the post script for DB component of the patch. As the database home owner execute:
$ /11204_PSU6/20834621/20485808/20420937/custom/scripts/postpatch.sh -dbhome /ora01/app/oracle/product/11.2.0.4/db_1

Reapplying file permissions on /ora01/app/oracle/product/11.2.0.4/db_1/srvm/admin/ractrans
Reapplying file permissions on /ora01/app/oracle/product/11.2.0.4/db_1/srvm/admin/getcrshome
Reapplying file permissions on /ora01/app/oracle/product/11.2.0.4/db_1/bin/gnsd
Reapplying file permissions on /ora01/app/oracle/product/11.2.0.4/db_1/bin/crsdiag.pl
Postpatch completed successfully


Run the post script. As the root user execute:
# /ora00/app/11.2.0.4/grid/rdbms/install/rootadd_rdbms.sh

-- 
If this is a GI Home, as the root user execute:
# /ora00/app/11.2.0.4/grid/crs/install/roothas.pl -patch

Using configuration parameter file: /ora00/app/11.2.0.4/grid/crs/install/crsconfig_params
ACFS driver install actions failed
CRS-4123: Oracle High Availability Services has been started.

The following steps load modified SQL files into the database. For an Oracle RAC environment, perform these steps on only one node.

Install the SQL portion of the patch by running the following command. For an Oracle RAC environment, reload the packages on one of the nodes.

# cd $ORACLE_HOME/sqlpatch/20406239
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> startup upgrade
SQL> @postinstall.sql
SQL> shutdown
SQL> startup 

Now run the following script for each database running out of this RDBMS Home.

SQL> $ORACLE_HOME/rdbms/admin/catbundle.sql PSU APPLY 

After installing the SQL portion of the patch, some packages could become INVALID. This will get recompiled upon access or you can run utlrp.sql to get them back into a VALID state.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> @utlrp.sql

Hope this helps...

Friday, June 5, 2015


Convert Oracle RAC Cluster to RAC One 


During one of the recent deployments, we came across the scenario, where we have to convert out two node RAC cluster to One Node RAC One node.
We tried to find proper MOS note but could not find any, though there are plenty of notes to convert the otherwise i.e. from RAC One to RAC.

So I thought it will be good to put up a note with all the detailed information. 

NodeA -

SYS@racon1 SQL> sho parameter service

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
service_names                        string      racon


$ srvctl config database -d racon
Database unique name: racon
Database name: racon
Oracle home: /ora01/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DATA1/racon/spfileracon.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: racon
Database instances: racon1,racon2
Disk Groups: DATA1,ARCH1,REDO1,REDO2
Mount point paths:
Services:
Type: RAC
Database is administrator managed

$ crsctl stat res  | grep ora.racon.db
NAME=ora.racon.db
< NodeA : racon1 : oracle : /users/oracle >
$ crsctl stat res ora.racon.db -p
NAME=ora.racon.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,group:oinstall:r-x,user:grid:r-x
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=2
.................
.................
GEN_START_OPTIONS@SERVERNAME(NodeA)=open
GEN_START_OPTIONS@SERVERNAME(NodeB)=open
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(NodeA)=racon1
GEN_USR_ORA_INST_NAME@SERVERNAME(NodeB)=racon2
HOSTING_MEMBERS=
INSTANCE_FAILOVER=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
ONLINE_RELOCATION_TIMEOUT=0
ORACLE_HOME=/ora01/app/oracle/product/11.2.0.4/db_1
ORACLE_HOME_OLD=
PLACEMENT=restricted
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_POOLS=ora.racon
SPFILE=+DATA1/racon/spfileracon.ora
START_DEPENDENCIES=hard(ora.DATA1.dg,ora.ARCH1.dg,ora.REDO1.dg,ora.REDO2.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns) pullup(ora.DATA1.dg,ora.ARCH1.dg,ora.REDO1.dg,ora.REDO2.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA1.dg,shutdown:ora.ARCH1.dg,shutdown:ora.REDO1.dg,shutdown:ora.REDO2.dg)
STOP_TIMEOUT=600
TYPE_VERSION=3.2
UPTIME_THRESHOLD=1h
USR_ORA_DB_NAME=racon
USR_ORA_DOMAIN=
USR_ORA_INST_NAME=
USR_ORA_INST_NAME@SERVERNAME(NodeA)=racon1
USR_ORA_INST_NAME@SERVERNAME(NodeB)=racon2
USR_ORA_OPEN_MODE=open
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.4.0

SYS@racon1 SQL> sho parameter listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
listener_networks                    string
local_listener                       string       (ADDRESS=(PROTOCOL=TCP)(HOST=
                                                 10.18.nn.nn)(PORT=1521))
remote_listener                      string      xxxx.com:1521


$ srvctl status database -d racon
Instance racon1 is running on node NodeA
Instance racon2 is running on node NodeB

-- Prior to converting an Oracle RAC database to an Oracle RAC One Node database, you must first ensure that the Oracle RAC database has only one instance.

$srvctl stop instance  -d racon -i  racon2

< NodeA : racon1 : oracle : /users/oracle >
$ srvctl stop instance  -d racon -i  racon2

< NodeA : racon1 : oracle : /users/oracle >
$ srvctl status database -d racon
Instance racon1 is running on node NodeA
Instance racon2 is not running on node NodeB


$ srvctl convert database -d racon  -c RACONENODE -i racon1 -w 30
PRCD-1214 : Administrator-managed RAC database racon has more than one instance

$ srvctl remove instance -d racon -i racon2
Remove instance from the database racon? (y/[n]) y


$ crsctl stat res ora.racon.db -p
NAME=ora.racon.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:r--,other::r--,group:dba:r-x,group:oinstall:r-x,user:grid:r-x
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLUSTER_DATABASE=true
DATABASE_TYPE=RAC
DB_UNIQUE_NAME=racon
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=database) PROPERTY(DB_UNIQUE_NAME= CONCAT(PARSE(%NAME%, ., 2), %USR_ORA_DOMAIN%, .)) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%) ELEMENT(DATABASE_TYPE= %DATABASE_TYPE%)
DEGREE=1
DESCRIPTION=Oracle Database resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
GEN_AUDIT_FILE_DEST=/ora47/adump/racon
GEN_START_OPTIONS=
GEN_START_OPTIONS@SERVERNAME(NodeA)=open
GEN_START_OPTIONS@SERVERNAME(NodeB)=open
GEN_USR_ORA_INST_NAME=
GEN_USR_ORA_INST_NAME@SERVERNAME(NodeA)=racon1


$ srvctl convert database -d racon  -c RACONENODE -i racon1 -w 30
PRCD-1242 : Unable to convert RAC database racon to RAC One Node database because the database had no service added

$ srvctl add  service -d racon -s racon -r racon1
PRCD-1210 : The service name racon cannot be same as the database default service name racon

$ srvctl add  service -d racon -s racon_svc -r racon1

$ srvctl config service -d racon -s racon_svc
Service name: racon_svc
Service is enabled
Server pool: racon_racon_svc
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: racon1
Available instances:

 < NodeA : racon1 : oracle : /users/oracle >
$ srvctl convert database -d racon  -c RACONENODE -i racon1 -w 30

$ srvctl config database -d racon
Database unique name: racon
Database name: racon
Oracle home: /ora01/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DATA1/racon/spfileracon.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: racon
Database instances:
Disk Groups: DATA1,ARCH1,REDO1,REDO2
Mount point paths:
Services: racon_svc
Type: RACOneNode
Online relocation timeout: 30
Instance name prefix: racon1
Candidate servers: NodeA
Database is administrator managed


$ srvctl relocate database -d racon -n NodeB
< NodeA : racon1 : oracle : /users/oracle >
$ srvctl status database -d racon
Instance racon1_2 is running on node NodeB
Online relocation: INACTIVE

$ srvctl relocate database -d racon -n NodeA
< NodeA : racon1 : oracle : /users/oracle >
$ srvctl status database -d racon
Instance racon1_1 is running on node NodeA
Online relocation: INACTIVE

So as you can see the conversion completed successfully and so is the relocation.. 
Hope that helps...

Friday, April 17, 2015

Upgrade 11203 DB to 11204 Using RMAN


We recently had an application requirement when the source 11203 database was needed by application team on different server with different name. Also the the database needs to upgraded from 11203 to 11204.

Following note describes the steps one needs to take this kind of exercise.


SOURCE DB 11.2.0.3 - NWK3
NEW DUPLICATE DB 11.2.0.4- NWD6A

##
## FIRST IDENTIFY THE DISKS AND DISK GROUPS NEEDED FOR THE NEW DATABASE. I USED THE EXISTING DATABASE CONFIGURATION AS TEMPLATE
##

ASMCMD [+] > lsdg NWC6A*

 NWC6ADATA1
 NWC6AARCH1
 NWC6AARCH3

ASMCMD [+] > lsdsk -G NWC6ADATA1
Path
/dev/asmd0008
/dev/asmd0009
/dev/asmd0010
/dev/asmd0011

ASMCMD [+] >  lsdsk -G NWC6AARCH1
Path
/dev/asma0011
/dev/asma0012

ASMCMD [+] >  lsdsk -G NWC6AARCH3
Path
/dev/asma0013
/dev/asma0014
/dev/asma0015
/dev/asma0016

#############
## NWD6ADATA1 - check available disks for creation of disk group.
#############

select path, group_number group_#, disk_number disk_#, mount_status, header_status, state, total_mb, free_mb
from v$asm_disk where path like '/dev/asmd%' and header_status='CANDIDATE' order by path ;

PATH
--------------------------------------------------------------------------------
   GROUP_#     DISK_# MOUNT_S HEADER_STATU STATE      TOTAL_MB    FREE_MB
---------- ---------- ------- ------------ -------- ---------- ----------
/dev/asmd0012
         0         28 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asmd0013
         0         29 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asmd0014
         0         30 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asmd0015
         0         31 CLOSED  CANDIDATE    NORMAL            0          0


#############
## NWD6AARCH3
#############
select path, group_number group_#, disk_number disk_#, mount_status, header_status, state, total_mb, free_mb
from v$asm_disk where path like '/dev/asma%' and header_status='CANDIDATE' order by path ;


PATH
--------------------------------------------------------------------------------
   GROUP_#     DISK_# MOUNT_S HEADER_STATU STATE      TOTAL_MB    FREE_MB
---------- ---------- ------- ------------ -------- ---------- ----------

/dev/asma1001
         0          0 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asma1002
         0          1 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asma1003
         0          2 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asma1004
         0          3 CLOSED  CANDIDATE    NORMAL            0          0


#############
## NWD6AARCH1
#############

/dev/asma2001
         0          4 CLOSED  CANDIDATE    NORMAL            0          0

/dev/asma2002
         0          5 CLOSED  CANDIDATE    NORMAL            0          0


##
## CREATE A NEW DATA DISK GROUP FOR NWD6A DATABASE
##

CREATE DISKGROUP NWD6ADATA1 EXTERNAL REDUNDANCY
DISK '/dev/asmd0012' NAME asmd0012, '/dev/asmd0013' NAME asmd0013, '/dev/asmd0014' NAME asmd0014,'/dev/asmd0015' NAME asmd0015;

Diskgroup created.
 

##
## CREATE A NEW ARCH1 DISK GROUP FOR NWD6A DATABASE
##

CREATE DISKGROUP NWD6AARCH1 EXTERNAL REDUNDANCY
DISK '/dev/asma2001' NAME asma2001, '/dev/asma2002' NAME asma2002;

Diskgroup created.

##
## CREATE A NEW ARCH3 DISK GROUP FOR NWD6A DATABASE
##
CREATE DISKGROUP NWD6AARCH3 EXTERNAL REDUNDANCY
DISK '/dev/asma1001' NAME asma1001, '/dev/asma1002' NAME asma1002, '/dev/asma1003' NAME asma1003, '/dev/asma1004' NAME asma1004;

Diskgroup created.


##
## NOW PREPARE THE PARAMETER FILE. EITHER ONE CAN PREPARE THE NORMAL FILE OR ## USE THE PARAMETER FILE FROM SOURCE DATABASE
## 

*.audit_file_dest='/ora47/adump/nwd6a'
*.compatible='11.2.0.3.0'
*.control_files='+NWD6ADATA1/control01.ctl','+NWD6ADATA1/control02.ctl','+NWD6ADATA1/control03.ctl'
*.db_block_size=8192
*.DB_CREATE_FILE_DEST=('+NWD6ADATA1')
*.db_domain=''
*.db_files=2000
*.db_name='nwd6a'
*.db_unique_name='nwd6a'
*.deferred_segment_creation=FALSE
*.diagnostic_dest='/ora01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=NWD6AXDB)'
*.log_archive_dest_1='LOCATION=+NWD6AARCH1'
*.open_cursors=300
*.os_authent_prefix=''
*.pga_aggregate_target=10670309376
*.processes=1500
*.remote_login_passwordfile='EXCLUSIVE'
*.sessions=1655
*.sga_max_size=3G
*.sga_target=3G
*.undo_tablespace='UNDOTBS1'



##
## Start the database in nomount mode..
##


< xxx : nwd6a : oracle : /users/oracle/hshah/GOLD_MAROON_FROM_TORQUISE_DONOT_DEL/nwk3_gold_cpy >
$ rman auxiliary /
connected to auxiliary database: NWD6A (not mounted)

run
{
allocate auxiliary channel c1  type disk;
allocate auxiliary channel c2  type disk;
allocate auxiliary channel c3  type disk;
allocate auxiliary channel c4  type disk;
allocate auxiliary channel c5  type disk;
DUPLICATE DATABASE TO 'nwd6a' pfile='/ora01/app/oracle/product/11.2.0.4/db_1/dbs/initnwd6a.ora' 
BACKUP LOCATION '/users/oracle/hshah/GOLD_MAROON_FROM_TORQUISE_DONOT_DEL/nwk3_gold_cpy' noredo noopen
LOGFILE
      GROUP 1 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 2 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 3 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 4 ('+nwd6adata1') SIZE  100M  REUSE;       
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
}

##
## ISSUES THAT CAME ACROSS...
##

1/ SOURCE DISK GROUP DID NOT EXIST - diskgroup "NWK3DATA1"
   ORA-15001: diskgroup "NWK3DATA1" does not exist or is not mounted
   ORA-15001: diskgroup "NWK3DATA1" does not exist or is not mounted

sql statement: CREATE CONTROLFILE REUSE SET DATABASE "NWD6A" RESETLOGS NOARCHIVELOG
  MAXLOGFILES     64
  MAXLOGMEMBERS      3
  MAXDATAFILES     1000
  MAXINSTANCES     1
  MAXLOGHISTORY      584
 LOGFILE
  GROUP   1 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
  GROUP   2 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
  GROUP   3 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
  GROUP   4 ( '+nwd6adata1' ) SIZE 100 M  REUSE
 DATAFILE
  '+NWD6ADATA1/nwd6a/datafile/system.286.877202211'
 CHARACTER SET WE8ISO8859P1

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 04/16/2015 19:45:21
RMAN-05501: aborting duplication of target database
RMAN-06136: ORACLE error from auxiliary database: ORA-01503: CREATE CONTROLFILE failed
ORA-00349: failure obtaining block size for '+nwk3data1'
ORA-15001: diskgroup "NWK3DATA1" does not exist or is not mounted
ORA-15001: diskgroup "NWK3DATA1" does not exist or is not mounted


CAUSE - This failure happend due log_file_name_convert and db_file_name_convert parameter. Hence those paraemeters were removed.

##
## AFTER MODIFYING THE PARAMETER FILE AND REMOVING CONVERT PARAMETER, THE DUPLICATION WORKED.
##
contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10041: Could not re-create polling channel context following failure.
RMAN-10024: error setting up for rpc polling
RMAN-10005: error opening cursor
RMAN-10002: ORACLE error: ORA-03114: not connected to ORACLE
RMAN-03002: failure of Duplicate Db command at 04/16/2015 21:19:47
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 29360134
Session ID: 3 Serial number: 5


##
## MADE CHANGES TO SCRIPT FOR NOREDO AND NOOPEN
## CHANGED THE LOCATION FOR THE ONLINE LOGFILE
##

$ rman auxiliary /
connected to auxiliary database: NWD6A (not mounted)

run
{
allocate auxiliary channel c1  type disk;
allocate auxiliary channel c2  type disk;
allocate auxiliary channel c3  type disk;
allocate auxiliary channel c4  type disk;
allocate auxiliary channel c5  type disk;
DUPLICATE DATABASE TO 'nwd6a' pfile='/ora01/app/oracle/product/11.2.0.4/db_1/dbs/initnwd6a.ora' 
BACKUP LOCATION '/users/oracle/hshah/GOLD_MAROON_FROM_TORQUISE_DONOT_DEL/nwk3_gold_cpy' noredo noopen
LOGFILE
      GROUP 1 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 2 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 3 ('+nwd6adata1') SIZE  100M  REUSE,
      GROUP 4 ('+nwd6adata1') SIZE  100M  REUSE;       
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
}



##
## AFTER RUNNING THE DUPLICATE WITH NOREDO AND NO OPEN THE DUPLICATE WENT
## FINE...
##


datafile 22 switched to datafile copy
input datafile copy RECID=21 STAMP=877251535 file name=+NWD6ADATA1/nwd6a/datafile/i_dmrep.278.877250233
Leaving database unopened, as requested
Finished Duplicate Db at 17-APR-15

2. Issue with UNDO TABLE SPACE

SQL> startup nomount;
ORACLE instance started.

Total System Global Area 2137886720 bytes
Fixed Size                  2248080 bytes
Variable Size            1442841200 bytes
Database Buffers          687865856 bytes
Redo Buffers                4931584 bytes
SQL> CREATE CONTROLFILE REUSE SET DATABASE "NWD6A" RESETLOGS NOARCHIVELOG
  2    MAXLOGFILES     64
  3    MAXLOGMEMBERS      3
  4    MAXDATAFILES     1000
  MAXINSTANCES     1
  5    6    MAXLOGHISTORY      584
  7   LOGFILE
  8    GROUP   1 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
  9    GROUP   2 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
 10    GROUP   3 ( '+nwd6adata1' ) SIZE 100 M  REUSE,
 11    GROUP   4 ( '+nwd6adata1' ) SIZE 100 M  REUSE
 12   DATAFILE
  '+NWD6ADATA1/nwd6a/datafile/system.286.877202211'
 13   14   CHARACTER SET WE8ISO8859P1
 15  ;


SQL> spool upgrade.log
SQL> alter database open resetlogs upgrade;
alter database open resetlogs upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-30046: Undo tablespace UNDO1 not found in control file.
Process ID: 32571418
Session ID: 298 Serial number: 1


##
## AFTER THE RESTORE AND RECOVERY DONE, STOP & START THE DUPLICATED DATABASE.
## CHECK FOR TEMP FILES AND LOG FILES.
##

SQL> select * from gv$tempfile;

   INST_ID      FILE# CREATION_CHANGE# CREATION_        TS#     RFILE# STATUS  ENABLED         BYTES     BLOCKS CREATE_BYTES BLOCK_SIZE NAME
---------- ---------- ---------------- --------- ---------- ---------- ------- ---------- ---------- ---------- ------------ ---------- 

         1          1       5.6483E+12 21-JAN-14          3          1 ONLINE  READ WRITE          0          0   9663676416       8192 +NWD6ADATA1
         1          2       5.6479E+12 10-MAY-13          3          2 ONLINE  READ WRITE          0          0   2147483648       8192 +NWD6ADATA1


##
## OPEN THE DB IN UPGRADE MODE.
##

SQL> alter database open resetlogs upgrade ;
Database altered.

SQL> select instance_name,status from v$instance;
INSTANCE_NAME    STATUS
---------------- ------------
nwd6a            OPEN MIGRATE


##
## RUN THE UPGRADE SCRIPT.
##

sql > @$ORACLE_HOME/rdbms/admin/catupgrd.sql


SQL> @$ORAC1120/rdbms/admin'UTLU112S.SQL;

SQL> @$ORA11202/RDBMS/ADMIN/CATUPPST.SQL
SQL> @$ORAC1120/rdbms/admin/utlrp.sql;

##
## AFTER UPGRADE OPEN THE DATABASE IN OPEN MODE
##

SQL> startup
ORACLE instance started.

Total System Global Area 3206836224 bytes
Fixed Size                  2250648 bytes
Variable Size            1174407272 bytes
Database Buffers         2013265920 bytes
Redo Buffers               16912384 bytes
Database mounted.
Database opened.
SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for IBM/AIX RISC System/6000: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production


##
## RECREATE SPFILE ON ASM
##

$ cat initnwd6a.ora
SPFILE='+nwd6adata1/nwd6a/parameterfile/spfile.257.877255249'


##
## ADD DATABASE TO OCR AS CRS RESOURCE
##

$ srvctl  add database -d nwd6a -o /ora01/app/oracle/product/11.2.0.4/db_1 -p '+nwd6adata1/nwd6a/parameterfile/spfile.257.877255249' -s OPEN -t NORMAL -y AUTOMATIC -a "nwd6adata1,nwd6aarch1,nwd6aarch3"


$ srvctl  start database -d nwd6a
< rri2rxodda201 : nwd6a : oracle : /ora01/app/oracle/product/11.2.0.4/db_1/dbs >
$ srvctl  status  database -d nwd6a
Database is running.

SQL> create or replace  directory EXPDP as '/ora49/dpdump/nwd6a' ;
Directory created.

Once you are done, your database is ready to roll...

Thursday, February 19, 2015

11g to 12c Upgrade/Migration Using Full Transportable TableSpace & Data Pump


With the Advent of 12c and Oracle trying to convincing the people to migrate to 12c, we decided to test one of our OEM repository DB running on 11.2.0.3 to 12c. The approach we choose is to use TTS + Data Pump feature to migrate the DB to newer version. 

When using conventional methods to move data, Oracle Data Pump uses either external tables or direct path unload to extract data. While the choice between these two access methods is based on the structure and types of the data being unloaded, both methods efficiently extract logical subsets of data from an Oracle database. 

In contrast, a transportable move of data and indexes involves the physical movement of one or more tablespace data files. The data segments inside the tablespace data files are not read individually. Instead, the export operation extracts the metadata that describes the objects containing storage within each data file, and each file is moved as a single entity
Moving large volumes of data using transportable tablespaces can be faster than conventional data movement because there is no need to interpret and extract individual rows of data or index entries. It is possible to move individual tables or partitions in a transportable manner, but the entire tablespace data file is moved in these cases as well. Understanding the difference between conventional and transportable data movement is helpful when considering the distinction between administrative and user tablespaces. 

For the purposes of a full transportable export, administrative tablespaces are the tablespaces provided by Oracle, such as SYSTEM, SYSAUX, TEMP, and UNDO. These tablespaces contain the procedures, packages, and seed data for the core Oracle database functionality and Oracle-provided database components such as Oracle Spatial, Oracle Text, OLAP, JAVAVM, and XML Database. 
In contrast, user tablespaces are those tablespaces defined by database users or applications. These may store user data, application data, and any other information defined by users of the database.

One consideration specific to full transportable export/import arises when there exists a database object (e.g., a partitioned table) that is stored across both user and administrative tablespaces. Storing an object in this way is generally not good practice, but it is possible. If there is an object with storage in both administrative and user tablespaces, then you can either redefine that object before transporting your data, or use conventional Data Pump export/import. The example later in this white paper shows how to detect this condition prior to starting a full transportable export. 

Following is the outline of the step which one can use. 

As a first step, one needs to identify the table-spaces needed for the application data and make sure that objects belonging to application schema are self contained. If they are not then one needs to first migrate them to single table space for easier management, however that's not mandatory. 
1. Identify the table spaces needed for applications.

SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces;

TABLESPACE_NAME              STATUS
------------------------------ ---------
MGMT_ECM_DEPOT_TS        ONLINE
MGMT_TABLESPACE             ONLINE
MGMT_AD4J_TS                  ONLINE


To determine whether our table spaces are self-contained, including verification that
referential integrity constraints will be valid after the transport, execute the following command on the source database.


SQL> EXECUTE DBMS_TTS.TRANSPORT_SET_CHECK('MGMT_ECM_DEPOT_TS,MGMT_TABLESPACE,MGMT_AD4J_TS', TRUE);
PL/SQL procedure successfully completed.


Note that you must include all user tablespaces in the database when performing this check for a full transportable export/import.
After invoking this PL/SQL procedure, you can see all violations by selecting from the
TRANSPORT_SET_VIOLATIONS view.
 

SQL> SELECT * FROM TRANSPORT_SET_VIOLATIONS;
no rows selected


If the set of tablespaces is self-contained, this view will be empty. If any violations are listed, then you must address these issues before proceeding with the full transportable operation.

2. Put TBS in read only mode..
SQL> alter tablespace MGMT_ECM_DEPOT_TS read only;
Tablespace altered.

SQL>  alter tablespace  MGMT_TABLESPACE read only;
Tablespace altered.

SQL>  alter tablespace MGMT_AD4J_TS read only;
Tablespace altered.
 


3. Export the metadata of the needed table spaces.

expdp parfile=expdp.par 
# vi expdp.par
full=Y
transportable=always
version=12
directory=DATA_PUMP_DIR
dumpfile=oemtestdb.dmp
metrics=y
exclude=statistics
#encryption_password=secret123word456
logfile=full_tts_export.log
EXCLUDE=TABLESPACE:"IN ( \'TABLESPACE:USERS\', \'TABLESPACE:UNDOTBS1\', \'TABLESPACE:TEMP\', \'TABLESPACE:EXAMPLE\') "


Final output -

Master table "SYS"."SYS_EXPORT_FULL_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYS.SYS_EXPORT_FULL_01 is:
  /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/log/gpdb.dmp
******************************************************************************
Datafiles required for transportable tablespace MGMT_AD4J_TS:
  +RAC_DATA/gpdb/datafile/mgmt_ad4j_ts.339.843266579
Datafiles required for transportable tablespace MGMT_ECM_DEPOT_TS:
  +RAC_DATA/gpdb/datafile/mgmt_ecm_depot_ts.341.843266557
Datafiles required for transportable tablespace MGMT_TABLESPACE:
  +RAC_DATA/gpdb/datafile/mgmt_tablespace.340.843266559
Job "SYS"."SYS_EXPORT_FULL_01" successfully completed at 10:03:27

4. Now if you are using ASM to store the files, then connect to ASM instance using asmcmd and copy the files on target server as follows

ASMCMD [+rac_data/gpdb/datafile] > ls -ltr
WARNING:option 'r' is deprecated for 'ls'
please use 'reverse'

Type      Redund  Striped  Time             Sys  Name
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    EXAMPLE.1618.843199351
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    MGMT_ECM_DEPOT_TS.341.843266557
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    MGMT_TABLESPACE.340.843266559
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    SYSAUX.500.843199155
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    UNDOTBS1.338.843265261
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    UNDOTBS1.499.843199157
DATAFILE  UNPROT  COARSE   MAR 27 04:00:00  Y    USERS.1617.843199353
DATAFILE  UNPROT  COARSE   MAR 27 09:00:00  Y    MGMT_AD4J_TS.339.843266579
DATAFILE  UNPROT  COARSE   MAR 27 10:00:00  Y    SYSTEM.1624.843199155

ASMCMD [+rac_data/gpdb/datafile] > cp mgmt_ad4j_ts.339.843266579 /u02/tmp
copying +rac_data/gpdb/datafile/mgmt_ad4j_ts.339.843266579 -> /u02/tmp/mgmt_ad4j_ts.339.843266579

ASMCMD [+rac_data/gpdb/datafile] > cp mgmt_ecm_depot_ts.341.843266557 /u02/tmp
copying +rac_data/gpdb/datafile/mgmt_ecm_depot_ts.341.843266557 -> /u02/tmp/mgmt_ecm_depot_ts.341.843266557

ASMCMD [+rac_data/gpdb/datafile] > cp mgmt_tablespace.340.843266559 /u02/tmp
copying +rac_data/gpdb/datafile/mgmt_tablespace.340.843266559 -> /u02/tmp/mgmt_tablespace.340.843266559


COPY THIS DATAFILES ON TARGET SERVER AND CHANGE PERMISSIONS ACCORDINGLY ( FOR THIS CASE I HAVE CONFIGURED THE ACFS MOUNT ACROSS THE CLUSTER SO I DONT REALLY NEED TO COPY THEM TO TARGET HOST, BUT ONLY TO TARGET LOCATION)

5. copy the dump file to DATA_PUMP_DIR location on target server..
DATA_PUMP_DIR - /u01/app/oracle/admin/gpdbupg/dpdump/                                                                                             
                                                                                                         
NOW ON 12C DATABASE IMPORT THE DUMP

impdp full=Y directory=DATA_PUMP_DIR dumpfile=gpdb.dmp logfile=impdpgpdbupg.log  transport_datafiles='/u02/app/product/12.1.0/datafiles/gpdbupg/gpdbupg/mgmt_ecm_depot_ts.341.843266557','/u02/app/product/12.1.0/datafiles/gpdbupg/gpdbupg/mgmt_ad4j_ts.339.843266579','/u02/app/product/12.1.0/datafiles/gpdbupg/gpdbupg/mgmt_tablespace.340.843266559'

OR 
# impdp parfile=impdp.par

# vi impdp.par
FULL=Y
DIRECTORY=DATA_PUMP_DIR
#ENCRYPTION_PASSWORD=<enc_passwd>
DUMPFILE=oemtestdb.dmp
LOGFILE=src112fullimp.log
#############################
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_ECM_DEPOT_TS.DBF'
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_TABLESPACE.DBF'
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_AD4J_TS.DBF'
#############################

You just finished your 11g to 12c Migration successfully!! All you need to do now is to open the allow the users to connect to this DB and start rolling!! 

There is an another approach to deploy full transportable export/import which employs the network mode of Data Pump, to eliminate the need for a dumpfile. In this case, we will assume that the tablespace data files are in a location, such as a Storage Area Network (SAN) device, which is accessible to both the source and destination systems. This enables you to migrate from a non-CDB into a PDB with one Data Pump command: 

1. Create a CDB on the destination system, including a PDB into which you will import the source database. 
2. Create a database link in the destination PDB, pointing to the source database. 
3. Set user tablespaces in the source database to READ ONLY. Copy the data files for tablespaces to target CDB/PDB. 
4. In the Oracle Database 12c environment, import directly from the source database into the destination PDB using full transportable export/import in network mode:

SQL> alter session set container=upgdb;
Session altered.

SQL> sho user
USER is "SYS"

SQL> CREATE DATABASE LINK src112 CONNECT TO system IDENTIFIED BY oracle USING 'TEST';

Database link created.

SQL> select * from sys.link_test@src112;

         N
----------
        10


# Create tablespaces on target database for application data.
SQL> show con_name

CON_NAME
------------------------------
UPGDB


#impdp sys/oracle@upgdb  parfile=upg.par


#vi upg.par
NETWORK_LINK=src112
VERSION=12
FULL=Y
TRANSPORTABLE=ALWAYS
EXCLUDE=STATISTICS
#ENCRYPTION_PASSWORD=<enc_passwd>
METRICS=Y
LOGFILE=tgtpdb_dir:src112fullimp.log
#############################
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_ECM_DEPOT_TS.DBF'
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_TABLESPACE.DBF'
TRANSPORT_DATAFILES='/u02/app/product/12.1.0/datafiles/upgdb/MGMT_AD4J_TS.DBF'
#############################


Whether you use conventional dump files or network mode, full transportable export/import is a convenient way to upgrade a database to a new version, migrate to a different operating system or hardware platform, migrate into a PDB – or even to perform all three of these upgrades and migrations in a single operation!