Storage Migration on OCR and Voting Disk

Dear Viewer,

Recently I have done with renaming of OCR and Voting Disks during the Storage Migration.

Here are the high level action plan which I performed in our Non-Prod RAC instances as root user,

OCR :
===

ocrcheck
ocrconfig -export /oracle/export/CRS/OCR_Backup -s online <-- Taking OCR Backup -- Monitor the recent log file under $CRS_Home/log//client
ocrconfig -replace ocr /dev/mapper/tst-db-01-02_cluster_ocr_001_1000M
-- Monitor the recent log file under $CRS_Home/log//client
ocrconfig -replace ocrmirror /dev/mapper/tst-db-01-02_cluster_ocrmirr_002_1000M
-- Monitor the recent log file under $CRS_Home/log//client
ocrcheck

VOTING DISK :
============

crsctl stop crs
crsctl query css votedisk
crsctl add css votedisk /dev/mapper/tst-db-01-02_cluster_vote_003_1000M -force
crsctl add css votedisk /dev/mapper/tst-db-01-02_cluster_votemirr_004_1000M -force
crsctl delete css votedisk /dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000 -force
crsctl delete css votedisk /dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000 -force
crsctl start crs

-- Monitor /var/log/message file for any errors.

Session Log :
===========

Getting Current Status Information :
==================================

[root@tst-db-01 crs_home]# export PATH=$PATH:/oracle/product/crs_home/bin
[root@tst-db-01 crs_home]# clear
[root@tst-db-01 crs_home]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 1048040
Used space (kbytes) : 3636
Available space (kbytes) : 1044404
ID : 1397237804
Device/File Name : /dev/mapper/MFGSTGclus02_1708_42B6_ocr_1000
Device/File integrity check succeeded
Device/File Name : /dev/mapper/MFGSTGclus02_1708_42B0_ocr_mirror1_1000
Device/File integrity check succeeded

Cluster registry integrity check succeeded

[root@tst-db-01 crs_home]# crsctl query css votedisk
0. 0 /dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000
1. 0 /dev/mapper/oraMFGSTGclus02_1708_42B3_voting_mirror1_1000

located 2 votedisk(s).
[root@tst-db-01 crs_home]# ps -eaf|grep pmon
root 943 31154 0 07:43 pts/0 00:00:00 grep pmon
[root@tst-db-01 crs_home]#

Voting Disk :
===========

1. Bring down the CRS

[root@tst-db-01 /]# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.

2. Take the backup

[root@tst-db-01 /]# dd if=/dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000 of=/oracle/export/CRS/voting.dmp
2096640+0 records in
2096640+0 records out

[root@tst-db-01 /]# ls -l /oracle/export/CRS/voting.dmp
-rw-r--r-- 1 root root 1073479680 Jun 22 08:51 /oracle/export/CRS/voting.dmp

3. Add Voting Disk

[root@tst-db-01 /]# crsctl add css votedisk /dev/mapper/tst-db-01-02_cluster_vote_003_1000M -force
Now formatting voting disk: /dev/mapper/tst-db-01-02_cluster_vote_003_1000M
successful addition of votedisk /dev/mapper/tst-db-01-02_cluster_vote_003_1000M.

4. Add addition Voting Disk

[root@tst-db-01 /]# crsctl add css votedisk /dev/mapper/tst-db-01-02_cluster_votemirr_004_1000M -force
Now formatting voting disk: /dev/mapper/tst-db-01-02_cluster_votemirr_004_1000M
successful addition of votedisk /dev/mapper/tst-db-01-02_cluster_votemirr_004_1000M.

5. Deleting old Voting Disk

[root@tst-db-01 /]# crsctl delete css votedisk /dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000 -force
successful deletion of votedisk /dev/mapper/oraMFGSTGclus02_1708_42B2_voting_1000.

[root@tst-db-01 /]# crsctl delete css votedisk /dev/mapper/oraMFGSTGclus02_1708_42B3_voting_mirror1_1000 -force
successful deletion of votedisk /dev/mapper/oraMFGSTGclus02_1708_42B3_voting_mirror1_1000.

6. Bring up the CRS

[root@tst-db-01 /]# /etc/init.d/init.crs start
Startup will be queued to init within 30 seconds.
[root@tst-db-01 /]# tail -f /var/log/messages
Jun 22 08:58:47 tst-db-01 prashasi: Oracle Cluster Ready Services starting by user request.
Jun 22 08:59:00 tst-db-01 logger: Cluster Ready Services completed waiting on dependencies.
Jun 22 08:59:03 tst-db-01 last message repeated 2 times
Jun 22 08:59:03 tst-db-01 logger: Running CRSD with TZ =

7. Verify the CRS Services

[root@tst-db-01 /]# crsctl check crs

CSS appears healthy
Cannot communicate with CRS
EVM appears healthy

8. Verify the Voting Disk Changes

[root@tst-db-01 /]# crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@tst-db-01 /]# crsctl query css votedisk
0. 0 /dev/mapper/tst-db-01-02_cluster_vote_003_1000M
1. 0 /dev/mapper/tst-db-01-02_cluster_votemirr_004_1000M

located 2 votedisk(s).


OCR Disk :
========

1. Take the backup

[root@tst-db-01 export]# cd /oracle/export
[root@tst-db-01 export]# cd CRS
[root@tst-db-01 CRS]# ocrconfig -export /oracle/export/CRS/OCR_Backup -s online

2. Add the OCR and OCR Mirror Disk

[root@tst-db-01 CRS]# ls -l /dev/mapper/tst-db-01-02_cluster_ocr_001_1000M
brw-rw---- 1 root disk 253, 385 Jun 21 12:18 /dev/mapper/tst-db-01-02_cluster_ocr_001_1000M
[root@tst-db-01 CRS]# ocrconfig -replace ocr /dev/mapper/tst-db-01-02_cluster_ocr_001_1000M
[root@tst-db-01 CRS]# ocrconfig -replace ocrmirror /dev/mapper/tst-db-01-02_cluster_ocrmirr_002_1000M

3. Verify the OCR Changes

[root@tst-db-01 CRS]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 1048040
Used space (kbytes) : 3636
Available space (kbytes) : 1044404
ID : 1397237804
Device/File Name : /dev/mapper/tst-db-01-02_cluster_ocr_001_1000M
Device/File integrity check succeeded
Device/File Name : /dev/mapper/tst-db-01-02_cluster_ocrmirr_002_1000M
Device/File integrity check succeeded

Cluster registry integrity check succeeded

[root@tst-db-01 CRS]#

Regards,
Sasikumar.M

When Opening Database - 'ORA-01113: file 1 needs media recovery '

Hello All,

After restore and recovery is complete ,trying to open database with resetlogs option showed the following error message

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '/u02/oracle/UPG/apps/apps_st/db/system01.dbf' was restored from an oldbackup

I have verified the time and the scn number until which I ran to restore, everything was correct but still unable to open the database.I tried applying archive logs and then to open failed again.

Research

I have issued the following command to recover 'recover database using backup controlfile' .

After some research and googling I issued the following command to open database

sqlplus > recover database using backup controlfile until cancel;

When prompted issued CANCEL.Now tried to open the database with resetlogs and database opened without any issues.

Session Details

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 370577110 generated at 06/12/2010 16:19:22 needed for thread
1
ORA-00289: suggestion :
/u01/oracle/UPG/db/apps_st/arch/UPG_1_2671_699752349.arc
ORA-00280: change 370577110 for thread 1 is in sequence #2671


Specify log: {=suggested | filename | AUTO | CANCEL}
/u01/oracle/UPG/db/apps_st/arch/UPG_1_2671_699752349.arc
ORA-00279: change 370578253 generated at 06/12/2010 16:27:55 needed for thread
1
ORA-00289: suggestion :
/u01/oracle/UPG/db/apps_st/arch/UPG_1_2672_699752349.arc
ORA-00280: change 370578253 for thread 1 is in sequence #2672
ORA-00278: log file
'/u01/oracle/UPG/db/apps_st/arch/UPG_1_2671_699752349.arc' no longer
needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}
/u01/oracle/UPG/db/apps_st/arch/UPG_1_2672_699752349.arc
ORA-00279: change 370610644 generated at 06/12/2010 18:27:47 needed for thread
1
ORA-00289: suggestion :
/u01/oracle/UPG/db/apps_st/arch/UPG_1_2673_699752349.arc
ORA-00280: change 370610644 for thread 1 is in sequence #2673
ORA-00278: log file
'/u01/oracle/UPG/db/apps_st/arch/UPG_1_2672_699752349.arc' no longer
needed for this recovery


Specify log: {=suggested | filename | AUTO | CANCEL}
CANCEL
Media recovery cancelled.
SQL>
SQL> alter database open resetlogs;

Database altered.

SQL> select * from v$recover_file;

no rows selected

SQL> select distinct status from v$datafile;

STATUS
-------
ONLINE
SYSTEM


- Hope this helps when performing simple recovery to a different server.

RMAN Duplication - UNTIL scn (.......) is ahead of last scn in archived logs (........)

Hello All,


When performing duplication of database with rman connected to the Source received the following error

upg db009 oracle102 > rman target sys/sys@upg auxiliary /

Recovery Manager: Release 10.2.0.4.0 - Production on Tue Jun 29 17:26:47 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: UPG (DBID=1899709187)
connected to auxiliary database: STAGE (not mounted)

RMAN>

RMAN> run {
SET UNTIL SCN 379298362;
allocate auxiliary channel t1 type disk;
allocate auxiliary channel t2 type disk;
allocate auxiliary channel t3 type disk;
switch datafile all;
duplicate target database to stage;
}2> 3> 4> 5> 6> 7> 8>

executing command: SET until clause

allocated channel: t1
channel t1: sid=1587 devtype=DISK

allocated channel: t2
channel t2: sid=1586 devtype=DISK

allocated channel: t3
channel t3: sid=1585 devtype=DISK


Starting Duplicate Db at 28-JUN-10
released channel: t1
released channel: t2
released channel: t3
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 06/28/2010 14:58:18
RMAN-06457: UNTIL scn (379298362) is ahead of last scn in archived logs (379296482)

RMAN> exit


Issue and Verification

On Checking the Source DB,found that the last scn was older than what I have specified in run block of RMAN.Hence issued checkpoint connected to the source DB.

SQL> select thread#, status, enabled, checkpoint_time, checkpoint_change# from v$thread;

THREAD# STATUS ENABLED CHECKPOINT_TIME CHECKPOINT_CHANGE#
---------- ------ -------- ------------------ ------------------
1 OPEN PUBLIC 28-JUN-10 379296482



SQL> alter system checkpoint;

System altered.

SQL> select thread#, status, enabled, checkpoint_time, checkpoint_change# from v$thread;

THREAD# STATUS ENABLED CHECKPOINT_TIME CHECKPOINT_CHANGE#
---------- ------ -------- ------------------ ------------------
1 OPEN PUBLIC 28-JUN-10 379299681

SQL>

Now re-started the duplication and it completed successfully.

AD Administration Warning

Hello Again,


When applying any patch or starting adadmin in R12,I happened to notice the following message on the screen ,but adpatch or adadmin continued without errors or failures.

AD Administration warning:
Product Data File ' /u02/oracle/UPG/apps/apps_st/appl/admin/zfaprod.txt ' does not exist for product "zfa".This product is registered in the database but the above file does not exist in APPL_TOP. The product will be ignored without error.

Message also appeared for the following products "zsa" and "jts".


This message was displayed on all the R12 environments we had.Did a bit research and found a note :- 421136.1.

The note states that 'These products were registered in the 11i database, and hence are producing these warning messages. Oracle has this warning for all the products that are licensed in the DB, but are obsolete in R12. Customers will have this warning only for those products which they have licensed in 11i but obsolete in R12.'

Also the note states 'Ignore the warnings as they do not affect the functionality.'
If the patch fails, skip the workers of these products using adctrl and continue on with the patch.

Users Receive Blank Screen when logging into EBS in a SSO Integrated System

Hello All,

Last week was such a hectic one with one kind of a peculiar problem not seen before.
Users created after a particular date,logging into Oracle Apps via third party authentication,integrated with OID/SSO were getting blank pages.

On analysis we found that the user_guid column with the 'FND_USER' table was null for these users.
Verified OID the attribute 'orclguid' was there ,but it was not synching with FND_USER table,inspite the profile option 'Applications SSO Auto Link User" (APPS_SSO_AUTO_LINK_USER)' was set to 'ENABLED'
at site level.

I had a very hard feeling that the linking between EBS and OID was broken.To my surprise that was the case .
I dont know how many of you are aware of an user named 'AppsDN'.This "AppsDN" user is a special administration account which eBiz uses to connect
to OID for management and administrative tasks when integrating with OID/SSO.

- Determine the 'AppsDN' username/password connected to EBS as apps user

SQL> select fnd_preference.get('#INTERNAL', 'LDAP_SYNCH','USERNAME') Apps_Instance_OID_Account from dual;

APPS_INSTANCE_OID_ACCOUNT
--------------------------------------------------------------------------------
orclapplicationcommonname=prod,cn=ebusiness,cn=products,cn=oraclecontext,dc=thirdparty,dc=users

SQL> select fnd_preference.eget('#INTERNAL', 'LDAP_SYNCH','EPWD','LDAP_PWD') Apps_Password from dual;

APPS_PASSWORD
--------------------------------------------------------------------------------
1applin


- Test the username/password.Use the following command on your node where OID is running to confirm if the password works or not

'ldapbind -D 'Apps Instance OID Account' -w 'Apps Password' -h 'OIDserver' -p 'OIDport' '

Eg:-
===
ldapbind -D orclapplicationcommonname=prod,cn=ebusiness,cn=products,cn=oraclecontext,dc=thirdparty,dc=users -w 1applin -h server11 -p 389


- On executing the above ldapbind command you should recieve 'Bind Successful'.In my case it was returning 'Password Policy Error :9000: GSL_PWDEXPIRED_EXCP :Your Password has expired'

- Navigate through the "Entry Management" tree to the appropriate context (the reversed path of the orclApplcationCommonName)
Follow the below steps to set the password in OID

1) Launch oidadmin console
2) Login as user ‘orcladmin’
3) Select and expand ‘Entry Management’
4) Select and expand ‘dc=users’
5) Select and expand ‘dc=thirdparty’ under ‘dc=users’
6) Select and expand ‘cn=OracleContext’ under ‘dc=thirdparty’
7) Select and expand ‘cn=Products’
8) Select and expand ‘cn=EBusiness’
9) Select orclApplicationCommonName=PROD
10) Under the Properties tab update the 'userpassword' attribute with the password from SQL Query in my case '1applin'
11) Click on ‘Apply’

- Then bounce the OID components.

- Now we requested the users to login into Oracle Apps and they no longer get the blank screen and OID was synching up the guid to FND_USER table.

10gR2 : OPatch failed with Unable to create patchObject

Hi Viewer,

After long time, I am posting an interesting issue which I faced when applying the CPU patch with OPATCH utility.

When I tried to run opatch lsinventory, I got the following error message

[oracle@proddb ContentsXML]$ opatch lsinventory
Invoking OPatch 10.2.0.4.2

Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.


Oracle Home : /oracle/product/10.2.0.4/PROD
Central Inventory : /oracle/product/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /oracle/product/10.2.0.4/PROD/oui
Log file location : /oracle/product/10.2.0.4/PROD/cfgtoollogs/opatch/opatch2010-06-09_02-10-45AM.log



Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject

OPatch failed with error code 73
[oracle@proddb ContentsXML]$

So I set the OPATCH_DEBUG=TRUE environment variable for debugging more on this issue, I found the following extra error information,

OracleHomeInventory::load() begins to process raw data from OUI to build up OPatch primitive classes
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8576156/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8576156/etc/config/inventory.xml
parserXMLFile : start
parserXMLFile : start
.
.
JarAction::getCompleteFileListPath() returns :ORACLE_HOME/rdbms/jlib/qsma.jar
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8436582/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8436582/etc/config/inventory.xml
parserXMLFile : start
parserXMLFile : start
.
.
PatchObject(oracleHomePath,patchID)
PatchObject::PatchObject looks for the actions file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8788977/etc/config/actions.xml
PatchObject::PatchObject looks for the inventory file /oracle/product/10.2.0.4/PROD/inventory/oneoffs/8788977/etc/config/inventory.xml
parserXMLFile : start


Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

Locker::release()
OUISessionManager::unRegister()
Un-Registering the caller : OPatch
LsInventory::getInstance() returns
LsInventorySession failed: Unable to create patchObject
Cleaning up the directory : "/oracle/product/10.2.0.4/PROD/.patch_storage/patch_unzip"...

OPatch failed with error code 73
[oracle@proddb ContentsXML]$

Referred the following support notes for further troubleshooting, but it didn't fix the issue.

SOLARIS: "opatch lsinventory" is failing with error code 73 on Remote Node in 10.2.0.4 RAC environment [ID 793600.1]
Opatch Lsinventory Fails With "Unable to create patchObject" OPatch failed with error code 73 [ID 751152.1]
Opatch error: "Apply Session failed: ApplySession failed to prepare the system. No detail available." [ID 864803.1]

Tried recreating OraInventory using runInstaller utility. But that option also did not help.

After spending a lot of time, I upgraded the OPATCH utility from 10.2.0.4.2 to 10.2.0.4.8. Finally it fixed the issue.

Comments are Welcome!!!