Error ORA-16724 Raised While Oracle Standby Database Tried to Catch up Redo Log Received from Primary Database

For what ever reason, if error ORA-16724 raised while you try to have oracle standby database to catch up the redo log received from primary database, you might need to use incremental backup to achieve your goal. It’s for sure you could simply drop standby database or and create a new one. But, that’s not always a feasible way.

Over the past weekend, I experienced such error while I was working on moving database to different Unix system. It’s very good to me to come across the good posting regarding to this topic. That’s “Resolving Gaps in Data Guard Apply Using Incremental RMAN Backup” by author Arup Nanda.

One thing I realized  is that it’s better to pick earlier SCN number from which to create incremental backup. For example, if current SCN on standby is 1301571, it’s no harm to do incremental backup starting from SCN 1300000. The reason I’m doing this is because the log file for SCN 1301571 on standby site is probably corrupted and starting from earlier SCN could re-generate redo log file for SCN 1301571.

This posting provided step-by-step approach to solve this issue. The only step I added is

SQL Stby> alter database recover managed standby database cancel;

The reason I’m doing for is because I experienced error message below when I tried to recover database via RMAN by using incremental backup files generated from primary database.

RMAN> recover database;

Starting recover at 26-JAN-15
using channel ORA_DISK_1
using channel ORA_DISK_2
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00003: /dbfiles/STBY/undotbs.dbf
destination for restore of datafile 00008: /dbfiles/STBY/development1.dbf
destination for restore of datafile 00009: /dbfiles/STBY/indexes.dbf
channel ORA_DISK_1: reading from backup piece /dbbackup/incre_back/5tptmjaj_1_1.rmb
channel ORA_DISK_2: starting incremental datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /dbfiles/STBY/system.dbf
channel ORA_DISK_2: reading from backup piece /dbbackup/incre_back/5uptmji8_1_1.rmb
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 01/26/2015 13:43:05
ORA-19870: error while restoring backup piece /dbbackup/incre_back/5tptmjaj_1_1.rmb
ORA-19573: cannot obtain exclusive enqueue for datafile 8

As standby database is going to apply redo log files received from primary database once it’s mounted, the datafiles are consequently locked. Issuing command above will stop the redo log apply and release lock on the the datafile.

Once I followed the steps in Arup’s posting, the ORA-16724 was solved and the standby database could be able to apply the redo log.

Furthermore, I got one warning message from data guard manager, saying “ORA-16826: apply service state is inconsistent with the DelayMins property”. To solve this, I just simply follow the posting authored by Navneet Upneja.  The workaround is quite straightforward.

SQL Stby> alter database recover managed standby database cancel;

Database altered.

SQL Stby> alter database recover managed standby database using current logfile disconnect;

Database altered.

The logic is to stop redo log apply first and then start redo log apply in real time mode.

Solution_for_Ora_16724_Standby_database (author: Arup Nanda)

Solution_for_Ora-16826_Standby_Database (author: Navneet Upneja)

Posted in My Reference | Tagged , , , | Leave a comment

File Removal Fails to Free Up Disk Space

Working on Unix/Linux, almost everyone experienced the disk size confusion while issuing df and du command against same directory. Sometime, although the file was technically removed from system level, the disk space is not released if other processes have the the same file open. The disk space will be freed while no process have this file open.

If you encountered above circumstance, one of effective approaches is to use /proc filesystem, which give access to all open files of all processes.

$  find /proc/*/fd -type f -links 0 \! -size 0 -ls
142971    3 --w-------   0 oracle   dba          2183 Dec  1 12:22 /proc/2332/fd/18
142972    1 --w-------   0 oracle   dba            94 Dec  1 12:22 /proc/2332/fd/19
 2759 1033346 -rw-r-----   0 oracle   dba      1048584192 Jan  5 11:51 /proc/6872/fd/259
  281 19168181 -rw-r-----   0 oracle   dba      19398664192 Jan  5 11:42 /proc/6905/fd/257

If you have the above output, just simply issue following command to truncate file, which will release the disk space.

$ cp /dev/null /proc/6905/fd/257
$ cp /dev/null /proc/6872/fd/259
Posted in My Reference, Oracle Point | Tagged | Leave a comment

Adding XML-DB feature to existing oracle database

Using DBCA to add new database feature to existing oracle database is common way. To do that, just simply launch DBCA graphical interface, choose “Configure Database Options”, and then follow the screen to finish it.

However, sometimes you might find that the radio box “Configure Database Options” is grayed out, as shown below.

dbca-adding-new-feature-grayedout

To enable “Configure Database Options”, one line need to be added to file oratab, which usually sit under /var/opt/oracle.

dbname:/oracle/product/database/11.2.0.4/dbhome_1:Y

Once above line is added, you will see “Configure Database Options” is enabled after you launch DBCA.

dbca-adding-new-feature

But, that’s not all. Because, you might experience further error such as below.

mismatching-casesensitive-dbname

Actually, this error is absolutely misleading one. You probably go for research and work on system parameter. But, it’s very likely that the dbname in file oratab doesn’t match the tnsname because it’s case sensitive. To avoid the happening of this error, simply make dbname in file oratab and tnsname identical without case sensitivity issue.

Posted in Oracle Case Study, Oracle Point | Tagged , | Leave a comment