Oracle Veeam Backup | VSS_FAILED_AT_FREEZE | ORA-00257: archiver error from remote SQLPlus | ORA-16038 ORA-19809 ORA-00312 found in Eventvwr | Fast Recovery Area is Full

Today, I was facing an issue where my Veeam Backup job failed on one Oracle Server hosting many Instances.

Only one of them was failing and Veeam reported the following error :

Processing SERVER_NAME Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Oracle VSS Writer - INSTANCE_NAME]. Class ID: [{26d02976-b909-43ad-af7e-62a4f625e372}]. Instance ID: [{bf669252-5552-433d-a0b5-cab28e14a19b}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f4].]

 

Checking the Server Eventlogs reported Event 46, Oracle.VSSWriter.INSTANCE_NAME

General Tab :

VSS-00046: failure to switch the current database redo logs

Cause : OCI call failed.

Action : Check the accompanying error message.

Additional info :
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
Oracle VSS writer version 11.2.0.2.0 Production
Error at line : 4856
Failure on Freeze event

Details Tab reported (example copied from Internet) :

ORA-16038: log 1 sequence# 49 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1:
'/opt/oracle/db/home/data/imapdb/redo01.log'

 

Also, trying to connect remotely using SQLPlus trowed the following :

ERROR:
ORA-00257: archiver error. Connect internal only, until freed.

 

With all these information, I suspected the Fast_Recovery_Area to be full and it has been confirmed by running the following SQL statement (locally to avoid SQLPlus connection issue explained above) :

SQL> SELECT * FROM V$RECOVERY_FILE_DEST;

NAME
--------------------------------------------------------
SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES
----------- ---------- ----------------- ---------------
D:\oracle\fast_recovery_area
 4322230272 3,4490E+10 759962624 809

Indeed, it is more than full.

 

Finally, I tried to delete expired Archivelogs using RMAN but it did not find any files to delete. It’s why I suggest to 1st run a Crosscheck and then delete expired items :

  • Start RMAN in a Command Prompt
  • Connect to your Instance
rman > connect target SYS/oracle@trgt
  • Run a Crosscheck
rman > crosscheck archivelog all;
  • Delete expired Archivelogs
rman > delete expired archivelog all;

 

In my case,  after deleting expired Archivelogs, the FRA was still Full so I decided to simply delete all of them and just keep the last 10 days by running :

rman>delete archivelog until time ‘SYSDATE-10’;

 

Veeam Backup & Oracle VSS is now working, Remote SQLPlus connection too. Solved. Hope this helps.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s