Use ufsrestore to restore multi-tape ndmp backup

This applies to an environment that includes Netbackup doing ndmp backups of Netapp filers.

Summary:

Under certain circumstances you may have trouble doing ufsrestore of a Netbackup NDMP backup that spans multiple tapes. It may appear you have a corrupt tape. Don’t give up hope. You can use the following commands to get to a solution: dd, cat and ufsrestore.

Background:

Normally you don’t need to do a ufsrestore of a Netbackup NDMP backup and even if you did, chances are you may only have a single tape to restore from. However, in this particular situation we were not able to do a Netbackup restore and the backup we needed to restore from was spread over 2 LTO3 tapes.

The reason we could not do a Netbackup restore of a Netbackup backup was because the backup was from another master server and we could not successfully complete a phase 2 import of the backup image. Therefore since we did not have access to the other master server, we were forced to attempt a ufsrestore or Netapp restore.

When performing the ufsrestore of the backup which spans two tapes, we first noticed that ufsrestore did not recognize the 2nd fragment of the backup as a dump file. Trying to open the file using the ufsrestore command simply let to us getting the error message “Not in dump format”. Similarly when the ufsrestore would complete restoring from the 1st tape and ask for the next volume it would never be able to recognize the volume even though the tape was properly mounted to the exact location. (i.e. the tape was mounted to file 1, record 2 on the 2nd tape. Therefore our results were always the same–we were getting a partial restore of data that included the full directory structure but was limited to only those files that were contained in the 1st fragment of the backup on the 1st tape.

Issue:

When Netbackup is performing an NDMP backup to tape and reaches the end of the first tape, it causes the Netapp dump to pause until it has had time to load a new tape. Once the new tape is loaded, Netbackup tells the Netapp to resume the dump to tape. However, during this process Netbackup writes a special proprietary header record to the continued dump file on the second tape that causes the dump file to no longer look like a dump file to any applications including ufsrestore and the Netapp restore command. The only application that will see the dump file as a dump file is the Netbackup bprestore program. It will be able to process the complete dump file and complete the full restore. Therefore, you cannot perform a ufsrestore or a Netapp restore of such a backup unless you follow the process outlined here.

Solution:

1. Load the first tape of the multi-tape NDMP backup in the tape library.
2. Use the mt -f command to position the tape to the correct file. Make sure to position to the 2nd record of the file skipping over the Netbackup header record and positioning to the dump file header record.
3. dd the tape file to disk as . In our situation I used the command syntax “if=/dev/rmt/0bn of=//  bs=32768k” If you don’t insure a large enough block size the command will not run.
4. Load the 2nd tape of the multi-tape NDMP backup in the tape library.
5. The continuation of the dump file starts with the 2nd record of the 1st file on the 2nd tape. :-)  Therefore use the mt -f command to position to this location on the tape skipping over the special proprietary Netbackup header and positioning to the continuation of data records contained in the dump file.
6. dd the tape file to disk as
7. Once both files are on disk, use the cat command to concatenate the two files into one single dump file.
8. cat  
9. At this point you will be able to ufsrestore the without receiving any errors.

One other issue I observed when doing the ufsrestore after the file concatenation was this. Whenever the ufsrestore tried to restore a core dump file (core.) it ended the ufsrestore with the error message: “Write error extracting inode , name /.” Unfortunately I had to work around this issue by excluding any core dump files from my file selections during the interactive ufsrestore. I actually added everything in the root directory “add *” and then navigated within to the appropriate locations and “deleted ” from my selections. If anyone has a better way to work around this issue please let me know.

Theory:

I have a theory regarding how this situation could occur in a Netbackup environment and how this could be a bug in the Netbackup software. If you are interested let me know and I’ll add that theory.