Troubleshooting TSM Scheduled Backup Failure Reports

IT Services

Troubleshooting TSM Scheduled Backup Failure Reports

Contents

1. MISSED
- 1.1. Is the node in question still active?
- 1.2. Is your TSM node locked?
- 1.3. Checking your machine
- 1.4. Summary
2. FAILED
3. SEVERED
4. Logging calls with the HFS Team

This article aims to assist you if you have received an e-mail advising of a MISSED, FAILED or SEVERED backup - such an email will have the subject heading "TSM Scheduled backup failure report".

If you have received this message then some or all of your data will NOT have been backed up. Therefore it is important both to resolve this issue as soon as possible to enable future scheduled backups to run; and also to perform a manual backup to ensure that your current data is backed up and secure.

The basics

Within the email there will be one or more Nodenames listed. Since each node has its own unique account name, password and installation of TSM, you will need to repeat this process for each of the nodes listed in the email.

Determining whether the scheduled backup was MISSED, FAILED or SEVERED

The email will state for each node whether the scheduled backup was MISSED, FAILED or SEVERED:
Figure /hfs/help/images/scheduledts01.gif []
For each node, check which status has been given and chose from the following options to continue:
Backup Status

1. MISSED

You have a node whose scheduled backup is reported to have been MISSED.

Manual backup - If you already know why this scheduled backup was missed and then you may just wish to run a manual backup.

To troubleshoot why your scheduled backup was missed proceed through the following steps.

1.1. Is the node in question still active?

The first thing to check is whether the machine that used this nodename has been replaced, rebuilt, or is no longer used.
- If the node has been replaced, rebuilt or is no longer used then please follow the instructions for de-registering the node so that it is no longer on the backup schedule.
- Check the TSM Self-Registration Page to ensure that you do not have two or more similarly-named registered nodes: it could be that one of them is active and being backed up, and the other does not exist and is therefore being reported as missed. If you have nodes registered to you that you know do not represent existing machines, please follow the instructions for de-registering the node.
- If the node is active and you still require it to be backed up then please proceed to the following section.

1.2. Is your TSM node locked?

If your TSM node is locked then you will not be able to back up to the HFS until it is unlocked. To check your node's status do as follows:
- Go to the TSM Self-Registration Page.
- Select the problem node and, from the drop-down list provided, choose [View Client Information].
- If your node's Locked status is Yes then you will need to contact hfs@ox.ac.uk to get it unlocked. You can do this by clicking on Request unlock on the aforementioned client information page. When it has been unlocked we suggest running a manual backup so that you can:
  - Test that the issue has been resolved;
  - Ensure that we have an up-to-date copy of your data.
- If your node's Locked status is No then also check the Last Password Change date. If the date shown is over a year old, then your password may have expired and need to be changed. To do this:
  - Click on the TSM Home Page button.
  - Select the required node and from the drop-down list provided, choose [Change Client Password] and follow the on-screen instructions to reset the node's TSM password.
  - Windows users will then need to update the TSM scheduler.
  - Once the password has been reset, we suggest running a manual backup so that you can:
    - Test that the issue has been resolved;
    - Ensure that we have an up-to-date copy of your data.
- If your account was neither locked nor had an expired password then please proceed to the following section.

1.3. Checking your machine

Was your machine left switched on overnight?
- NO - Are you registered for the Wake on LAN service?
  - NO - Your machine must be switched on to run a scheduled backup. If the machine was off when the schedule was due to run then the schedule will have been missed. You either need to leave your machine on overnight, or to register for the Wake on LAN service. Please see further our page on how to set a machine to switch on and off for scheduled backups. You are advised to run a a manual backup to ensure your data is backed up.
  - YES - Check in dsmsched.log to see if there is an entry at around your scheduled backup time, which would indicate that your computer was switched on. (The location of dsmsched.log can be looked up in our list of TSM Options Files.) Did your machine switch on, so that the backup could run?
    - NO - If the machine did not switch on, report this problem to the Help Centre as this is an issue with the Wake on Lan service. You may also wish to run a manual backup.
    - YES - Did you put your machine to sleep rather than shut it down, and if so are you running Windows Vista or 7?
      - YES - A Windows Vista/7 machine woken from sleep mode will return to sleep two minutes after wake-up - so please do not put your machine to sleep if you wish it to be woken up, but next week shut it down instead.
      - NO - if you are not a Windows Vista/7 user who put their machine to sleep, proceed to the next step to check if your machine switched itself off.
- YES - Your machine may still have switched itself off or gone into sleep mode, meaning that the scheduled backup was missed. Would your machine have definitely stayed switched on?
  - NO - most machines will go into a sleep mode after a certain period. To prevent this, you need to set power management correctly, as follows:
    - In Windows:
      - In XP/2003, go to Start > (Settings >) Control Panel > Power Options > Power Schemes (tab), and check that System standby is set to Never. Then, if your machine is a laptop whose lid you close when you leave it on for backup, click on Advanced (tab) and under When I close the lid of my portable computer choose Do nothing.
      - In Vista/7, go to Start > Control Panel > Power Options > Change when the computer sleeps (tab), and check that Put the computer to sleep is set to Never. Then click on the arrow to go back to Power Options and also check that under Choose what the power button does (tab), When I press the power button is not set to Sleep. On the same screen, if your machine is a laptop whose lid you close when you leave it on for backup, check that When I close the lid is set to Do nothing.
    - On a Mac: Go to System Preferences > Energy Saver and check that the computer (not the display) is set never to go to sleep.
  - YES - if the machine was left on and power management is set correctly, proceed to the next step.
Was your machine left connected to the physical Oxford University network?
- NO - if you were connected via a VPN connection or not connected to the network at all then the scheduled backup would not have started. You must have a physical network connection to the Oxford University network for the scheduled backups to run.
- YES - if the machine was on and there was a physical connection to the Oxford University network, please see our page on Checking the Client Scheduler for Windows, Mac or Linux/Unix machines.

1.4. Summary

You should now have performed enough troubleshooting to ensure that you know why the scheduled backup was missed and hopefully put corrective measures in place to ensure subsequent scheduled backups are successful. If you have been unable to determine the likely cause of why the backups are being missed then please follow the steps below for logging calls with the HFS Team with the appropriate amount of data.

If you have not already done so, we recommend that you run a manual backup.

To find out when your next scheduled backup is, please see the FAQ item When is my scheduled backup due to run?.

2. FAILED

You have a node whose scheduled backup is reported to have FAILED.

A failed backup generally means that TSM was successful in starting a backup but that it was unable to complete it successfully. Further investigation is required to determine how much of your data was backed up - it could be some, all, or none of it that got sent to the HFS. Objects on a user's machine that may cause a schedule to fail include:

Files that are exclusively locked open by another program and cannot be backed up, e.g. database files.
Files that are corrupt, making them unreadable.
Files that are excessively large, causing them to make the network connection time out.
Folder/File structures that breach TSM maximum file length restrictions.
Folder/File structures that create memory issues on the client machine, causing backup to fail.

Another possibility is that TSM is wrongly configured: if it is looking for a file system of partition that does not exist then such a backup would be deemed a failure - e.g., if TSM is set to back up D: but there is no D: drive present.

Troubleshooting FAILED scheduled backups

Checking the dsmsched.log file

If you are IT Support Staff, or are an advanced user and are confident reviewing and interpreting log files, then please follow the suggestions below. You will need to open the file dsmsched.log, whose location on your machine is listed in table 1.

Table 1. Log File Locations
Platform	File	Location
Windows	`dsmerror.log`, `dsmsched.log`	`C:\Program Files\tivoli\tsm\baclient`
Linux, Unix	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	`/var/log` or `/opt/tivoli/tsm/client/ba/bin`
Mac OS X	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	`/Library/Logs/Tivoli/TSM`
Netware	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	Installation directory

Once dsmsched.log is opened (on which, see below), you will need to search for ANS entries. These are in the format of ANS####? - where the # represents a number, and the ? represents either an E (Errors), W (Warnings) or I (Informational). Informational messages will not be a cause of a scheduled backup failing or being severed.

2.1. Examining dsmsched.log using a text editor:

Browse to the appropriate location and open dsmsched.log.
Once the log is open (this may take a while if it is large), scroll to the bottom of the log file: this is where the most recent information will be.
Depending on your operating system, you now need to search through the log file. In Windows, select [Edit] and then [Find] (or use CTRL+F) to bring up the search box.
In the search box type ANS, then select up and click Find Next.
Look for ANS entries that end in either an E or a W.

2.2. Examining dsmsched.log using a spreadsheet

If you find your log difficult to read then try using a spreadsheet package such as Microsoft Excel or Open Office Calc.

Open your spreadsheet package.
Click on [File] and then [Open].
In the box provided change Files of Type to be [All Files].
In the Look in: section browse to the appropriate location and then open dsmsched.log.
Generally a box will appear giving you various import options - leave the defaults and select OK.
The log file will open, with each time-stamped entry from the log appearing on a separate line.
Once there, select [Edit] and then [Find] (or use CTRL+F) to bring up the search box.
In the search box type ANS, then select up and click Find Next.
Look for ANS entries that end in either an E or an W.
Searching tip
- Excel users - In the search window you can enter ANS????E to search for Errors or ANS????W to search for warnings.
- Open Office users - In the search window click on the More Options button and tick the Regular Expressions tick box; then you can search for ANS????E to search for Errors or ANS????W to search for warnings.

2.3. Error messages

For each ANS####E or ANS####W you need to review the text which follows the error code to determine whether this could have been a cause of the scheduled backup failure. You will most likely find the message "ANS1512E Scheduled event ... failed" and at least one other message as well. Examples of common messages that cause scheduled backup failures are listed below.

2.3.1. 'ANS4037E Object ... changed during processing'

TSM may send most of your data but ultimately report overall scheduled backup failure if other files are left open. TSM only deems a schedule to have failed if one or more files have been prevented from backup in a certain way - not all file failures cause schedule failures. But Windows in particular does sometimes lock open files in such a way that it causes TSM to call a schedule failed - when really only a small number of files failed to get backed up.

In general, therefore, it is best to close all files and programs before a backup runs. To locate the problem, first of all please check your dsmerror.log to see if any file failures were caused by one or more files being changed while TSM was trying to back up. There may be lines like:

30-01-2008 00:25:28 ANS1228E Sending of object '/var/log/test.log' failed
30-01-2008 00:25:28 ANS4037E Object '/var/log/test.log' changed during processing.  Object skipped.
30-01-2008 00:25:31 ANS1802E Incremental backup of '/' finished with 1 failure

Additional information near the end of dsmsched.log will show the total number of failed files. In order to find the relevant part of text it is usually easiest to go to the end of the document, and then scroll upwards until you find an end-of-schedule report similar to the following example:

30-01-2008 00:26:04 --- SCHEDULEREC STATUS BEGIN
30-01-2008 00:26:04 Total number of objects inspected:  214,391
30-01-2008 00:26:04 Total number of objects backed up:       16
30-01-2008 00:26:04 Total number of objects updated:          0
30-01-2008 00:26:04 Total number of objects rebound:          0
30-01-2008 00:26:04 Total number of objects deleted:          0
30-01-2008 00:26:04 Total number of objects expired:          0
30-01-2008 00:26:04 Total number of objects failed:           1
30-01-2008 00:26:04 Total number of bytes transferred:    70.20 MB
30-01-2008 00:26:04 Data transfer time:                    5.07 sec
30-01-2008 00:26:04 Network data transfer rate:        14,151.96 KB/sec
30-01-2008 00:26:04 Aggregate data transfer rate:        588.20 KB/sec
30-01-2008 00:26:04 Objects compressed by:                    0%
30-01-2008 00:26:04 Elapsed processing time:           00:02:02
30-01-2008 00:26:04 --- SCHEDULEREC STATUS END
30-01-2008 00:26:04 --- SCHEDULEREC OBJECT END WEEKDAILY_OUCS 30-01-2008 00:00:00
30-01-2008 00:26:04 ANS1512E Scheduled event 'WEEKDAILY_OUCS' failed.  Return code =12.
30-01-2008 00:26:04 Sending results for scheduled event 'WEEKDAILY_OUCS'.
30-01-2008 00:26:04 Results sent to server for scheduled event 'WEEKDAILY_OUCS'.

Note, however, that it is quite normal for a few files to fail to get backed up. If you find that the files that failed also failed on days when the schedule completed successfully, then those file failures are very unlikely to be what caused the schedule to fail as a whole.

If you cannot close the file(s) that is/are causing the schedule failure before scheduled backup occurs, then you should exclude them from backup. Files that are continually open - such as log files and database files - would fall into this latter category. See further our pages on excluding files and folders from backup and backing up open files with TSM.

2.3.2. 'ANS1071E Invalid domain name entered' or 'ANS1063E The specified path is not a valid file system or logical volume name' or 'ANS1134E Drive is an invalid drive specification'

If TSM is configured to back up drives or partitions that it cannot see, then scheduled backups will fail with a message like one of the following:

30-06-2011 00:45:56 ANS1071E Invalid domain name entered: '/data/fred'

30-06-2011 00:45:56 ANS1063E The specified path is not a valid file system or logical volume name

30-06-2011 00:45:56 ANS1134E Drive is an invalid drive specification

Either the error message itself, or a message preceding or following it, will state which drive or partition is causing the problem.

There are three likely possible reasons for such an error:

If your machine is a Mac, then this could be due to a bug with early versions of TSM 6.1 for Mac, which do not analyse backup domain statements correctly.
The listed domain entry does not exist as a drive or partition:
- One reason for this could be that a folder/directory has been specified as a separate domain. The latter will cause an error because only drives or partitions may normally be used as domains: hence TSM cannot find the drive /data/fred and so it deems that the schedule has failed. In this case, /data/fred must be a folder/directory that is part of the larger partition /data or part of the root partition /.
- Alternatively it could be that a drive is listed as part of the backup domain but is no longer present on the machine. Perhaps the drive has been removed; or perhaps (on Windows machines only) the TSM backup domain contains references to UNC paths that are no longer valid (e.g. because the machine has been renamed).
There is a space in the domain name - in this case quotation marks need to be used around the drive name, because otherwise TSM will assume that you mean several domains. E.g. the above error message would occur if you wanted to back up the drive /data/fred backup but you specified the incorrect DOMAIN /data/fred backup instead of the correct DOMAIN "/data/fred backup".

To fix this problem:

If your machine is a Mac, ensure that you are running TSM 6.1.3 or higher - please check your TSM version using our instructions under Which version of TSM am I running? - and if necessary upgrade TSM.
If this is not a Mac, or if it is but upgrading does resolve the problem, run TSM (Mac users must use [TSM Tools for Administrators]) and go [Edit] > [(Client) Preferences] > [Backup] (tab) and correct your backup domain. If you are running TSM 6.1 or higher, you now need to restart the TSM scheduler: see further our instructions for Windows, Mac and Linux/Solaris on how to do this.

2.3.3. 'ANS1492S Invalid virtual mountpoint ...: File not found' (Linux/UNIX only)

The error message 'ANS1492S Invalid virtual mountpoint ...: File not found' indicates a problem similar to that described in the previous section. In this case, TSM could not find a directory that has been nominated in dsm.sys as a virtual mount point. For more information on virtual mount points, see the relevant section of our page on backing up machines which have high file counts.

To fix the problem, remove the line in dsm.sys or else correct it to point to an existing directory. Then check for the offending virtual mount point's name in dsm.opt: if your domain is not set to ALL-LOCAL then you will need to remove or correct it there too. Lastly, stop and restart the TSM scheduler, following the instructions for setting up automatic backups on Linux.

2.3.4. 'ANS1512E Scheduled event ... failed' - but no other ANS warning/error messages

Sometimes TSM may think that the schedule has failed because of a communication problem with the HFS server. In this case, you will be able to tell from the end of dsmerror.log and dsmsched.log that no files failed during the backup. For example, you may see a report in the latter file like the following:

01-11-2007 16:27:42 --- SCHEDULEREC STATUS BEGIN
01-11-2007 16:27:42 Total number of objects inspected:   31,029
01-11-2007 16:27:42 Total number of objects backed up:       62
01-11-2007 16:27:42 Total number of objects updated:          0
01-11-2007 16:27:42 Total number of objects rebound:          0
01-11-2007 16:27:42 Total number of objects deleted:          0
01-11-2007 16:27:42 Total number of objects expired:          0
01-11-2007 16:27:42 Total number of objects failed:           0
01-11-2007 16:27:42 Total number of bytes transferred:    52.47 MB
01-11-2007 16:27:42 Data transfer time:                  106.71 sec
01-11-2007 16:27:42 Network data transfer rate:          503.49 KB/sec
01-11-2007 16:27:42 Aggregate data transfer rate:        219.49 KB/sec
01-11-2007 16:27:42 Objects compressed by:                    0%
01-11-2007 16:27:42 Elapsed processing time:           00:04:04
01-11-2007 16:27:42 --- SCHEDULEREC STATUS END
01-11-2007 16:27:42 --- SCHEDULEREC OBJECT END WEEKDAILY_OUCS 10-01-2008 10:00:00
01-11-2007 16:27:42 ANS1512E Scheduled event 'WEEKDAILY_OUCS' failed.  Return code =12.
01-11-2007 16:27:42 Sending results for scheduled event 'WEEKDAILY_OUCS'.
01-11-2007 16:27:42 Results sent to server for scheduled event 'WEEKDAILY_OUCS'.

In this case, TSM has inspected 31,029 files and has backed up 62 of them. The number of failed files is zero. The TSM client has experienced an error when signing-off from the server and has recorded this as a failure. However, it is clear that the scheduled backup itself has completed and the failure message can be ignored.

2.3.5. 'ANS1030E The operating system refused a TSM request for memory allocation'

This error can occur because the amount of memory (RAM) which the TSM scheduler uses can grow with time, until it may reach a point where there is insufficient free memory for scheduled backups to be able to run. To prevent this, it is recommended that TSM users stop and restart the TSM scheduler periodically: however, if your machine is rebooted regularly then restarting the scheduler is unlikely to be necessary, because the service is restarted every time you reboot.

2.3.6. Backup simply stops - no ANS error or warning message

It is possible, though very unusual, that your machine may run out of memory during a backup and then TSM will cut out. This is most likely to happen on a Mac which is holding a very large number of files (over a million) on one drive. If your backups on a Mac, both scheduled and manual, cut out without warning, please see backups fail to complete.

2.3.7. 'ANS4023E Error processing ...: file input/output error' or 'ANS4046E There is an error processing ... the object is corrupted and unreadable' or 'ANS4047E There is a read error on .... The file is skipped.'

If TSM is having trouble reading certain files, then it could be because they are corrupted. If this is the case then you will see error messages in your dsmerror.log about certain files being unreadable by TSM. For example, they may take the form:

ANS4023E Error processing '/var/log/test.log': file input/output error.

ANS4046E There is an error processing '/var/log/test.log': the object is corrupted and unreadable.

ANS4047E There is a read error on '/var/log/test.log'. The file is skipped.

If the fault is only software-related, then the problem can be fixed by checking the disk. Basic steps are as follows, though you may want to do further research before implementing them.

In Windows, in My Computer, right-click on the offending drive (e.g. C:), and select [Properties] > [Tools]. Then, under 'Error-checking', click on Check Now and tick the box marked 'Automatically fix file system errors', then Start. Click Yes to the question about running a disk check the next time the computer is restarted. Any file system errors that are easily fixable will then be fixed on the next reboot.
On a Mac, in Finder, go [Applications] > [Utlitiies] > [Disk Utility]; then, in the left-hand window, click on the relevant drive and, in the [First Aid] tab click Verify Disk or, if appropriate, Repair Disk.
In Unix/Linux, use the command fsck to check your disk - please refer to your system documentation for the appropriate procedure.

In the worst case scenario, if you have file errors despite trying to fix them, or if you are concerned that your hard disk may have a fault, please see your local IT for advice.

3. SEVERED

You have a node whose scheduled backup is reported to have been SEVERED.

A SEVERED backup generally means a loss of communication between the TSM client on your machine and the HFS TSM server, whilst the backup was in progress. Possible reasons include:

Intervention at the client (user) end - the user forcibly cancelled the backup/stopped TSM services, or switched the machine off during the backup.
Intervention at the server end - the backup may have been cut off from the HFS server for exceeding a daily limit.
Failure (e.g. crashing) of the client machine.
TSM client machine going into sleep mode/state of hibernation.
Several large files causing multiple connection timeouts between the server and client.
Dropped network connection either at the client end, or somewhere on the network between the client and the server.
Firewall intervention prohibiting/delaying network traffic.

If you are aware of your machine crashing or the backup being forcibly cancelled then you may wish to simply run a manual backup.

To troubleshoot why your backup was SEVERED please follow the above troubleshooting steps provided for FAILED backups.

4. Logging calls with the HFS Team

If you have been through the troubleshooting steps provided and your issue has not been resolved, you need to log a call with the HFS Team by replying to the email you received advising you of the MISSED, FAILED or SEVERED backup (or e-mailing hfs@ox.ac.uk) - with the files listed below.

In order to provide effective support for this issue the HFS Team need all of the following files (for the appropriate operating system) attached to the email from the client machine affected:

Table 2. Log File Locations
Platform	File	Location
Windows	`dsmerror.log`, `dsmsched.log`	`C:\Program Files\tivoli\tsm\baclient`
Linux, Unix	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	`/var/log` or `/opt/tivoli/tsm/client/ba/bin`
Mac OS X	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	`/Library/Logs/Tivoli/TSM`
Netware	`dsmerror.log`, `dsmsched.log`, `tsm-install.log`	Installation directory

Table 3. Config File Locations
Platform	File	Location
Windows	`dsm.opt`	`C:\Program Files\tivoli\tsm\baclient`
Linux, Unix	`dsm.sys`, `dsm.opt`	`/usr` or `/opt /tivoli/tsm/client/ba/bin`
Mac OS X	`dsm.sys`, `dsm.opt`	`/Library/Preferences/Tivoli Storage Manager`
Netware	`dsm.opt`	Installation directory

For Windows users we have an automated way of sending us the required files - please see our page on log file collection for Windows.

Once sent your email will be automatically passed to the HFS Team who will review and advise you of any further action required.