Collocation is the process of grouping your nodes together (collocating) to help improve restore performance.

Over time as you add more and more nodes to your environment, the data for these nodes will slowly become more and more dispersed across your tapes as you perform backups.  When restoring the data if only a small number of files are required, this will not be a problem but when a restore is needed for a large number of files and those files have become dispersed across a large number of tapes then restores time will become an issue. I have personally experienced an issue when restoring a large volume that took over 2 days when not using collocation.

Instead of having every node’s data being placed on tapes at TSM’s discretion, collocation allows you to define groups of nodes to be stored together on your tapes.

As with everything there is a trade-off, collocation groups will require the use of more tapes but can significantly improve restore times.

Badly configured collocation will cause the consumption of very large amount of tapes.  And in my experience this has been the number one cause of the “Where have my all my scratch tapes gone” and “Why do I have so many filling tapes” , we will explore this later in the document.

Collocation is set at the Storage Pool so it is possible to have different levels of collocation based on Storage Pools.  As Collocation is really designed to help with Physical Tapes it only applies to Primary Sequential and Copy Storage Pools.

There are a number of difference types of collocation methods that can be used

NO – Collocation is disabled for the Storage Pool

FILESPACE – TSM Attempts to put each file space on its own tape

NODE – TSM Attempts to put each Node on its own tape

GROUP – TSM will put groups of Nodes on their own tapes as defined in the collocation groups.  If no collocation groups are defined then the storage pool reverts to Node Collocation.

NOTE:  If Group Collocation is set and there are no collocation groups defined or there are nodes not defined in any collocation group then you very quickly run out of scratch tapes and see a lot of filling tapes with not much data on them

Firstly let’s look at how to check to see if collocation is set.  From the Admin CLI issue the command

Q STGPOOL F=D and check the status of the collocate field

From TSM Studio open the Storage category and open the Storage Pools Dataview and the check the value in the Collocate column

To change the collocation method being used

Admin CLI:  UPDATE STGPOOL COLLOCATE=GROUP

From TSM Studio

Go to the Storage Pools dataview and right click on the storage pool(s) you want to change

and choose Collocation

Choose the collocation method and press Ok

The next thing to do will be create the collocation groups

From TSM Studio open the Configuration category and open the Collocation dataview

Right Click and Choose Add Group

Enter the Group Details and then press Ok

Next Add Members to the Group, From the Collocation Dataview right click the Group and Choose Edit Group Members

NOTE: A Node can only be a member of one collocation group

Simply move nodes in and out of the group and press Ok

Read Here for Trouble Shooting Tips

Recommendations:  If you have plenty of tapes and will never run out then use FILESPACE or NODE.

If Tapes are at a premium and you dont care how long a restore will take then dont use collocation

Otherwise you will need to find a happy medium between tape usage and restore times using the Group method.  I have seen a number of different methods for collocating node, such as grouping by application type or OS Type.  I suggest putting nodes with small amounts of data that does not change very often together and putting nodes with large amounts of data by themselves.  For the rest of the nodes try and keep each group from using more that 50 tapes for any node in the group ( this number will depend on the number of drives you have, how fast the library can load the tapes and how fast the drive can move to the required location on the tape) to ensure fast restore times.

I have seen collocation disabled for copypools as a lot of organizations already replicate the critical data to their Business Recovery sites using other methods

As with everything I say in these articles test, test end test again

NOTE:  Virtual Tape Libraries do not have the same limitations as Physical Tapes. As the mount times and seek times are negligible collocation can be disabled without affecting restore times.

Permalink.

In this article I will show three ways that you can verify that your Backups were successful using TSM Studio.

The first is from the Events Table

To view the events table goto the Monitor Category and then double click on the Events Dataview

Select the Date Range

The events data is displayed

There are a number of other related views in this Category such as Missed and Failed Backups

View the Backup output from the Summary Table

To view the events table goto the Monitor Category and then double click on the Summary Dataview

Select the Date Range

Use the Filter Backup to just show Backups

Choose Backup

Filter the Successful Column to display Backup Jobs that have failed

A list of all the Failed Backups will now be displayed

The third way to check is to look at File Spaces

The file spaces dataview can be found under the Storage Category

Check the Backup Start Date and the Backup End Date.  If the Backup End Date is less than the Backup Start Date or the Backend End Date is blank then there is a problem with the backup of the Node owning this filespace

We also have a specific Dataview to show all suspect File Spaces, goto to teh Troubleshooting Category and select File Spaces where Backup End Date is less than Backup Start Date

Permalink.

The reuse delay parameter is used to define how long a tape will stay in a pending state. It is specified in days and can be applied individually to primary sequential and copy storage pools (in other words storage pools that handle tapes).

Why do we need this parameter?   This parameter is used to ensure that if you had to restore your TSM Database to a previous days backup that the tapes will be in a consistent state with the TSM Database as at the point of the restore.

To Illustrate this, assume that the reuse delay parameter is set to 0 so that tapes become scratch tapes as soon as they are empty.

Monday

A TSM Database Backup is taken on Monday

Tape A1 currently has data for Node ZZ

Tuesday

The data for Node ZZ on tape A1 expires and because the reuse delay parameter is set to 0 tape A1 becomes a scratch tape

Tape A1 is now immediately used to backup node YY’s data

Wednesday 

For some reason, you have to restore to the last TSM database backup which was taken on the Monday.

Because the database is now running as at Monday TSM believes that tape A1 holds the data for Node ZZ.  A decision is made to restore data for Node ZZ which resides on A1 and TSM complains that the volume is corrupt because the data on the tape is not what TSM is expecting. In this case if the reuse delay parameter had been set to 2 days then tape A1 would not have been written on and the restore would work as expected.

How do I check for the tape status ?

Tapes that are empty but not yet available to be used as scratch tapes have a status of PENDING. To view pending tapes in TSM Studio open the Libraries and Volumes category and double click on the Volumes Dataview. As seen below check the “Status” column and look for tapes in a ‘PENDING” state.

To see when these tapes will become scratch tapes, check the Pending Date.  This is the date that the tape went into a pending state. Add the Reuse Delay value to determine when the tape will go from a Pending State to becoming a scratch tape.

To find the Reuse Delay value check the Storage Pool that the Volume is in

Then goto the Storage Pools dataview locate the same Storage Pool name and check the Reuse Delay Column

How do I set the value for this parameter ?

I recommend that you set the value of this parameter to be equal with the number of days or TSM Database backups you keep.

To change the reuse delay parameter from the command line

Update STGPOOL STGPOOL_NAME REUSEDELAY=days

Where STGPOOL_NAME is the name of the storage pool and days is the value for the REUSEDELAY parameter

UPDATE STGPOOL TAPEPOOL REUSEDELAY=2 for 2 days

To change the reuse parameter from TSM Studio goto the Storage Pools Dataview, select the storage pool that you wish to modify, right click to bring up the context menu. Select Reuse Delay and choose the value (in days) you wish to set. Press Ok

Permalink.

Because TSM Studio saves the last known position of the window, it is possible that if the resolution changes, multiple screens were used or the window was accidently dragged off the screen that the TSM Studio is now on a part of the Window Screen that is not visible. To resolve this issue use ALT+CTRL+W to reset the position of the window

Permalink.

The Mount Retention parameter specifies how long to keep a tape in a tape drive after it has finished being used. By default the value of this parameter is set to 60 minutes. So a tape will stay in the drive for 1 hour after it is finished with.

Why?  The main reason for having this parameter high is that a tape mount is a relatively expensive operation in that is can take some time for a tape library to unmount and mount a tape, so if the same tape is likely to be used again in the near future it will not need to re-mounted if it is already in the tape drive thus saving time either for backup or restore tasks. If you have a shared library environment or do not have a lot of drives then I recommend you bring this value right down to around 5 mins ( or even 1 minute if tape drive contention is a big issue ). With Virtual Tape Libraries tape mounts operations are extremely fast and high mount retention values are actually detrimental to the performance of the system. In this case the mount retention parameter should be set to 1 minute. The mount retention parameter is defined in a device class. Device Classes are defined per Tape Library and each device class affects all the drives in the library it is defined with.

To change the Mount Retention parameter using the command line update devclass device_class_name MOUNTRETENTION=minutes   where device_class_name is the name of the device class minutes is the number of minutes to keep the tape in the drive after it is finished being used

To change the Mount Retention parameter using TSM Studio Firstly open the Device Classes dataview in the Configuration Category and double click to open the dataview

Find the Library Column and locate the library that you wish to change

Select this row and right click to bring up the context menu for this object and select Change Mount Retention

Choose the Mount Retention Value and press Enter to Change

Permalink.

If SQL Server is running on the same Server as the TSM Studio Server service then we recommend 12GB of memory as a minimum to ensure the healthy running on TSM Studio Server.

By default SQL Server will be configured to use all available memory this will need to be changed so that there is enough free memory for  TSM Studio Server to function. It is recommended  that SQL Server be given 4GB of memory and the other 8GB is left available for TSM Studio Server.

To change the default memory settings in SQL Server. Start up SQL Server Management Server right click on the server and select properties

Select Memory and then enter 4096 into the Maximum server memory field

Permalink.

In TSM Studio Server 2.9.011 and above there are a number of added enhancements to help tune TSM Studio for you environment.

The first area to check is on the Concurrency Tab of the TSM Studio Server properties dialog which can be accessed from Configuration–>Properties as shown here



 

The main setting to check here are Concurrent Reports and Concurrent Collector Threads. Report creation can consume large amount of memory so it is recommended to keep this number low.



Also some of the collections can consume a large amount of transaction log space especially the activity log collector.  Each collector  can be individually tuned.  Goto Configuration->Collection schedules and edit the Activity Log collector and change the Concurrent Collectors to a small number (less than 5 )



Also check out the below link for SQL Server memory settings


https://www.spiritsoftware.biz/knowledgebase/microsoft-sql-…-studio-server

Permalink.

As of TSM Studio Server version 2.9.1.0 security has been tightened up and no longer stores any security information in the registry.

The default logon to TSM Studio Server is now admin with a default password of password, (Domain field should be blank otherwise TSM Studio Server will assume it is a windows userid).

We strongly advise changing the admin password using the following procedure.

To change the password for the inbuilt admin userid use the Security–>Change Admin Password

Security in TSM Studio Server is now defined by assigning role(s) and Windows groups to a TSM Studio Server security group.

The TSM Studio Server security system will automatically create a group named Admins that has access to all functions in TSM Studio Server, the first time it is loaded it will look for the old group definitions in the registry and load them into this group otherwise it will populate with following default groups Administrators, Domains Admins and TSMStudioAdmins.

Also two roles will be created, one called Administrators which is assigned to the Admins Server group and one called Operational Report Viewer for allowing access to the Operational Report Viewer.

To access TSM Studio Server Groups select Security–>Groups

To modify the Admins Group Double Click or select edit from the Toolbar

Add Windows Group that should have admin access to TSM Studio Server, One Group per line

Permalink.

Alternate Row BackColor Example

Permalink.

Permalink.

This will display Line Numbers as shown below

Permalink.

Permalink.

Permalink.

If you find that it takes upwards of 30 seconds of  establish a session to the TSM Server from the Admin CLI (dsmadmc.exe). It maybe because your TSM server is unable to resolve the clients name or the TSM Server is unable to communicate with any DNS Servers.

Permalink.

TSM has the ability to encrypt data at the client node before sending the data to the TSM server. There are two methods that are available

1. Transparent Encryption

• This is where the encryption key is managed by and stored on the TSM server
• If the client node needs to be rebuilt data can be easily restored
• Data can be restored back to any node that is allowed to impersonate the original node

2. Client Side Encryption

• This is where the encryption key is manually managed and stored on the client using an encryption password
• More secure as data can only be restored if the encryption password is known
• If the password is lost then the data cannot be restored

To enable encryption at the client there are two parameters for setting up and a couple of include and exclude statements for selecting or excluding which files are to be encrypted.

ENCRYPTKEY

The ENCRYPTKEY option is used to choose either transparent encryption or client-side encryption. For client-side encryption there are two options to choose from

ENCRYPTKEY=SAVE  ( Client-Side )

This option will prompt for an encryption password on the initial backup and then store this password in the password file. The password will be retrieved from this file for each subsequent backup.

ENCRYPTKEY=PROMPT ( Client-Side)

This option will prompt for an encryption password for each backup and restore. To be able to restore the data the same password that was using when backing the data up will be required

ENCRYPTION=GENERATE (Transparent)

This option will have TSM generate an encryption key password which is stored on the TSM server and managed by the TSM server.

 ENCRYPTIONTYPE

The ENCRYPTIONTYPE parameter selects what type of encryption is used either DES56 or AES128 with the AES128 algorithm being the stronger of the two

Next is to select which file or directories to include in the backup

use the include.encrypt statement to include files and directories to be encrypted and takes the same format as any other include statement
use the exclude.encrypt statement to exclude files and directories to be encrypted and takes the same format as any other exclude statement

example

ENCRYPTKEY=GENERATE
ENCRYPTIONTYPE=AES256
INCLUDE.ENCRYPT /home/…/
EXCLUDE.ENCRYPT /home/…/test.fil
INCLUDE.ENCRYPT  C:…*
EXCLUDE.ENCRYPT  C:windows…*

When using the client-side encryption the encryption passwords are stored in the TSM.PWD files in unix or in the registry for windows

I would recommended using transparent encryption unless you have a specific requirement not to.

I am option asked how to prove that the data is encrypted. There is no way to do this with TSM and they only way to do this is use a network packet tracing tool such as wireshark. If you are interested on how to do this just send me an email  gelliott@spiritsoftware.biz

For more information see Chapter 5 of IBM Tivoli Storage Manager: Building a Secure Environment

Permalink.

The following command will show all unique management classes used by a node

SELECT DISTINCT(CLASS_NAME) FROM BACKUPS WHERE NODE_NAME = ‘XXXXXX’

to see only management classes used on directories use

SELECT DISTINCT(CLASS_NAME) FROM BACKUPS WHERE NODE_NAME = ‘XXXXXX’ AND TYPE = ‘DIR’

to see only management classes used on files use

SELECT DISTINCT(CLASS_NAME) FROM BACKUPS WHERE NODE_NAME = ‘XXXXXX’ AND TYPE = ‘FILE’

Permalink.

In Windows you can use the Preview Include-Exclude option in the backup client.

Start the backup client and go to the utilities menu and select Preview Include-Exclude

In the Preview Include-Exclude Dialog, choose the type either Backup or Archive ( Your normal nightly jobs will usually be of type backup )

Choose either to just show included files, excluded files or both. This information is ascertained from settings in your dsm.opt file

Choose  the directories and files to be reported on. If you have a large file system and you choose a top level directory this command may take a long time to run and produce a large output file

Lastly choose the output file and select ok

When the task finishes open the output file with notepad and it will look as follows with the files on the left and the management class on the left hand side

The same preview include-exclude can be achieved with the backup-archive command line in both unix and windows as follows.  This example will produce the same out as above

the -traverse=y tells the backup-archive client to include subdirectories

Permalink.

This article describes installation and configuration of the TSM Scheduler Service on windows. The TSM Scheduler service is used by the TSM Scheduling Services to automate client node backups.

I am using the version 6 client for these examples, even though the screens look different it is still the same process for the version 5 clients

After installing the Windows Backup Archive client start the Backup-Archive GUI go to the Utilities–>Setup Wizard

Select Next

Select Next on TSM Schedule Wizard Screen

Choose Install a new or additional scheduler

Choose a Name for the Service. This the name you will in the Windows Services Manager

This is what you will see in Windows Service Manager

Choose an options file to be associated with this TSM Scheduler Service. It is possible to run multiple TSM Scheduler Service’s using different option files

Choose the node name for this TSM Service. Again it is possible to use a different node name for each TSM Scheduler Service and input the password. This node name will need to be registered on the TSM Server or the service will not start.

Choose the userid that the service will start as.  This userid will need to have access to all the files it must backup or be a member of the backup operators group in windows

Choose the location for the scheduler service log files

NOTE:  Some of these options are stored with the TSM Scheduler service and not in the options file and will override the option file settings. These values are stored in the registry at the following location

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTSM Client SchedulerParameters     (replace TSM Client Scheduler with the name you used when installing the service )

Permalink.

SELECT NODE_NAME, COUNT(DISTINCT VOLUME_NAME) AS VOLUME_COUNT FROM VOLUMEUSAGE GROUP BY NODE_NAME ORDER BY VOLUME_COUNT

or in reverse order

SELECT NODE_NAME, COUNT(DISTINCT VOLUME_NAME) AS VOLUME_COUNT FROM VOLUMEUSAGE GROUP BY NODE_NAME ORDER BY VOLUME_COUNT DESC

Permalink.

For TSM version 5

SELECT * FROM NODES WHERE NODE_NAME NOT IN ( SELECT NODE_NAME FROM COLLOCGROUP WHERE NODE_NAME IS NOT NULL )

For TSM version 6 and above

SELECT * FROM NODESVIEW WHERE NODE_NAME NOT IN ( SELECT NODE_NAME FROM COLLOCGROUP WHERE NODE_NAME IS NOT NULL )

Permalink.

TSM v6 and Above

SELECT NODE_NAME FROM NODESVIEW WHERE NODE_NAME NOT IN (SELECT NODE_NAME FROM OCCUPANCY)

Permalink.

For TSM v 5

SELECT COUNT(*) FROM NODES

For TSM v6 and above

SELECT COUNT(*) FROM NODESVIEW

Permalink.

SELECT DOMAIN_NAME,SCHEDULE_NAME,COUNT(NODE_NAME) AS NUMBER_OF_NODES FROM ASSOCIATIONS GROUP BY DOMAIN_NAME,SCHEDULE_NAME

Permalink.