Recovering from offline volumes by using the CLI

If a Tier 3 recovery procedure completes with offline volumes, then it is likely that the data that is in the write-cache of the node canisters is lost during the failure that caused all of the node canisters to lose the block storage system cluster state. You can use the command-line interface (CLI) to acknowledge that there was data that is lost from the write-cache and bring the volume back online to attempt to deal with the data loss.

If you run the recovery procedure but there are offline volumes, you can complete the following steps to bring the volumes back online. Some volumes might be offline because of write-cache data loss or metadata loss during the event that led all node canisters to lose cluster state. Any data that is lost from the write-cache cannot be recovered. These volumes might need extra recovery steps after the volume is brought back online.

Note: If you encounter errors in the event log after you run the recovery procedure that is related to offline arrays, use the fix procedures to resolve the offline array errors before you fix the offline volume errors.
Important: For systems that are using data reduction pools, contact IBM support for assistance in recovering offline volumes.

Complete the following steps to recover an offline volume after the recovery procedure is completed:

  1. Delete all IBM FlashCopy function mappings and Metro Mirror or Global Mirror relationships that use the offline volumes.
  2. If there are corrupted volumes in a data reduction pool, the user must run the recovervdiskbysystem command to recover all volumes.
    Note: Use this command only under the supervision of IBM Support personnel.
  3. If there are corrupted volumes in a pool, and the volumes are space efficient or compressed, run the following command:

    repairsevdiskcopyvdisk_name | vdisk_id

    This command brings the volume back online so that you can attempt to deal with the data loss.
    Note: If running the repairsevdiskcopy command does not start the repair operation, then use the recovervdisk command.
  4. If the volume is not a space efficient or compressed volume, and it is outside of a data reduction pool, then run the recovervdiskbysystem command. This brings all corrupted volumes back online so that you can attempt to deal with the data loss.
  5. Refer to What to check after running the system recovery for what to do with volumes that are corrupted by the loss of data from the write-cache.
  6. Re-create all FlashCopy mappings and Metro Mirror or Global Mirror relationships that use the volumes.
Parent topic: Recover system procedure