Restoring the system configuration

This configuration restore procedure is designed to restore information about your configuration, such as volumes, local Metro Mirror information, local Global Mirror information, storage pools, and nodes. The data that you wrote to the volumes is not restored. To restore the data on the volumes, you must restore application data from any application that uses the volumes on the clustered system as storage separately. Therefore, you must have a backup of this data before you follow the configuration recovery process.

If encryption was enabled on the system when its configuration was backed up, then at least 3 USB flash drives need to be present in the node canister USB ports for the configuration restore to work. The USB flash drives do not need to contain any keys. They are for generation of new keys as part of the restore process.

You must regularly back up your configuration data and your application data to avoid data loss. If a system is lost after a severe failure occurs, both configuration for the system and application data is lost. You must restore the system to the exact state it was in before the failure, and then recover the application data.

During the restore process, the nodes and the storage enclosure will be restored to the system, and then the MDisks and the array will be re-created and configured. If there are multiple storage enclosures involved, the arrays and MDisks will be restored on the proper enclosures based on the enclosure IDs.

Important: There are two phases during the restore process: prepare and execute. You must not change the fabric or system between these two phases.

If you do not understand the instructions to run the CLI commands, see the command-line interface reference information.

To restore your configuration data, follow these steps:

Verify that all nodes are available as candidate nodes before you run this recovery procedure. You must remove errors 550 or 578 to put the node in candidate state. For all nodes that display these errors, follow these steps:
1. Point your browser to the service IP address of one of the nodes, for example, https://node_service_ip_address/service/.
2. Log on to the service assistant.
3. From the Home page, put the node into service state if it is not already in that state.
4. Select Manage System.
5. Click Remove System Data.
6. Confirm that you want to remove the system data when prompted.
7. Exit service state from the Home page. The 550 or 578 errors are removed, and the node appears as a candidate node.
8. Remove the system data for the other nodes that display a 550 or a 578 error.
  All nodes previously in this system must have a node status of Candidate and have no errors listed against them.
  
  Note: A node that is powered off might not show up in this list of nodes for the system. Diagnose hardware problems directly on the node using the service assistant IP address and by physically verifying the LEDs for the hardware components.
  
  Warning: If you use the management GUI for the initial setup to restore the system configuration, check if a default call home email user was created. If it was created, delete the default call home email user in order for the T4 system recovery to proceed successfully.
Verify that all nodes are available as candidate nodes with blank system fields. Perform the following steps on one node in each control enclosure:
1. Connect to the service assistant on either of the nodes in the control enclosure.
2. Select Configure Enclosure.
3. Select the Reset the system ID option. Do not make any other changes on the panel.
4. Click Modify.
Create a system.
- If your system is a Lenovo Storage^® V7000 Gen2 system, use the technician port.
- If your system is a Lenovo Storage V7000 Gen1 system, use the initialization tool that is available on the USB flash drive. Select the Initialize a new Lenovo Storage V7000 (block system only) option from the Welcome panel of the initialization tool.
In a supported browser, enter the IP address that you used to initialize the system and the default superuser password (passw0rd).
The setup wizard is shown. Be aware of the following items:
1. Accept the license agreements.
2. Set the values for the system name, date and time settings, and the system licensing. The original settings are restored during the configuration restore process.
3. Verify the hardware. Only the control enclosure on which the clustered system was created and directly attached expansion enclosures are displayed. Any other control enclosures and expansion enclosures in other I/O groups are added to the system later.
If you set up email notification in the setup wizard, you must now remove that email user and server so that the original configuration can be restored.
Issue the following CLI command to remove the new email user:
```
rmemailuser 0
```
Issue the following CLI command to remove the new email server:
```
 rmemailserver 0
```
From the management GUI, click Access > Users and configure an SSH key for the superuser.
By default, the newly initialized system is created in the storage layer. The layer of the system is not restored automatically from the configuration backup XML file. If the system you are restoring was previously configured in the replication layer, you must change the layer manually now. For more information about the replication layer and storage layer, see the System layers topic in the Related concepts section at the end of the page.
If the clustered system was previously configured as replication layer, then use the chsystem command to change the layer setting.
For configurations with more than one I/O group, add the rest of the control enclosures into the clustered system.
1. From the management GUI, select Monitoring > System.
2. Select Actions > Add Enclosures.
3. Continue to follow the on-screen instructions to add the control enclosures. Decline the offer to configure storage for the new enclosures when asked if you want to do so.
Identify the configuration backup file from which you want to restore.
The file can be either a local copy of the configuration backup XML file that you saved when you backed-up the configuration or an up-to-date file on one of the nodes.

Configuration data is automatically backed up daily at 01:00 system time on the configuration node.

Download and check the configuration backup files on all nodes that were previously in the system to identify the one containing the most recent complete backup
1. From the management GUI, click Settings > Support.
2. Click Show full log listing.
3. For each node (canister) in the system, complete the following steps:
  1. Select the node to operate on from the selection box at the top of the table.
  2. Find all the files with names that match the pattern svc.config.*.xml*.
  3. Double-click the files to download them to your computer.
The XML files contain a date and time that can be used to identify the most recent backup. After you identify the backup XML file that is to be used when you restore the system, rename the file to svc.config.backup.xml.

Copy onto the system the XML backup file from which you want to restore.

pscp full_path_to_identified_svc.config.file 
superuser@cluster_ip:/tmp/svc.config.backup.xml

Issue the following CLI command to compare the current configuration with the backup configuration data file:
```
svcconfig restore -prepare
```
This CLI command creates a log file in the /tmp directory of the configuration node. The name of the log file is svc.config.restore.prepare.log.
Note: It can take up to a minute for each 256-MDisk batch to be discovered. If you receive error message CMMVC6200W for an MDisk after you enter this command, all the managed disks (MDisks) might not be discovered yet. Allow a suitable time to elapse and try the svcconfig restore -prepare command again.

Issue the following command to copy the log file to another server that is accessible to the system:

pscp superuser@cluster_ip:/tmp/svc.config.restore.prepare.log 
full_path_for_where_to_copy_log_files

Open the log file from the server where the copy is now stored.
Check the log file for errors.
- If you find errors, correct the condition that caused the errors and reissue the command. You must correct all errors before you can proceed to step 17.
- If an error indicates that the system layer will not be restored, return to 8, configure the layer setting correctly, and then continue the restore process from 12.
- If you need assistance, contact the Lenovo Support Center.
Issue the following CLI command to restore the configuration:
```
svcconfig restore -execute
```
This CLI command creates a log file in the /tmp directory of the configuration node. The name of the log file is svc.config.restore.execute.log.

Issue the following command to copy the log file to another server that is accessible to the system:

pscp superuser@cluster_ip:/tmp/svc.config.restore.execute.log 
full_path_for_where_to_copy_log_files

Open the log file from the server where the copy is now stored.
Check the log file to ensure that no errors or warnings occurred.
Note: You might receive a warning that states that a licensed feature is not enabled. This message means that after the recovery process, the current license settings do not match the previous license settings. The recovery process continues normally and you can enter the correct license settings in the management GUI later.

When you log in to the CLI again over SSH, you see this output:
```
IBM_2076:your_cluster_name:superuser>
```
After the configuration is restored, verify that the quorum disks are restored to the MDisks that you want by using the lsquorum command. To restore the quorum disks to the correct MDisks, issue the appropriate chquorum CLI commands.

You can remove any unwanted configuration backup and restore files from the /tmp directory on your configuration by issuing the following CLI command:

svcconfig clear -all