Emergency shutdown of services
Control Center monitors each service's short-term storage usage trends and current usage levels. When Control Center predicts that a service is about to exhaust storage space, it initiates an automatic emergency shutdown of the service. By shutting down while enough space is available to perform recovery operations, Control Center minimizes the risk of data corruption.
Emergency shutdown is performed for services that are in a resource pool that has DFS access. Services that are in pools that do not have DFS permissions and do not write to the DFS continue running.
- On the Applications tab, the Thin Pool Usage graph shows used and available bytes for the thin pool.
- On the Applications page for each service, the DFS Usage graph shows used and available bytes for the DFS.
By comparing usage to the available storage and the amount of space that must be reserved, Control Center determines when a service must be shut down before filling the thin pool or DFS, and initiates the emergency shutdown. The browser interface identifies services in the emergency shutdown state, as does issuing the serviced service status command in the command line.
To minimize data loss, Control Center shuts down services in the following order: databases; services that cannot be recovered; indices and services that are difficult to recover; any other services. Services in emergency shutdown status cannot be restarted until the underlying cause of the shutdown is resolved.
- Examine the service that was shut down to determine why it was using excessive storage
and correct the issue. For example:
- If an application was writing a large amount of performance data to the tenant device, add space to the device. See Control Center application data storage requirements.
- If too many snapshots are stored on the device, delete those that you no longer need. See Creating snapshots and rolling back.
- If a usage anomaly might have occurred, wait for usage levels to return to normal.
- Clear the emergency shutdown flags.
- Start the service by using the browser interface or command line interface. Control Center starts services in the reverse order of shutdown.