High Availability (HA) and Disaster Recovery (DR) are both important factors to consider when deploying Claroty solutions. Deploying and planning for HA/DR allows greater availability and continued business operations, which is critical to get the most value out of the Claroty solutions.

When running Continuous Threat Detection (CTD) or Secure Remote Access (SRA) as a VM on vSphere, it is recommended to leverage the existing features on the vSphere cluster that provide HA and DR capabilities. For instance, DRS automatically migrates the CTD VMs to a separate host if the current ESXi host for CTD starts to degrade. More information about how to leverage vSphere HA and DR capabilities beyond the test cases in this section can be found in the High availability overview.

CTD HA and DR

One of the most critical functions within the CTD solution is asset and network visibility. This is how CTD learns about information on the network. If visibility is somehow hampered, then the platform may potentially miss a critical event that can indicate some sort of compromise (for example, a new asset is added to the network and is communicating with a known bad domain name). Maintaining the availability of CTD to ingest network traffic is critical for continued effective monitoring of the network.

CTD vSphere HA failover

CTD Server has been validated to work with the native vSphere failover feature (using DRS). The test case below can be run to validate a failover from one host to another in the same cluster during a host failure. Verification is done by validating that the IP connectivity between CTD Server and CTD Sensor is maintained after the failover.

Note: This same type of test and HA failover on vSphere can be done for connections between CTD Servers and Claroty EMC.

Prepare for HA failover test

CTD Sensor is connected to CTD Server and actively sending data.
CTD Sensor is on a separate ESXi host than the CTD Server.

Prior to the simulated node failure, make sure that CTD is receiving data from the attached CTD Sensor.
Log in to CTD, navigate to Settings > Management > Deployment Architecture.
Confirm that the Sensor IP is listed and Status is listed as Connected.
Navigate to Settings > Data Sources > Interface Configuration > 'CTD site name'.
Click the Record button for the management interface (circle button).
In the following prompt, enter information such as:
1. File name: Pre-HA-test
2. Filters for dumpcap command: host <IP-address-CTD-sensor>
Click Save.
Let this run for 5–10 seconds and then click the Stop button (square button).
Click the Download button (next to the Record/Stop button).
Open the PCAP file and confirm that there are SSH packets between the CTD Server and CTD Sensor.

Test CTD HA failover

Log in to the vSphere cluster hosting the CTD Server.

Note: It is best to choose an ESXi server in the cluster that is dedicated for testing. This ESXi server should only be running the CTD VM.
Induce a failure to the ESXi host (such as a shutdown).
Return to the vSphere cluster. The ESXi host should list Not Responding. Refresh the vSphere client until the CTD VM is back to being connected and the Host has changed to a new one.
Return to CTD and ensure that it is accessible.
Navigate to Settings > Management > Deployment Architecture.
Confirm that the Sensor IP is listed and that its Status is listed as Connected.
Navigate to Settings > Data Sources > Interface Configuration > 'CTD site name'.
Start another PCAP capture to confirm that CTD is still receiving data from the CTD Sensor.
1. Click the Record button for the Management interface (circle button).
2. In the following prompt, enter information such as:
  1. File name: Post-HA-test
  2. Filters for dumpcap comamnd: host <IP-address-CTD-sensor>
3. Click Save.
Confirm that there are still active communications between CTD Site and CTD Sensor over port 22 (SSH).

CTD backup and restore

CTD instance has been populated with asset information.
CTD instance has users and security settings configured.

One method to ensure continued operations for CTD in the case of a disaster is to back up configuration files and restore those settings. This test case validates that CTD successfully generates a backup file of the host’s current state, including system settings and data. Keep in mind that the frequency of backups should be defined by the organization’s target RPO.

Log in to a CTD Server.
Navigate to Visibility > Overview.
Record the statistics listed on the page, such as number of total assets.
Navigate to Risk & Vulnerabilities > Overview.
Record the statistics listed on the page, such as assets with unsecured protocols.
Navigate to Threat Detection > Overview.
Record the statistics listed on the page, such as open stories.
Navigate to Investigation > Protocol Summary.
Record the statistics listed on the page, such as the protocols, count and port number.
Navigate to Settings > User Management > Users.
Record list of users.
Navigate to Settings > Management > General > OS Configuration > Backup & Restore.

Note: This runs an immediate backup of the files. Users can also navigate to Settings > Management > General > BACKUPS and set scheduled backups or local backups from there. Only OS Configuration backups can be done from the CTD Sensor.
(Option 1: Local backup): Click Run Backup.
(Option 2: SMB backup): Click LOCAL and change it to SMB.
1. Fill out information for the local SMB share.
2. Click Test – Confirm thatSuccess is displayed.
3. Click Run Backup.
CAUTION: Only do these next steps in a testing environment, and verify that the user has knowledge of default credentials.
To reset the system entirely, navigate to Settings > Management > General > SYSTEM RESET.
Click the System Reset button, unselect Keep my users and groups, and click OK.
After simulating data loss using the system reset, log back into CTD.
Navigate to Settings > Management > OS Configuration > Backup & Restore.
Under Restore, click Choose backup. Select the exported backup and then click Upload backup.
After the backup is uploaded, click Restore Latest.

Note: The user may have to sign out and sign back in.

To verify, perform the following steps:

Navigate to the following pages and verify that the results match the recorded information from the Configuration section of this test case:
1. Visibility > Overview
2. Risk & Vulnerabilities > Overview
3. Threat Detection > Overview
4. Investigation > Protocol Summary
5. Settings > User Management > Users
6. Settings > Management > Custom Attributes

SRA HA and DR

HA is a crucial component for SRA to offer continued availability for remote sessions. Planning for and deploying HA helps avoid scenarios where third-party support may be unable to provide remote support if the system experiences downtime. SRA provides built-in HA capabilities for both SAC and Site, but can also benefit from vSphere HA features. For DR, it is important to create backups of system configurations in case of any loss, so it is quick to bring back remote session capabilities. Another consideration is to backup any recorded sessions, which are stored on the SRA hosts by default.

Test SRA SAC vSphere HA failover

SRA SAC and SRA Site are deployed with at least one target server configured.
SRA SAC is deployed on a vSphere cluster with HA enabled.

The SRA SAC has been validated to work with the native vSphere failover feature (using DRS). The following test case can be run to validate a failover from one host to another in the same cluster during a host failure. Verification is done by successfully initiating a remote session before and after a host failover.

Log in to the SRA SAC.
Confirm that All sites are up is listed on the top-right of the screen.
Select a server to connect to and click the Connect button.
Confirm a successful connection and that the user can interact with the session.
Before the failover, log in to the vSphere cluster hosting the SRA SAC.

Note: It is best to choose an ESXi server in the cluster that is dedicated for testing. The ESXi server should only be running the CTD VM.
Induce a failure to the SRA SAC’s ESXi host (such as a shutdown).
Return to the vSphere cluster. The ESXi host should list Not Responding and the VM states Disconnected.
Refresh the vSphere client until the SRA SAC VM is back to being connected and the Host entry in the VM's Summary tab has changed to a new one.
Refresh or reopen a connection to the SAC. Log back in and verify that All sites are up is listed on the top right of the screen.
Connect to the same remote server as before, using the Connect button.
Confirm a successful connection and that the user can interact with the session.

Test SRA Site vSphere HA failover

SRA SAC and SRA Site are deployed with at least one target server configured.
SRA Site is deployed on a vSphere cluster with HA enabled.

The SRA Site has been validated to work with the native vSphere failover feature (using DRS). The following test case can be run to validate a failover from one host to another in the same cluster during a host failure. Verification is done by verifying that the SAC is still able to connect to the site and that a remote session can be initiated using the Site post failure.

Log in to the SRA SAC.
Confirm that All sites are up is listed on the top right of the screen.
Select a server to connect to and click the Connect button.
Confirm a successful connection and that the user can interact with the session.
Before the failover, log in to the vSphere cluster hosting the SRA Site.

Note: It is best to choose an ESXi server in the cluster that is dedicated for testing. The ESXi server should only be running the CTD VM.
Induce a failure to the ESXi host (such as a shutdown).
Return to the vSphere cluster. The ESXi host should list Not Responding and the VM will state Disconnected.
Refresh the vSphere client until the SRA Site VM is back to being connected and the Host entry in the VM's Summary tab has changed to a new one.
Refresh or reopen a connection to the SAC. Log back in and verify that All sites are up is listed on the top right of the screen.
Connect to the same remote server as before, using the Connect button.
Confirm a successful connection and that the user can interact with the session.

SRA HA – SAC

SRA SAC is deployed as an HA configuration and is accessible using VIP.
There is a connected site with at least one target server.

SRA offers a native HA feature that creates an HA cluster between two SAC hosts. At a high level, the SAC nodes replicate databases between each other and share a Virtual IP (VIP) address, which is used to access the SAC HA cluster. One SAC is chosen as a primary node, meaning that the VIP forwards traffic to the primary SAC. The secondary SAC is on standby and becomes active if the primary SAC fails. This test validates the native SRA HA feature by simulating a failure to the primary SAC node. Afterwards, the test verifies that the SAC VIP is still accessible as the secondary node takes over as primary.

Browse to the SAC VIP.
Log in with an admin account and click Connect on one of the servers listed on the dashboard.
Confirm a successful remote connection.
Exit the session, but stay logged in.
Simulate a failure of the primary SAC. Either disconnect the primary SAC management interface or shut off the device/VM to simulate a loss of connection.
After the failover, refresh the browser page for the VIP. Confirm that the user is still logged in. This might take some time, depending on the setup.
Go to Activities.
Confirm that there is an entry with Action: High Availability - System failover succeeded. On the same entry, the Description should state that the secondary SAC is now the primary node.
Return to the Dashboard page and open the same remote session as before.
Confirm that it connects successfully.

SRA HA – Site

SRA Site is deployed as an HA configuration and is accessible using VIP.
SRA Site is connected to an SRA SAC.
There is a target server for remote session connectivity.

SRA offers a native HA feature that allows a HA cluster between two Site hosts. At a high level, the Site nodes replicate databases between each other and share a Virtual IP (VIP) address, which is used to access the Site HA cluster. One Site is chosen as a primary node, meaning that the VIP forwards traffic to the primary Site. The secondary Site is on standby and becomes active in case the primary Site fails. This test validates the native SRA HA feature by simulating a failure to the primary Site node. Afterwards, the test verifies that the Site VIP is still accessible and that connections can be initiated using the Site HA cluster.

Log in to the SRA Site using the VIP. Confirm that the HA cluster is listed as up.
Log in to the SRA SAC with an admin account and click Connect on one of the servers listed on the dashboard.
Confirm a successful remote connection.
Exit the session, but stay logged in.
Simulate a failure of the primary Site. Either disconnect the primary Site's management interface or shut off the device/VM to simulate a loss of connection.
After the failover, refresh the page of the Site VIP in the browser and confirm that the status now states that the HA cluster is down.
Navigate to Activities.
Confirm that there is an entry with Action, High Availability - system failover succeeded. The Description should state that the secondary site is the now the primary node of the Site cluster.
Return to the SRA SAC and reopen a remote session on the same server.
Confirm that the session successfully connects.

SRA recommended strategies for DR

Depending on how SRA is deployed, there may be multiple options to prepare the SRA solution for disaster recovery. For instance, if an SRA component is deployed as a VM on a vSphere cluster, then users can leverage existing features on vSphere to help with DR. A specific example of this is to failover an SRA VM to an alternate site that is part of the vSphere cluster but is geographically separate from the rest of the hosts on the cluster. vSphere offers other features that can help recover to a known good state, such as creating templates, which is essentially a copy of a VM at a snapshot in time which is used to create new copies of VMs. If an SRA template is created prior to installation of SRA software, users can drastically reduce their time to normal operation instead of having to deploy from scratch.

Your Browser is Out of Date

HA and DR with Claroty