Milestone Systems develops products for large-scale facilities with high-security requirements. The XProtect family of products ensures end-to-end protection of video integrity while maximizing hardware performance.

The XProtect VMS products provide video management capability for environments covering a wide range of use cases and scales. Milestone Systems offer four versions of XProtect:

• XProtect Corporate
• XProtect Expert
• XProtect Professional+
• XProtect Express+

This suite of versions can support applications that range from protecting individual stores from vandalism to managing a multi-site, high-security facility. All solutions offer centralized management for all devices, servers, and users, including a flexible rules processing system driven by schedules and events. Our development work for this Design Guide was performed using the XProtect Corporate version.

The remainder of this document is specific to the XProtect VMS 2022 R1 Corporate version with significant changes to the system architecture and many new features not available in earlier versions. Implementations of the XProtect Corporate VMS system typically consist of the following main components:

• The central management servers
• XProtect Download Manager
• One or more recording servers
• One or more installations of XProtect Management Client
• One or more installations of XProtect Smart Client
• One or more users of XProtect Web Client and XProtect Mobile client (optional)

Figure 2.   Milestone XProtect Corporate architecture

Scale-out architecture

To enable scaling up to thousands of cameras across multiple sites, the XProtect system consists of several components that handle specific tasks. For systems with more than 100 cameras, Milestone recommends that you use dedicated servers for all or some of the components.

Figure 3.   Milestone scale-out architecture

The full set of components shown here need not be available in all installations initially. Components such as failover recording servers or mobile servers can be added if the functionality they offer is needed later for hosting and providing access to both XProtect Web Client and XProtect Mobile.

Depending on hardware and configuration, smaller systems with 50 to 100 cameras can run on a single server.

Configuration best practices

The following guidelines are based on previous internal testing and reports from customer field experience. None of these guidelines were exceeded during testing.

CPU

Do not assign more vCPUs to your VMs than the number of physical cores on the host machine.

Memory

The total amount of memory allocated to the VMs and the hypervisor should not exceed the total amount of physical memory available from the host.

Storage

• Install Microsoft Windows and Microsoft SQL databases on vSAN devices according to the space and performance needs suggested by each application vendor.
• For video recorder virtual machines, do not store archived video on the OS drive.
• Make the OS partition at least 120 GB.
• For Tier1 storage, make use of dedicated drive from vSAN storage with Raid5 Thick Storage policy.
• For Tier 2 archived video, configure Dell APEX Data Storage service file for video storage.

Networking

• When provisioning multiple archiving VMs on a host, do not exceed data transmission rates of 96 MB/s per node:
• In-Guest iSCSI networks with other configurations might result in degraded performance.
• Isolate video traffic on a different VLAN from vSAN storage traffic.
• At least 10 GbE network cards are required (40 GbE preferred) for shared traffic links (management, video, and storage) configured with a Virtual Switch.
• Alternatively, a dedicated 1 GbE network card per VM for video traffic can be sufficient in some cases.
• Network configurations that can result in multicast traffic being sent to all hosted VMs simultaneously should be avoided to limit the potential for negative performance impacts.

High availability

High Availability is an important consideration for security systems deployment to avoid loss of camera stream data during an unplanned outage. While no system can achieve 100% availability, deploy the following components with high availability considerations.

The following VMS components can run on a single virtual machine for small environments.

• Management server
• Event server
• Log server
• SQL Server

However, Milestone recommends hosting SQL Server on a dedicated server for applications with many devices and/or many event transactions. For dedicated SQL Server installations, the design must then consider fault tolerance for both the database and the three remaining services (management, events, and logging). We tested SQL Server installed with support for Always On Availability Groups (AAG) following the best practices documented by Microsoft. We dedicated two virtual machines for hosting two stand-alone (non-FCI) instances of SQL Server 2019. We created a virtual network name and virtual IP in Active Directory to support our AAG configuration. We followed this AAG configuration guide from Microsoft without modifications.

Management server failover

The management server can be installed on multiple servers within a cluster of servers. This ensures that the system has very little downtime. If a server in the cluster fails, another server in the cluster automatically takes over the failed server's job running the management server. The automatic process of switching over the management server services to run on another server in the cluster takes up to 30 seconds. For more information about high availability options for the Management Server, see the Milestone Systems XProtect VMS documentation for clustering multiple management servers.

Recording server failover

A failover recording server has two services installed:

The Failover Server service handles the processes of taking over from the recording server. This service is always running, and constantly checks the state of relevant recording servers.

The Failover Recording Server service enables the failover recording server to act as a recording server.

• The failover server service checks the state of relevant recording servers as defined by settings in the FailoverServerConfig.xml file. The typical polling interval is every 0.5 seconds. If a recording server does not reply within two seconds and other conditions such as a successful ping of the management server are met, then the recording server is considered unavailable and the failover recording server takes over.
• Any failover solution will not provide complete redundancy but will provide a reliable way of minimizing downtime. When a failed primary recording server becomes available again, it will merge any recordings for recorders stored on the failover recording server back to its own storage. A loss of recordings during this process is very unlikely.
• Recordings stored by the failover recording server are automatically merged into the primary recording server's databases. The time it takes to merge depends on the number of recordings, network capacity, and more. During the merging process, you cannot browse recordings from the period during which the failover recording server took over.
• In the event of a recording server failure, users should hardly notice that a failover recording server has taken over. A short interruption in service occurs, usually only for a few seconds, when the failover recording server takes over.
• During the service transfer exchange, users cannot access the video from the failed recording server. Users will be able to resume viewing live video as soon as the failover recording server has taken over.
• The recordings stored on the failover recording server can be played back for the period after the failover recording server took over. Clients cannot play back older recordings stored only on the failed recording server until that recording server is functional again and has taken over from the failover recording server.

Cold standby failover recording servers

In a cold standby failover recording server setup, you group multiple failover recording servers in a failover group. A cold standby failover group must contain at least one failover recording server.

The entire failover group is dedicated to taking over from any of several preselected recording servers if one of these becomes unavailable. You can create as many failover groups as needed to achieve the required level of outage protection. In a cold standby setup, the Failover Recording Server service is only started when the cold standby failover recording server takes over from the recording server. The time required to start the service depends on several factors including CPU characteristics, disk configuration, network performance, the number of camera devices configured, security settings, and more.

When an active recording server fails, one of the available servers in the assigned primary failover group will take over. Grouping failover recording servers can provide a pool of servers in reserve that can protect one or more active recording servers. If the configured primary failover group contains more than one failover recording server at the time of failure, then one will be assigned to take over. You can specify a secondary failover server group that can supply servers to take over for a failed active recording server if all the recording servers in the primary group are in use. A failover recording server can only be a member of one failover group.

Recording servers in a failover group are ordered in a sequence. The sequence determines how the failover recording servers will take over from a failed active recording server. By default, the sequence reflects the order in which you have incorporated the failover recording servers in the failover group: the first in is the first in the sequence. You can change this order if needed.

If a failover recording server in a cold standby failover group must take over from another recording server during the merging process, it postpones the merging process with the first failed recording server (Server A) and takes over for the newly failed recording server (Server B). When recording server B becomes available again, the failover recording server takes up the merging process and allows both recording server A and recording server B to merge back recordings simultaneously.

Hot standby failover recording servers

Hot standby failover recording server setup requires a dedicated failover recording server for each protected recording server. This one-to-one mapping allows the system to quickly transition a failover recording server from "standby" mode by synchronizing the correct or current configuration of the recording server to which it is dedicated.

A hot standby failover recording server can take over much faster than a cold standby failover recording server. In a hot standby setup, the Failover Recording Server service is always running, allowing the hot standby server to take over faster than the cold standby failover recording server. Hot standby servers are assigned in a one-to-one pairing with recording servers. A failover group cannot also protect a primary recording server protected with a hot standby. You cannot assign failover servers that are already assigned to a failover group as hot standby recording servers.

A hot standby failover recording server must only start its cameras on failover to deliver feeds because the Failover Recording Server service is always running. During the startup period, you cannot store recordings or view live video from affected cameras.

If a recording server with a hot standby server fails again during the merge back process, the hot standby server will take over again while retaining the recordings from the previous failed period. The hot standby recording server keeps all recordings until they are merged back to the primary recorder or until the failover recording server runs out of disk space.

A hot standby server cannot be configured with failover protection from either a standby failover group or another hot standby server.

VM designs

This section details the VMs used to host XProtect services for this testing:

High level architecture

For our design, the Milestone XProtect VMs are distributed across the Dell APEX Private Cloud cluster by DRS based on affinity rules. The mapping process can be visualized as follows:

Figure 4.   Milestone VMs across the cluster