Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
Design Guide—Virtualized Computer Vision for Smart Transportation with Milestone
Introduction Architecture concepts and requirements Solution Architecture Validation
Validation overview
Performance validation
Performance results and findings
Performance summary
High availability validation
High availability results and findings
High availability summary
Sizing the solution Backup and restore operations Monitoring the solution Troubleshooting recommendations Summary and conclusions References

Performance results and findings

Performance results and findings


  • Milestone XProtect performance tests

    Single recorder performance test

    VM specification

    The VM specification for the single recorder test included:

    • 6 vCPU
    • 12 GB Memory
    • 1 - 200 GB local vSAN disk (OS)
    • 1 - 2 TB local vSAN disk (Tier 1 Storage)
    • 1 - 10 Gb Network card

    Test results

    The performance data collected for a single Recorder processing a load of 70 camera streams is below.

    Figure 10. CPU results
    Figure 11. Memory results
    Figure 12. Writes results

    Findings

    • The percentage memory utilization for a recorder allocated 12GB and processing 70 camera streams was higher than expected with peaks around 90%.
    • The maximum Tier 1 vSAN writes were ~26.5 MB/s. This is consistent with the best practices for Milestone sizing for a single VM where disk writes should be between 24 to 30 MB/s.

    Overall

    • The XProtect recorders performed well but we recommend that memory should be increased to 16GB when sizing a recorder.

    Aggregate Recorder test for a VxRail node

    Test specification

    • Validate that 3 Milestone Recorders can run on a single VxRail host without exceeding the 96 MB/s thresholds set in previous Milestone testing by Dell.

    We ran the following recorders on a single node of the VxRail cluster:

    • Milestone Recorder 1
    • Milestone Recorder 2
    • Milestone Recorder 3

    Test results

    The disk write bandwidth data for a single VxRail host with 3 recorders streaming a total of 210 cameras is:

    Figure 13. Single host writes

    Findings

    • The amount of vSAN writes was < 80 MB/s. This is below the 96 MB/s threshold established by Milestone Systems and Dell. and therefore, is not an issue.

    Overall

    • Hosting a maximum of 3 recorders per VxRail host is our recommendation for customer environments.

    BriefCam performance tests

    Our performance testing in support of the recommendation in this Design Guide was focused on the BriefCam RESPOND real-time analytics feature (Alert Processing Service). The integration between the RESPOND functionality and the Milestone Systems VMS uses a plug-in component on the BriefCam Alert Process server and the Real-Time Streaming Protocol (RTSP) interface supplied by Milestone Systems to access video streams with the lowest latency possible.

    A real-time task processing request goes through several preprocessing steps before being able to generate alerts. The BriefCam RESPOND user interface allows operators to monitor the status of real-time tasks to track how many are queued, recovering (between the queued and processing state), and actively processing alerts. We tested performance by adding requests for new real-time alert processing tasks until we began to detect that the number of queued requests was increasing but were not able to change state and enter the processing stage.

    Test specification

    • Determine the maximum number of real-time alert processing tasks supported by a single VM with a full Nvidia A40 vGPU profile.
    • Validate that the load balancing service in BriefCam will evenly distribute real-time alert processing tasks across a 4-node active/active cluster of BriefCam Alert Processing servers that are hosted on virtual machines each with a full Nvidia A40 vGPU.

    Single VM test results

    We tested both the maximum numbers of a single-use case workload stream (face recognition or person detection in a restricted zone) plus various mixtures of the workload specifications. In all tests, the results were consistent. In this test, cameras were added until we began to see that new streams were remaining in a queued state.

    The total CPU and GPU Utilization in the build-up to processing for 50 Face Recognition Cameras are as follows:

    Figure 14.

    Detailed GPU utilization metrics are shown below:

    Figure 15.

    The Green line in the above graph shows that the BriefCam Alert Processing Service does not perform any re-encoding of the video format that is streamed from the Milestone Systems VMS. The BriefCam Milestone Systems plug-in installed on the Processing Server knows the format of video produced by the RTSP service (H264) which is also known to the processing engine. This VMS-specific product integration reduces GPU workload and allows more resources to be used for alert processing.

    Multi-server load balanced Real-time Alerting

    The BriefCam Alert Processing Service can function as a scale-out active/active cluster by installing and configuring multiple instances of the service that run on different servers. When multiple service instances are available, a cluster mesh service detects how many service instances are running and on what machines. When new real-time alert processing tasks are requested, the request is queued using the PostgreSQL database. The service mesh then assigns tasks from the queue to available servers that are running the alert processing service.

    We configured 2 VMs running the BriefCam server components to process real-time alerts. Our goal was to validate that the total number of streams processed would be allocated evenly among the cluster nodes. The maximum number of streams processed during the cluster test was 80, There was no request queuing at that level of workload consistent with the finding that the single server maximum is 50 streams shown above.

    We added new camera streams to the workload in groups of 10 with a several-minute pause in between to verify that all streams were in processing mode before proceeding. The table below shows the comparison of resource utilization for the two load-balanced Alert Processing servers.

    Table 9. BC63-SVR-3
    CPU % utilization GPU % utilization GPU video decoding % utilization
    Min Avg Max Min Avg Max Min Avg Max
    BC63-SVR-3 66 75 88 75 81 84 13 37 71
    BC63-SVR-4 62 75 88 60 71 78 18 45 80

    Findings

    • The GPU is the most heavily used resource for Alert Processing servers.
    • The BriefCam active/active load-balancing is deficient in spreading the workload across the servers in a cluster.

    Ipsotek performance tests

    The Ipsotek alert processing testing contained a mixture of Face Recognition and Restricted Zone camera simulation scenarios. The Ipsotek platform supports multiple tracking modes that can be configured for a camera:

    Video Analytics
    Traditional pixel-based analytics.
    AI - Normal
    AI based tracking of full people. This was used for Restricted Zone tests.
    AI - Crowded
    AI based tracking for crowded scenes where head and shoulders are tracked.
    Face Detection
    Focus on detecting faces. This was used for Face Recognition tests.

    A "Face in Watchlist" rule was created to generate a report for people identified from a camera stream and on a watchlist. We used this watch list scenario for all Face Detection camera processing. In addition, an "In Zone" rule was created to detect people entering a restricted zone.

    Ipsotek uses the Nvidia GPU encoder to reencode the video received from different cameras and generate synchronized video with I-Frames produced every second at a fixed bit rate. This allows Ipsotek to:

    • Estimate storage requirements based on the retention policy.
    • Access video and images on disk to an exact second without needing to seek in mp4 segments.
    • Stream video over HLS protocol with a minimum buffering time of 2 s to be viewed on a web interface.
    • Ensure that we get a low latency user experience when an operator is requesting to random access images and video on disk.

    Test specifications

    • Determine the maximum number of real-time alert processing tasks that can be supported by a single VM with a full Nvidia A40 vGPU.
    • Determine the maximum number of real-time alert processing tasks that can be supported by a 3-node active/active cluster of Ipsotek Processing servers hosted on virtual machines each with a full Nvidia A40 vGPU.
    • Use a combination of 45 face recognition and 45 restricted area/person detection workloads and collect performance metrics. In particular, focus on GPU Encoder and Decoder usage.

    Single VM test results

    We tested both the maximum numbers of a single-use case workload stream (face recognition vs person detection in a restricted zone) plus various mixtures of the workload specifications. In all tests, the results were consistent. Cameras were added until the max of 30 cameras were active and no more could be processed.

    The total CPU and GPU Utilization in the build-up to processing for 30 Face Recognition Cameras are as follows:

    Figure 16. CPU and GPU utilization for face recognition cameras

    The breakdown of GPU utilization is as follows:

    Figure 17. GPU utilization

    Findings

    • The max number of Face Recognition or Restricted Zone cameras is consistent at 30 cameras per Ipsotek processing server.
    • The encoder component of the A40 GPU was exhausted during this test.

    VM cluster test results

    The Ipsotek platform does not currently support vGPUs, so all GPU resources must be assigned to the VM as a passthrough device. This means that it is not possible to migrate VMs as was shown in High availability validation. To build a viable Ipsotek cluster in a virtualized environment, it is necessary to provision all Ipsotek Processing nodes at system setup time. The goal is then to load the system so that sufficient capacity exists across the cluster to ensure that during a failover the cameras can migrate to an available node and continue processing.

    In this testing, the approach was to load four of the five nodes in the cluster to their maximum. This ensures that there is sufficient capacity to disperse the workload across the cluster in a failure scenario. A total of 60 Face Recognition Cameras and 60 Restricted zone cameras were enabled.

    The maximum count of active cameras across three processing nodes was 120.

    Findings

    • The maximum number of real-time alert processing tasks was 30 for a single node. We also validated that this result was consistent across the three VxRail nodes that were allocated for CV application hosting.
    • The CPU was ~50% when all cameras were processing so 8 CPUs per Ipsotek processing node is sufficient.
    • When 30 cameras were enabled on a single node, the GPU utilization was at ~50% but the GPU Hardware Encoder approached 100% and no further-incoming video streams were accepted showing that video encoding is the most resource-intensive component of the alter processing workload.

    Full system performance tests

    The purpose of this test was to validate if a combination of three applications from three different vendors can share a common platform using VxRail and VMware without introducing processing delays. We have the individual application results from above for our baseline. Our performance data for this test was collected while the following workloads were processed in parallel.

    • A total of 840 cameras are being recorded in Milestone XProtect.

    • 200 BriefCam real-time alert processing tasks analyzing camera streams from the Milestone Real-Time Streaming Protocol (RTSP).

    • 120 Ipsotek Face Recognition and Restricted Zone tasks analyzing camera streams from Milestone XProtect using RTSP.

    Overall cluster performance results

    The following chart shows a snapshot of the CPU utilization of the DRS cluster at a point in time: during the full system testing.

    Figure 18. CPU utilization of the DRS cluster

    Findings

    • The workload is distributed across all 5 nodes in the cluster using DRS.
    • No issues when streaming 840 cameras.
    • Sufficient capacity exists that allows one node be taken offline for maintenance.
    • Sufficient capacity exists to handle any spike in the load.

    Overall

    • This cluster is able to handle the workload without issue.

    VM level view

    When the full system load test with 840 cameras was running, the top 10 most active VMs were identified by CPU MHz consumption.

    The following charts shows those top 10 busiest VMs in the cluster:

    Figure 19. The cluster's top 10 busiest VMs

    Findings

    • The Milestone XProtect recorders are consuming the most CPU across the system.

    Overall

    • It is expected that the Milestone Recorders are utilizing about 10 MHz per VM. The VMs are allocated 6 vCPUs and are running at less than 75%.