Introduction

In today’s world, the deployment of security cameras is a common practice.  In some public facilities like airports, travelers can be in view of a security camera 100% of the time. The days of security guards watching banks of video panels being fed from hundreds of security cameras are quickly being replaced by computer vision systems powered by artificial intelligence (AI).  Today’s advanced analytics can be performed on many camera streams in real-time without a human in the loop. These systems enhance not only personal safety but also provide other benefits, including better passenger experience and enhanced shopping experiences.

Modern IP cameras are complex devices.  In addition to recording video streams at increasingly higher resolutions (4k is now common), they can also encode and send those streams over traditional internet protocol IP to downstream systems for additional analytic processing and eventually archiving.  Some cameras on the market today have enough onboard computing power and storage to evaluate AI models and perform analytics right on the camera.

The Problem

The development of IP-connected cameras provided great flexibility in deployment by eliminating the need for specialized cables.  IP cameras are so easy to plug into existing IT infrastructure that almost anyone can do it.  However, since most camera vendors use a modified version of an open-source Linux operating system, IT and security professionals realize there are hundreds or thousands of customized Linux servers mounted on walls and ceilings all over their facilities. Whether you are responsible for <10 cameras at a small retail outlet or >5000 at an airport facility, the question remains “How much exposure do all those cameras pose from cyber-attacks?”

The Research

To understand the potential risk posed by IP cameras, we assembled a lab environment with multiple camera models from different vendors. Some cameras were thought to be up to date with the latest firmware, and some were not. 

Working in collaboration with the Secureworks team and their suite of vulnerability and threat management tools, we assessed a strategy for detecting IP camera vulnerabilities   Our first choice was to implement their Secureworks Taegis™ VDR vulnerability scanning software to scan our lab IP network to discover any camera vulnerabilities. VDR provides a risk-based approach to managing vulnerabilities driven by automated & intelligent machine learning.

We planned to discover the cameras with older firmware and document their vulnerabilities.  Then we would have the engineers upgrade all firmware and software to the latest patches available and rescan to see if all the vulnerabilities were resolved.

Findings

Once the SecureWorks Edge agent was set up in the lab, we could easily add all the IP ranges that might be connected to our cameras. All the cameras on those networks were identified by SecureWorks VDR and automatically added to the VDR AWS cloud-based reporting console. 

Discovering Camera Vulnerabilities

The results of the scans were surprising.  Almost all discovered cameras had some Critical issues identified by the VDR scanning.  In one case, even after a camera was upgraded to the latest firmware available from the vendor, VDR found Critical software and configuration vulnerabilities shown below: 

One of the remaining critical issues was the result of an insecure FTP username/password that was not changed from the vendor’s default settings before the camera was put into service. These types of procedural lapses should not happen, but inadvertently they are bound to.  The password hardening mistake was easily caught by a VDR scan so that another common cybersecurity risk could be dealt with. This is an example of an issue not related to firmware but a combination of the need for vendors not to ship with a well-known FTP login and the responsibility of users to not forget to harden the login.

Another example of the types of Critical issues you can expect when dealing with IP cameras relates to discovering an outdated library dependency found on the camera. The library is required by the vendor software but was not updated when the latest camera firmware patches were applied.

Camera Administration Consoles

The VDR tool will also detect if a camera is exposing any HTTP sites/services and look for vulnerabilities there. Most IP cameras ship with an embedded HTTP server so administrators can access the cameras' functionality and perform maintenance.  Again, considering the number of deployed cameras, this represents a huge number of websites that may be susceptible to hacking.  Our testing found some examples of the type of issues that a camera’s web applications can expose:

The scan of this device found an older version of Apache webserver software and outdated SSL libraries in use for this cameras website and should be considered a critical vulnerability. 

Conclusions

In this article, we have tried to raise awareness of the significant Cyber Security risk that IP cameras pose to organizations, both large and small. Providing effective video recording and analysis capabilities is much more than simply mounting cameras on the wall and walking away. IT and security professionals must ask, “Who’s watching our IP cameras?  Each camera should be continuously patched to the latest version of firmware and software - and scanned with a tool like SecureWorks VDR. If vulnerabilities still exist after scanning and patching, it is critical to engage with your camera vendor to remediate the issues that may adversely impact your organization if neglected. Someone will be watching your IP cameras; let’s ensure they don’t conflict with your best interests.

Dell Technologies is at the forefront of delivering enterprise-class computer vision solutions.  Our extensive partner network and key industry stakeholders have allowed us to develop an award-winning process that takes customers from ideation to full-scale implementation faster and with less risk.  Our outcomes-based process for computer vision delivers:

• Increased operational efficiencies: Leverage all the data you’re capturing to deliver high-quality services and improve resource allocation.
• Optimized safety and security: Provide a safer, more real-time aware environment
• Enhanced experience: Provide a more positive, personalized, and engaging experience for customers and employees.
• Improved sustainability: Measure and lower your environmental impact.
• New revenue opportunities: Unlock more monetization opportunities from your data with more actionable insights

Where to go next...

Beyond the platform - How Dell Technologies is leading the industry with an outcomes-based process for computer vision

Dell Technologies Workload Solutions for Computer Vision

Secureworks

Virtualized Computer Vision for Smart Transportation with Genetec

Virtualized Computer Vision for Smart Transportation with Milestone

Introduction

Ansible is an astoundingly useful and convenient DevOps tool that helps streamline the process of managing remote hosts. However, it does have a learning curve and requires at least some technical knowledge to use efficiently given that it is a CLI (Command Line Interface) tool. Fortunately, there are several modern, feature-complete User Interface options for managing and running an Ansible instance on a remote server that can be controlled directly from a web browser. This, along with their open-source nature, makes the process of using Ansible and running playbooks much more intuitive and convenient; even for experienced team members that are familiar with using Ansible from the command line. This blog describes our evaluation of the most relevant aspects of each UI, including their features and accessibility.

AWX: The premiere open-source UI

Figure 1: AWX Dashboard

AWX is the most well-known and feature complete UI for Ansible. It provides a sleek and intuitive interface that neatly organizes the configuration options by category and allows for the use of Role-Based Access Control. This gives users the option to regulate who can see or modify certain settings and files. Key amongst these are job templates, which serve as a set of parameters and definitions that allow the same job to be executed many times. Additionally, the built-in dashboard provides a visually pleasing yet extensive overview of past jobs as well as their outcome, along with other relevant information about the AWX configuration. Last, but certainly not least, AWX allows for secure and encrypted storage of credentials and vault passwords, allowing them to be shared between team members safely and effortlessly.

Ansible Semaphore: Easy to start, easy to use

Figure 2: Ansible Semaphore UI

Compared to AWX, Ansible Semaphore is more simplistic in every sense, with its straightforward installation process and streamlined UI coming at the cost of features that the other UI options we evaluated have. For example, Ansible Semaphore does not support high availability, meaning that it cannot automatically recover from component failure and can result in longer downtimes. However, this tool can easily be setup to pull ansible playbooks from GitHub, store credentials for GitHub/your machine, and run playbooks through a simple task template. Inside of a template you can specify hosts to run on (inventory), variables (environment), and extra command line arguments. That being said, Sempahore’s best feature is quite possibly its dashboard. Designed on Google’s Material UI, Ansible Semaphore’s dashboard is very easy to navigate and has a simplistic look to showcase the critical information for each run.

Rundeck Community Edition: More than just Ansible

Figure 3: Rundeck UI

Rundeck Community Edition gives users the basic functionality that is needed to execute playbooks inside a UI and, just like Ansible Semaphore, it is very easy to install and get up and running. Rundeck is a general automation tool so you can do more than just execute ansible playbooks, but the dashboard is not quite as easy to use as Ansible Semaphore and it is not as visually appealing.  Some features of Rundeck CE include creating multi-step jobs, running shell commands, and executing local commands.  While the community edition boasts many features beyond just running Ansible playbooks, the most desired features such as high availability and certified enterprise plugins are reserved for the enterprise or cloud editions.

Custom Coding the UI: Full Control

If none of the aforementioned solutions seem ideal to you, or they do not appropriately address your requirements, you always have the option of designing and creating your own UI solution. Doing so will grant you an appropriately scaled solution that meets all of your needs and requirements while also allowing you to express your creativity and originality. For example, if you want to offer some niche features to the UI like a “revert operation” that will undo a previously run playbook or displaying the completion percentage of a job that is in progress, then a custom UI could be your best option. However, this approach requires an immense amount of effort compared to the other options we discussed to develop and properly maintain a secure solution. One approach we investigated was to build a robust REST API running on an ansible-capable remote host for the backend services and a web frontend running on the same host. The two components of the application can then use HTTP requests to communicate and run and/or modify the pertinent files locally on the server.

Conclusion

Figure 4: UI Comparison Summary

Leveraging an Ansible UI is a great way to easily extend the functionalities and capabilities of Ansible to non-CLI experts by making server management and automation more accessible. Namely, it provides a less error prone execution and a more consumable way of seeing job progress and output for all users. Every option described above has its pros and cons, and it’s important to factor in the setup/installation process of each option. Incidentally, despite AWX being our top choice due to its maturity and feature set, its installation process is notoriously difficult. Because of this, our team decided to make our own guide describing what made the installation work for us. If you are interested in learning more about AWX’s setup process, feel free to check out the installation tutorial blog created by our team by clicking on the link right here!

Introduction

As the use of virtualization (VMs and containers) expands rapidly in many organizations, automation is needed for virtual server management to address the tedious and repetitive tasks. Ansible is a powerful tool for automation, deployment, and configuration management that has historically required living on the command line interface (CLI). The open-source version of Ansible Tower is AWX - a web-based user interface (UI) for Ansible. When we wanted to explore how AWX works we quickly realized that the existing AWX installation guides need an overwhelming amount of trial and error to make work. This blog presents how to execute a reliable installation process and also explains the automation of the process that reduced our installation to just running a single command. Our comparison and selection of an Ansible UI from a list of 4 options is documented in this blog if you want to learn about that effort.

AWX overview

AWX is a UI solution that sits on top of the Ansible CLI supporting functionality such as visualization of host management and running job status including Ansible playbooks, specification of job parameters, and login authentication. Since AWX is an open-source version of an enterprise product, it has very limited official documentation. During our testing we encountered issues such as insufficient dependency specs, failure to pull Docker images, and inability to visualize our AWX instance. There are many different unofficial guides, but unfortunately, very few of them work reliably without the need for debugging. This blog documents a simple and reliable method for installing AWX.

Prerequisites

Our goal was to deploy AWX on a management system that can connect to a workload environment for VM automation. The only prerequisites you need to get started is to have Ansible installed in the management system and to have your Docker Hub login credentials available. It is crucial to store your Docker Hub username and password in a file named secret.enc under the vars folder of the playbook in following format:

Then, you should encrypt the file using a command similar to the one below using Ansible Vault.

Testing system details

Table 1: System Details

Components to be deployed by the Installation Playbook

Table 2: Components to be Deployed

Installation Process

Figure 1: High-level Overview of the Components in the Installation

The goal is to have a running instance of AWX accessible with a browser. With this design, the user only needs one command to run the playbook that installs AWX. This command asks for the sudo permission so the playbook can use elevated privileges whenever necessary. A vault password is also requested to use the encrypted Docker Hub credentials described above for a successful login into Docker. Minikube and Docker are automatically installed by the installation playbook. Minikube is the backbone of this installation process and provides the resources that the AWX instance is installed on. Docker ensures that the Minikube pods are ready for initializing AWX. 

Here is an outline of the background process for the Ansible playbook:

1. The playbook installs the necessary prerequisites. 

2. The playbook logs into and sets up Docker.

3. A Minikube instance is run with specified configurations. 

Figure 2: Creation of Minikube Instance

4. An image pull secret is created and patched to the service account based on the Docker Hub credentials for successful image pulls.

Figure 3: Creation of an Image Pull Secret

5. AWX operator is deployed and it runs as a pod[PD1] .

Figure 4: Deployment of AWX Operator

Figure 5: Running AWX Operator Pod[SM2] 

6. AWX instance is deployed with 4 pods for the instance and 1 pod for postgres.

Figure 6: Deployment of AWX Instance

Figure 7: Deployment File (ansible-awx.yml)

Figure 8: Running Pods for AWX Instance and Postgres[SM3] 

7. Expose the port for the AWX instance through port forwarding and display the IP address and login information for accessing the instance.

After running the Ansible install AWX playbook, the login information including username, password, and IP address with port for the AWX instance will be displayed as a part of the detailed output. 

Figure 9: An Example of the Playbook Output with Login Information

Then, you can access the dashboard for AWX using your host’s IP address and port 32483 with login credentials provided from the above output. 

Figure 10: AWX Dashboard After a Successful Installation and Login

Common errors and solutions

A few errors that you may encounter during the installation process:

• ImagePullBackOff: Kubernetes fails to pull container images from Docker Hub. It is important to make sure that you are logged into Docker Hub successfully using Ansible Vault. You can also login manually using docker login, but it is less secure.
• Certificate and connection related errors: Ensure that VM resources are sufficient for running Minikube with predefined specifications. If multiple users are working on the same server with several Kubernetes clusters, such errors may also occur due to resource limitations.

Conclusion

This blog introduces a quicker and more convenient way to reliably install AWX. With a simple goal of having a running AWX instance on a server, this blog demonstrates a straightforward solution to achieve that goal while many other existing guides need much more customizations and configurations for the successful execution of an AWX deployment.