Cloud-Native Workloads: Object Detection in Video Streams
Wed, 02 Mar 2022 22:18:43 -0000
|Read Time: 0 minutes
See containers and Kubernetes in action with a streaming video analysis Advanced Driver Assistance System on APEX Cloud Services.
Initially published on November 11, 2021 at https://www.dell.com/en-us/blog/cloud-native-workloads-object-detection-in-video-streams/.
A demo may be the best way to make a concept real in the IT world. This blog describes one of a series of recorded demonstrations illustrating the use of VMware Tanzu on APEX Cloud Services as the platform for open-source based cloud-native applications leveraging containers with Kubernetes orchestration.
This week we’re showcasing an object detection application for an Advanced Driver-Assistance System (ADAS) monitoring road traffic in video sources that leverages several open-source projects to analyze streaming data using an artificial intelligence and machine learning (AI/ML) algorithm.
The base platform is VMware Tanzu running on APEX Private Cloud Service. APEX Private Cloud simplifies VMware cloud adoption as a platform for application modernization. It is based on Dell EMC VxRail with VMware vSphere Enterprise Plus and vSAN Enterprise available as a 1- or 3-year subscription with hardware, software and services (deployment, rack integration, support and asset recovery) components included in a single monthly price. VMware Tanzu Basic Edition was added post-deployment to create the Container-as-a-Service (CaaS) platform with Kubernetes running integrated in the vSphere hypervisor.
Object detection in video sources requires managing streaming data for analysis, both real time and storing that data for later analysis. This demo includes the newly announced Dell EMC ObjectScale object storage platform which was designed for Kubernetes as well as the innovative Dell EMC Streaming Data Platform for ingesting, storing and analyzing continuously streaming data in real time.
The image recognition application leverages several open-source components:
- Pravega software that implements a storage abstraction called a “stream” which is at the heart of the Streaming Data Platform.
- Apache Flink real time analytics engine for the object detection computations.
- Tensor Flow for the object detection model.
- Jupyter as the development environment for data flow and visualization.
The demo shows these components running in Tanzu Kubernetes Grid clusters to host the components of the object detection demo. It looks from the perspective of a data scientist who configures the projects and data flows in the Streaming Data Platform. Also, the Jupyter notebooks are configured to push the data into the Pravega stream and display the video with the object detection.
You can view the demo here.
Demos like these are a great way to see how the Dell Technologies components can be combined to create a modern application environment. Please view this demo and provide us some feedback on other demos you’d like to see in the future.
You can find more information on Dell Technologies solutions with VMware Tanzu here.
Author: Bob Ganley
Related Blog Posts
How Dell Technologies Simplifies Data Protection
Thu, 04 Aug 2022 19:31:34 -0000
|Read Time: 0 minutes
As we are moving to the era of digitalization, more data is vulnerable to corruption, compromise, and loss. As a result, organizations are focusing on data protection strategies to improve security and data integrity. The key principle of data protection is to safeguard and make data available under all circumstances. Dell Technologies offers a variety of data protection products to make sure that data is secure and readily available. In this blog I’ll be covering a few of the outstanding data protection products that Dell provides.
Data protection and backup solutions
Cloud
As organizations are redistributing and extending their data to the cloud or to multiple clouds, protecting that data becomes critical. By implementing a multi-cloud data protection solution, users and developers can avoid cloud vendor lock-in by keeping data independent of the cloud, to reduce worry about high egress charges, migration risk, or time required to move data. Multi-Cloud Data Services for Dell PowerProtect, enabled by Faction, supports a data centric approach for users, allowing them to separate their data and cloud management strategies across public cloud providers. Multi-Cloud Data Services for Dell PowerProtect is a fully managed service that provides a single target for backup and recovery for AWS, Microsoft Azure, Google Cloud, and Oracle Cloud.
VMware
Data protection solutions from Dell Technologies gives customers the power to confidently plan their modern data protection strategies through constant innovation, agile engineering, and tight integration with VMware - all at a lower cost of ownership (TCO). Dell offers protection solutions for VMs, applications, and Kubernetes for a single platform for containerized products and workloads. The solution simplifies deployment and management of VMware workloads on-premises or in one or more public clouds.
Dell PowerProtect Cyber Recovery
PowerProtect Cyber Recovery provides proven, modern, and intelligent protection to isolate critical data, identify suspicious activity, and accelerate data recovery, allowing you to resume normal business operations with confidence in case of a cyberattack. Critical data is stored immutably in a hardened vault, enabling recovery with assured data availability, integrity, and confidentiality. CyberSense is the analytics engine that is integrated with Cyber Recovery to help find data corruption when an attack penetrates the data center. CyberSense enables confident recovery of good data and offers insight into attack vectors from within the Cyber Recovery vault.
Data protection and backup appliances
Dell PowerProtect DD series appliances
DD series is our purpose built backup target appliance. The DD operating system (DDOS) provides inline deduplication for data protection and disaster recovery (DR) in the environment. DDOS is the intelligence that powers DD series and PowerProtect systems. It provides the agility, security, and reliability that enables the platform to deliver scalable, high-speed and cloud-enabled protection storage for backup and archive. DD series appliances vary in storage capacity and data throughput. PowerProtect DD Virtual Edition (DDVE) is the software-defined version of the DD series: a virtual deduplication appliance that provides data protection for small-to-medium sized business, enterprise and service provider environments both on-prem and in-cloud.
Dell PowerProtect DP series appliances
DP series is an integrated appliance solution thatprovides complete backup, replication, recovery, deduplication, instant access and restore, search & analytics, seamless VMware integration — plus, cloud readiness with disaster recovery (DR) and long-term retention (LTR) to the cloud — all in a single appliance. With the DP series, customers reduce their time-to-protect and become more agile with a reliable solution that’s fast, protected more efficiently, and can be recovered quickly.
Data protection and backup software
Dell Data Protection Suite
Dell PowerProtect Data Manager
PowerProtect Data Manager software protects data and delivers governance control for modern cloud workloads across your evolving physical, virtual, and cloud environments. Address ever-changing growth and IT complexity by leveraging Dell software defined data protection platforms. PowerProtect Data Manager delivers next generation data protection that enables faster IT transformation, while giving you the assurance that you can easily safeguard and quickly unlock your data’s value. This software-defined solution includes self-service protection, centralized governance, and SaaS-based reporting and monitoring.
Dell Avamar
Avamar is backup and recovery software that provides integrated data deduplication technology. Avamar solves the challenges that are associated with traditional data protection, enabling fast and reliable backup and recovery for remote offices, VMware environments, DD series and Cloud Tier, and data center LANs. Unlike traditional solutions, Avamar reduces the size of backup data at the source—before it transfers across the network and stores to disk. As a result, Avamar delivers fast, efficient daily full backups despite slow or congested infrastructure, and data encryption for added security.
Avamar is available as a virtual edition or as a component of Data Protection Suite, which offers you a complete suite of data protection software options.
Dell NetWorker
NetWorker software provides unified backup and recovery of enterprise applications and databases. With centralized administration, NetWorker helps you take advantage of the data protection that fits your needs best: deduplication, backup to disk and tape, snapshots, replication, and NAS. NetWorker protects both physical and virtual environments including VMware and Microsoft Hyper-V. It provides the same enterprise-level experience, whether you are protecting data and applications within the data center or in the public cloud.
NetWorker is available as virtual edition (NVE) or as a component of the Data Protection Suite, which offers you a complete suite of data protection software options.
Dell Data Protection Advisor
Data Protection Advisor allows you to unify and automate monitoring, alerting, and reporting across your data protection software and storage environments, including on-premises and cloud. Data Protection Advisor enables you to make informed decisions faster, to increase efficiency.
Dell RecoverPoint for Virtual Machines
RecoverPoint for Virtual Machines enables quick recovery of VMware virtual machines to any point in time. RecoverPoint for VMs provides continuous data protection (CDP) for operational recovery and disaster recovery. It is hypervisor-based, software-only data replication that integrates with VMware vCenter.
Dell APEX Backup Services
APEX Backup Services is a cloud-based data protection solution that ensures predictable, controllable costs without increasing complexity. It deploys in minutes and provides unlimited on-demand scaling to ensure that growing data volumes are always protected. An intuitive web-based experience provides centralized visibility and management across workloads and users, to ensure that your data is always protected.
With changing times, data protection solutions provided by Dell have also changed, and for the better. The variety of data protection products provided by Dell Technologies makes sure that data is easy to access, readily available, secured, and uncorrupted.
For more details, see the following resources:
Author: Charu, Engineering Technologist 2
Streamlining Unit Testing with GitHub Copilot: A Developer’s Guide – Spring Boot Java and Golang Application
Thu, 21 Mar 2024 18:47:09 -0000
|Read Time: 0 minutes
Unit testing is the backbone of robust software development. It ensures that the individual components of your codebase function correctly in isolation. However, writing comprehensive unit tests can be a labor-intensive task, especially when dealing with complex logic or large codebases. Enter GitHub Copilot – an AI-powered tool that promises to revolutionize the way developers write code.
The scenario
Imagine you’re building a Spring Boot or Golang based API. You’ve created a CustomerService class that contains methods to perform various business logic operations related to customers. Now, let’s see how GitHub Copilot can help generate unit tests and accelerate the process of writing unit tests for this class. GitHub Copilot aims to boost productivity by reducing boilerplate code and allowing developers to focus on business logic.
Spring Boot example
We’ll begin by creating the CustomerController class, which currently lacks any content. By effectively prompting Copilot, we can generate two essential operations: createCustomer and getCustomer.
Figure 1. Copilot prompt for creating a controller class with two operations
Note: The input to the prompt should be effective with all the necessary details.
Based on the prompt inputs, GitHub Copilot generated a class CustomerController with two methods: createCustomer and getCustomer.
Figure 2. Copilot auto generated code for a controller and two operations
Now that the CustomerController class has the necessary functions, the next critical step is to create unit tests to validate their functionality. Initially, the CustomerControllerTest file is empty. We’ll use effective prompting techniques to assist GitHub Copilot in generating unit tests for both the createCustomer and getCustomer functions.
Figure 3. Copilot prompt for writing unit test for Controller class
Upon generating the test function CustomerControllerTest using the GitHub Copilot command, you are now presented with a decision to accept or discard the generated test functions. Your task involves a thorough review, allowing you to make an informed choice regarding their acceptance or rejection.
Figure 4. Copilot prompt for accepting the auto generated unit tests
Upon acceptance, all generated contents are added to CustomerControllerTest class.
Figure 5. Copilot auto generated unit test cases
In the CustomerControllerTest function, errors such as import errors may arise. Utilize GitHub Copilot’s /fix feature to address and rectify these errors as they appear.
Figure 6. Copilot /fix prompt to fix errors in the code
Enhance your understanding of your code using the /explain feature. Simply select the code snippet you wish to understand better and employ the explain feature. This functionality provides developers with deeper insights into their codebase. Additionally, GitHub Copilot will provide suggestions such as ‘missing pom dependency’ to aid in code improvement, as seen in the following example.
Golang examples
Now, let’s explore how we can leverage similar features when developing a Go application.
Main.go currently stands as an empty file awaiting the creation of a router for two essential operations: creating and retrieving customers. Ensure all necessary details are included within the comments, following the format demonstrated in the following figure.
Figure 7. Copilot prompt for creating a router for Get and Create customer operations in GO
Note: The input to the prompt should be effective with all the necessary details.
Based on the prompt inputs, GitHub Copilot generated router code with two endpoints: createCustomer and getCustomers.
Figure 8. Copilot auto generated code for router and Get and Create customer endpoints
Next, we must develop handlers for both creating and retrieving customers. Currently, the CustomerHandler file remains void of any content. Let’s initiate a prompt to generate the necessary operations, as shown in the following figure.
Figure 9. Copilot prompt for writing a customer handler for Get and Create customer operations
Note: Please ensure that the input provided to the prompt is comprehensive and includes all necessary details for effective results.
According to the provided prompt inputs, GitHub Copilot has generated a class named CustomerHandler containing two methods: createCustomer and getCustomers.
Figure 10. Copilot auto generated code for Get and Create customer operation
The next step involves creating unit tests. Currently, CustomerHandler_test remains empty. GitHub Copilot will generate the unit tests for both creating and retrieving customers using the prompt shown in the following figure.
Figure 11. Copilot prompt for writing unit test
Now that the CustomerHandler_test class has been generated, as displayed in the following figure, GitHub Copilot provides you with the option to either accept or discard these tests. Take the time to review them thoroughly and make your decision accordingly.
Figure 12. Copilot auto generated unit test cases
Conclusion
Writing a REST Controller
Without Copilot: Writing a Customer Controller from scratch might take a developer an hour, depending on their familiarity with Spring Boot and the complexity of the controller.
With Copilot: By providing suggestions and auto-completion for code snippets, Copilot has the potential to cut development time by around 20-40%.
For example, it typically takes 1 hour to write the Customer Controller without Copilot. With Copilot's assistance, it might take around 20-30 minutes, saving approximately 30-40 minutes.
Writing unit test cases
GitHub Copilot can significantly reduce the time required to write unit tests by providing suggestions and auto-completion for test cases based on the provided prompt.
With Copilot's assistance, the time required to write unit tests for CustomerControllerTest was reduce by around 20-40%, dependent on the complexity of the controller and the level of test coverage desired. For example, it typically takes 2 hours to write unit tests without Copilot's assistance. With Copilot's help, it might take around 40 min to 1 hour, saving approximately 1 to 1.2 hours.
In conclusion, GitHub Copilot is a valuable tool for developing a Customer Service basic CRUD operation and its unit tests for createcustomer and getcustomer. Copilot significantly reduces the time and effort required for coding tasks, resulting in enhanced productivity and efficiency. Moreover, the reduction in lines of code ensures cleaner, more maintainable codebases while accelerating sample customer service development.
Happy writing, DEVs!
Authors:
Manoj Kumar Manoj Kumar | LinkedIn
Ratnesh Yadav Ratnesh (रत्नेश) Yadav | LinkedIn