Integrating CloudIQ Webhooks with BigPanda Events
Tue, 22 Nov 2022 17:27:08 -0000
|Read Time: 0 minutes
This tutorial blog demonstrates how to use CloudIQ Webhooks to integrate CloudIQ health notifications with BigPanda (https://www.bigpanda.io/), an event management processing tool. This allows users to integrate CloudIQ notifications with events from other IT tools into BigPanda. We will show how to create a REST API Integration in BigPanda and provide an example of intermediate code that uses Google Cloud functions to process Webhooks.
BigPanda overview
BigPanda offers a solution that has a modern twist on event management process. The main product consists of a fully customizable cloud-hosted event management console for event integration, reporting, correlation, and enrichment.
Webhook overview
A CloudIQ Webhook is a notification that is sent when a health issue changes. CloudIQ sends the Webhook notification when a new or resolved health issue is identified in CloudIQ. A Webhook is an HTTP post composed of a header and JSON payload that is sent to a user configured destination. Webhooks are available under the Admin > Integrations menu in the CloudIQ UI. Users must have the CloudIQ DevOps role to access the Integrations menu.
Webhook event details
A Webhook consists of data in the header and the payload. The header includes control information; the payload is a JSON data structure that includes useful details about the notification and the health issue. Examples of the header and payload JSON files can be found here.
BigPanda integration
In CloudIQ, we enable Webhook integration by configuring a name, destination, and the secret to sign the payload.
In BigPanda, we have a couple of possibilities for third-party integration:
- BigPanda's Open Integration Hub allows users to implement a mapping of elements from a graphical interface.
- The raw REST API allows users to integrate with BigPanda programmatically.
In our example, we use the REST API. Note that some of the requirements of the Open Integration Hub (alert severity, configurable application key, and so on) are not configurable today in CloudIQ Webhooks.
Architecture
The main challenge when integrating CloudIQ health events with BigPanda alerts is implementing a mapping function to translate CloudIQ fields to BigPanda fields.
To do this, we will use a serverless function to:
- Receive the health event from a CloudIQ Webhook trigger
- Convert the CloudIQ health event to a BigPanda alert
- Post that alert to BigPanda
In this integration, the serverless function is a Google Cloud Function. Any other serverless framework can work.
Create a BigPanda REST application
The first step is to create an application for integration in BigPanda. Do the following:
1. Log into the BigPanda console.
2. Click the Integrations button at the top of the console.
3. Click the blue New Integration button.
4. Select Alerts Rest API (the first card).
5. Set an integration name, then click Generate App Key.
6. Save the generated app key and bearer token.
If you forgot to save the “application key” or “token”, you can obtain them later by selecting `Review Instructions`.
Note that the “application key” and “token” will be needed later to configure the trigger to post data to that endpoint.
Create the GCP Cloud function
This step is very similar to what has been presented in the CloudIQ to Slack tutorial. The only changes are that we are using a golang runtime and we store the authentication token in a secret instead of in a plain text environment variable.
- Select Create Secret from the Secret Manager.
2. Provide a name (BP_TOKEN in this example).
3. Paste the Authorization token from the HTTP headers section of the BigPanda integration into the ‘Secret value’ field.
4. Select Create Function and provide a function name (ciq-bigpanda-integration in this example).
5. Under the Trigger section, keep a trigger type of HTTP and select Allow unauthenticated invocations.
6. Take note of the Trigger URL because it will be used as the Payload URL when configuring the Webhook in CloudIQ.
7. Select SAVE.
8. Expand the RUNTIME, BUILD AND CONNECTIONS SETTINGS section.
9. Under the RUNTIME tab, click the + ADD VARIABLE button to create the following variable:
BP_APP_KEY. The value is set to the application key obtained after creating the BigPanda integration.
10. Select the SECURITY AND IMAGE REPO tab.
11. Select REFERENCE A SECRET.
12. Select the BP_TOKEN secret from the pulldown.
13. Select Exposed as environment variable from the Reference Method pulldown.
14. Enter BP_TOKEN as the environment variable name.
15. Select DONE, then click Next.
16. Select Go 1.16 from the Runtime pulldown.
17. Change the Entry point to CiqEventToBigPandaAlert.
18. Replace the code for function.go with the example function.go code.
19. Replace the go.mod with the example go.mod code.
20. Select DEPLOY.
Implement the Mapping
Using Go's static typing first approach, we have clearly defined `struct` for the input (`CiqHealthEvent`) and output (`BigPandaAlerts`).
Most of the logic consists of mapping one field to the other.
func CiqEventMapping(c *CiqHealthEvent, bp *BigPandaClient) *BigPandaAlerts { log.Println("mapping input CloudIQ event: ") log.Printf("%+v", c) alert := BigPandaAlerts{ AppKey: bp.AppKey, Cluster: "CloudIQ", Host: c.SystemName, } if len(c.NewIssues) > 0 { for _, v := range c.NewIssues { alert.Alerts = append(alert.Alerts, BigPandaAlert{ Status: statusForScore(c.CurrentScore), Timestamp: c.Timestamp, Host: c.SystemName, Description: v.Description, Check: v.RuleID, IncidentIdentifier: v.ID, }) } } return &alert }
Two things to note here:
1. Because CloudIQ doesn't have the notion of severity, we convert the score to a status using the code below.
2. CloudIQ has an event identifier that will help to deduplicate the alert in BigPanda or reopen a closed event in case of a re-notify.
// BigPanda status values: ok,ok-suspect,warning,warning-suspect,critical,critical-suspect,unknown,acknowledged,oksuspect,warningsuspect,criticalsuspect,ok_suspect,warning_suspect,critical_suspect,ok suspect,warning suspect,critical suspect func statusForScore(s int) string { if s == 100 { return "ok" } else if s <= 99 && s > 95 { return "ok suspect" } else if s <= 95 && s > 70 { return "warning" } else if s <= 70 { return "critical" } else { return "unknown" } }
Build
Behind the scenes, the GCP Cloud Functions are built and executed as a container. To develop and test the code locally (instead of doing everything in the GCP Console), we can develop locally and then build the package using buildpack (https://github.com/googlecloudplatform/buildpacks) as GCP does:
pack build \ --builder gcr.io/buildpacks/builder:v1 \ --env GOOGLE_RUNTIME=go \ --env GOOGLE_FUNCTION_SIGNATURE_TYPE=http \ --env GOOGLE_FUNCTION_TARGET=ciq-bigpanda-integration \ ciq-bigpanda-integration
Run
After the build is successful, we can test it with something similar to:
docker run --rm -p 8080:8080 -e BP_TOKEN=xxxxx -e BP_APP_KEY=yyyyy ciq-bigpanda-integration
Alternatively, you can create a “main.go” and run it with:
FUNCTION_TARGET=CiqEventToBigPandaAlert go run cmd/main.go
Deploy
Users can choose to deploy the function outside of the GCP console. You can publish it with:
cloud functions deploy ciq-bigpanda-integration --runtime go116 --entry-point ciq-bigpanda-integration --trigger-http --allow-unauthenticated
Configure CloudIQ
It is time to point the CloudIQ Webhook to the GCP Function trigger URL. From the Admin > Integrations menu in CloudIQ, go to the Webhooks tab.
- Click Add Webhook.
- Enter a Name for the Webhook.
- Enter the Payload URL. This is the Trigger URL from the GCP Function.
- Because we did not use a Webhook secret, enter any text.
- Click ADD WEBHOOK to save the configuration.
Testing
From CloudIQ
To ease the simulation of a Webhook event, go to the CloudIQ Integration and click the TEST WEBHOOK button. This sends a ping request to the destination. You can also go to CloudIQ and redeliver an existing event.
Easy post script
For an actual event and not just a `ping`, use the `easy_post.sh` script after configuring the appropriate ENDPOINT.
#!/bin/bash HEADERS_FILE=${HEADERS_FILE-./headers.json} PAYLOAD_FILE=${PAYLOAD_FILE-./payload.json} ENDPOINT=${ENDPOINT-https://webhook.site/6fd7d650-1b5b-4b8c-9781-2043005bdf2d} mapfile -t HEADERS < <(jq -r '. | to_entries[] | "-H \(.key):\(.value)"'< ${HEADERS_FILE}) curl -k -H "Content-Type: application/json" ${HEADERS[@]} --request POST --data @${PAYLOAD_FILE} ${ENDPOINT}
Conclusion
If everything flows correctly, you will see the health alerts delivered to the BigPanda console. This allows users to consolidate CloudIQ notificaitons with events from other IT tools into a single monitoring interface.
Resources
- BigPanda API Reference - Alerts
- Google Cloud - the Go Runtime
- Dell Technologies Developer - CloudIQ Webhooks
Author: Derek Barboza
Related Blog Posts
Talking CloudIQ: PowerEdge
Wed, 08 Nov 2023 16:32:28 -0000
|Read Time: 0 minutes
Introduction
In my previous blogs, I have focused on a specific feature in CloudIQ. This blog talks about various CloudIQ features for Dell’s PowerEdge servers. Dell CloudIQ continues to expand its feature set for PowerEdge assets. CloudIQ integrates with Dell’s OpenManage Enterprise at each of your sites, to efficiently collect and aggregate telemetry data to give you a multisite, enterprise-wide view of all your PowerEdge servers and chassis. And with OpenManage Enterprise 4.0, onboarding your PowerEdge servers to CloudIQ is easier than ever!
Health, inventory, and performance
Since the introduction of PowerEdge support in CloudIQ, health, inventory, and performance monitoring for PowerEdge servers have all been available. CloudIQ provides an overall health score for each PowerEdge server and recommended remediation when an issue is identified. Inventory reporting provides numerous properties about each server, including contract status, component firmware versions, licensing information, and hardware listings to name a few. CloudIQ displays key performance metrics and not only shows historical trends but identifies performance anomalies and provides performance forecasting. This information allows you to see unexpected performance patterns, and plan future resource needs based on trending workloads.
Figure 1. Example of a performance forecasting chart for PowerEdge
Cybersecurity
Cybersecurity is a feature in CloudIQ that allows you to compare your existing security configuration settings to a predefined set of desired security configuration settings. The configuration is continuously monitored, notifying you when a configuration does not meet its desired setting. Cybersecurity monitors up to 31 server configuration settings and 18 chassis configuration settings tied to NIST security standards. Without automated continuous checking, it's impractical to manually check all settings on all servers every day. Lab tests show that it takes six minutes on average to manually check just 15 settings on a single server.
Users can also see a list of applicable Dell Security Advisories (DSAs) for their PowerEdge systems. By intelligently matching attributes like models and code versions, users can quickly see which DSAs are applicable to their systems, allowing them to take immediate action to remediate these security vulnerabilities.
Figure 2. The Security Assessment page for a PowerEdge chassis
System Management
You can now initiate BIOS and firmware updates for PowerEdge servers and chassis from CloudIQ. Users with a Server Admin role in CloudIQ can initiate these upgrades across multiple systems with just a few clicks. This feature simplifies the process of keeping your fleet of servers consistent and secure.
Figure 3. Multisystem update for PowerEdge servers and chassis
Virtualization View
The integration of PowerEdge into the Virtualization View consolidates and simplifies resource information about PowerEdge servers running ESXi. Available details include the OS version, model, resource consumption per virtual machine, and health issues with recommendations for remediation. A hyperlink lets you quickly navigate to the system details page for the PowerEdge server for more troubleshooting. Another hyperlink directs you to vCenter to perform virtualized resource administration.
Figure 4. PowerEdge support in the Virtualization View
Carbon footprint monitoring
CloudIQ has introduced carbon footprint analysis support for PowerEdge servers and chassis. CloudIQ takes power and energy metrics and calculates carbon emissions based on international standards and conversion factors for location. CloudIQ Administrators can override and customize these values with their own unique location emission factors.
Figure 5. Energy, power, and carbon emissions for a PowerEdge server
Custom reports and IT integrations
You can generate custom reports using both tables and charts for PowerEdge servers:
- Tables are available to provide lists of assets, code versions, contract information, capacity metrics, and average performance metrics.
- Charts can be used to see historical performance trends and performance anomalies.
You can also take advantage of custom tags in your reports. For example, you can create a list of PowerEdge servers in a certain business unit with their BIOS and firmware versions, contract expiration dates, average power consumption, and service tags. And with Webhooks and REST API access, you can integrate data and events from CloudIQ with ServiceNow, Slack, and other IT tools to help you monitor your entire Dell IT infrastructure.
Figure 6. Custom reporting table for PowerEdge with custom tags
Conclusion
As IT resources become more remote and isolated, it has become increasingly time consuming to maintain, manage, and secure resources in the data center and at the edge. CloudIQ simplifies monitoring and management by providing a single portal to view all your PowerEdge servers across your entire environment. With cybersecurity monitoring of PowerEdge servers and chassis, you can quickly see where security configuration settings may be incorrectly set or accidentally changed, opening those systems to cyberattacks, and receive instructions to remediate. With the new maintenance and management features, CloudIQ simplifies the process of keeping your entire fleet at consistent, secure, and desired BIOS and firmware versions. The carbon footprint page in CloudIQ helps you meet sustainability goals. And with Webhook and REST API support, CloudIQ can be integrated with other IT tools to help you monitor not only your PowerEdge servers, but your entire Dell IT portfolio.
Resources
This Knowledge Base Article discusses how to onboard PowerEdge devices to CloudIQ.
For a quick demo about CloudIQ PowerEdge support, see the CloudIQ videos section on the Info Hub.
Direct from Development Tech Note: Dell CloudIQ Cybersecurity for PowerEdge: The Benefits of Automation
See other informative blogs: Overview of CloudIQ, Proactive Health Scores, Capacity Monitoring and Planning, Cybersecurity, and Custom Reports and Tags.
How do you become more familiar with Dell Technologies and CloudIQ? The Dell Technologies Info Hub site provides expertise that helps to ensure customer success with Dell Technologies platforms. We have CloudIQ demos, white papers, and videos available at the Dell Technologies CloudIQ page. Also, feel free to reference the white paper CloudIQ: A Detailed Overview which provides an in-depth summary of CloudIQ.
Author: Derek Barboza, Senior Principal Engineering Technologist
CloudIQ - Carbon Footprint Analysis
Wed, 04 Oct 2023 16:03:26 -0000
|Read Time: 0 minutes
In this blog post, we’ll cover a topic that is top of mind for all organizations, small and large--Energy Efficiency. I’ll also highlight how Dell Technologies helps customers increase energy efficiency using our vast portfolio. First, let’s define what Energy Efficiency is.
“Simply put, energy efficiency means using less energy to get the same job done
– and in the process, cutting energy bills and reducing pollution.”
Reference: Energy Efficiency | ENERGY STAR
As organizations undergo digital transformation and modernization, there is a massive explosion in the amount of data that needs to be stored. This data expansion is driven by technologies like Cloud Computing, Artificial Intelligence, and streaming services, just to name a few. This in turn impacts how much power organizations are now consuming in their data centers, which forces IT vendors to make their solutions more efficient and reduce emission and carbon footprint.
Dell Technologies has been helping customers harness the power of technology to drive human progress for several decades. Our latest Environmental, Social and Governance report focuses on the investments Dell has made to support these initiatives.
If you’re interested in delving deeper, check out Dell's FY23 Environmental, Social and Governance.
Energy concerns were of paramount importance for our customers in 2022, not only in response to rising energy costs but also as they worked toward reducing emissions. As a leader in sustainable technology, Dell partnered with customers to make the transition to more energy efficient data centers with advanced cooling and thermals, power management tools, and as-a-Service (aaS) solutions to “right size” data storage. With the cost of energy commodities expected to be on average 46% higher in 2023, we will continue to set the standard on data center infrastructure solutions to drive efficient operational and environmental outcomes for our customers.
Dell reinvests over $4B in R&D on an annual basis, continuing to lead the market with our innovation in storage and data reduction efficiencies to save energy and reduce our carbon and hardware footprint.
Dell’s commitment to reducing carbon footprint is exemplified by the introduction of innovative ideas to optimize our portfolio. Recognized as one of the winners of Fast Company’s 2023 World Changing Ideas Awards, Dell’s Concept Luna was designed to showcase how the future of electronic devices can be one where they’re repaired instead of thrown out. Feel free review the full article, How Dell is infusing sustainability across its businesses, to learn more
Based on what we covered so far, we truly believe that informing our customers of critical data points that contribute to overall awareness of power, energy consumption, and carbon footprint is essential.
CloudIQ Carbon Footprint: integrating energy efficiency across systems
Several years ago, Dell Technologies developed a product called CloudIQ, the cloud-based AIOps proactive monitoring and predictive analytics application for Dell systems. CloudIQ leverages machine learning and other algorithms, notifications, and recommendations to help customers optimize compute, storage, data protection, and network health, performance, and capacity. CloudIQ supports a broad range of Dell Technologies products, including:
- Servers -- PowerEdge
- Storage -- PowerStore, PowerMax, PowerScale, PowerVault, Unity, Unity XT, XtremIO, and SC Series
- Data protection -- PowerProtect DD and PowerProtect Data Manager
- Converged and hyperconverged infrastructure -- VxBlock, VxRail, and PowerFlex
- Networking -- PowerSwitch and Connectrix, plus Dell Technologies APEX Data Storage Services
Over 90% of our customers actively use CloudIQ as their centralized dashboard to inform them proactively about KPIs across their Dell Technologies estate.
Introducing Carbon Footprint, an additional capability within CloudIQ designed to provide insights for power, energy consumption, and carbon footprint forecasting across all systems. At the time of the initial release, we are supporting the following products from our portfolio:
- PowerEdge
- VxRail
- Unity
- PowerScale
- Connectrix Systems
and focusing on the following KPIs:
- Total carbon emissions for this year (YTD)
- Energy consumption trends (monthly and YTD)
- 24H power consumption, 24H load on average
- Historical and forecast data for energy and carbon footprint
Later in 2023, we will also add support for PowerSwitch.
Having Carbon Footprint enabled and KPIs exposed within CloudIQ is beneficial to internal stakeholders within an organization and allows you to make confident decisions when optimizing your environment.
Based on the Software-as-a-Service (SaaS) model and agile development methodology employed by CloudIQ, you’ll benefit from having access to new features as soon as they become available.
Most Dell Technologies products supported by CloudIQ leverage our call home functionality called SupportAssist / Secure Connect Gateway. Depending on the product, you will need to enable the CloudIQ feature, after which the CloudIQ dashboard will populate with data.
For the full overview of the CloudIQ product, please see the detailed review whitepaper here.
Accessing and using Carbon Footprint
To access the Carbon Footprint feature in the CloudIQ dashboard, select Monitor > Carbon Footprint on the left-hand side of the CloudIQ console, as shown in the following figure.
On this screen, CloudIQ users with the CloudIQ Admin role will be able to adjust and personalize their geographical location metrics, such as CO2E and PUE, as illustrated in the following figure. The location labels reflect the specific locations where the physical assets are installed.
Side note: What do these metrics mean?
- Carbon dioxide equivalent (CO2e) refers to the number of metric tons of CO2 emissions with the same global warming potential as one metric ton of another greenhouse gas. Other greenhouse gases, like methane, have different global warming potentials--a measurement of the potential impact a greenhouse gas has on global warming over a given period--compared to carbon dioxide. By converting all greenhouse gas emissions into CO2e units, it becomes easier to compare the impact of different types of emissions and to create strategies for reducing GHG emissions.
- Power usage effectiveness (PUE) is used to determine the energy efficiency of a data center. The best PUE ratio is 1.0, indicating a perfectly efficient data center in which 100% of the facility's power is delivered to IT equipment. This means that no power is used for any other purpose in the facility such as cooling, lighting, or any other overhead that supports the equipment.
The Total Carbon Emissions CO2e section can be displayed using either a Bar Chart or a Line Chart. Simply select the gear wheel on the right-hand side and pick your preferred view.
The Total Carbon Emissions CO2e chart can increase or decrease based on how the system’s energy / emission factor / PUE changes over time. If new systems are added, the total will increase. Similarly, the total can decrease if power is capped (as is available for PowerEdge), workloads are reconciled, and/or some systems are shut down.
For larger environments with multiple assets, applying filters is a breeze. The following example shows the system filtered based on Unity arrays only.
This table displays several columns that represent the asset itself, its location, site name, etc. In addition, we show the following data points:
- YTD Energy (kWh) - YTD value is from when power consumption data collection started, which may not have been the start of the calendar year
- Energy Forecast (kWh) - Forecasted energy consumption at the end of the year (December 31st) in Kilowatts (kWh)
- YTD CO2e (kg) - YTD value of carbon emissions measured when data collection started, which may not have been the start of the calendar year.
- CO2e Forecast (kg) - Forecasted CO2 (carbon dioxide) equivalent produced at the end of the year (December 31st) in Kilograms (kg)
The entire table with all assets or a subset thereof can be exported into a CSV file.
To see more details for each of the assets and how they perform in comparison to historical data, select the details icon next to the asset itself. As displayed in the following figure, the two graphs will display data points over the last seven (7) days and forecasted data points for the next thirty (30) days. By toggling the radio button, you can switch from one view to another. The grey area shows a range based on historical data collected for the previous seven (7) days, and the blue line is charted based on the last seven (7) days. If the blue line is within the boundaries of the grey area, this means there have been no unforeseen changes in Energy Consumption or the CO2e in the past seven (7) days.
What we’ve shown here is just the beginning. We will continue improving and enhancing CloudIQ capabilities to ensure you enjoy the most relevant and accurate KPIs and can act upon them expeditiously.
Where do we go from here?
Dell has several other tools and assessments that you can benefit from. For example, we can run a tool called Live Optics that collects configuration and performance data from your Dell and third-party products and produces a report that can be analyzed to propose options like optimization, consolidation, or a refresh with a new technology. The other option is to request a #GetEfficient report, which will be directly focused on reduction in physical footprint and power consumption.
Resources
- Dell CloudIQ: A Detailed Review
- How Dell is infusing sustainability across its businesses
- Dell FY23 Environmental, Social and Governance - Executive Summary
Author: Michael Aharon & Derek Barboza