Breaking the glass

Note: “Breaking the glass” is a term used by Microsoft to describe the process for unlocking Azure Stack Hub’s Privileged Endpoint (PEP/ERCS). This process allows the use of PowerShell scripts that require elevated permissions. This process requires the assistance of Dell Technologies Support or Microsoft Support.

Before getting started with the firmware and driver updates, contact either Dell Technologies Support or Microsoft Support to unlock or “break the glass” in the PEP session. Doing so allows monitoring of the update process in real time while each SU node is being placed into Maintenance mode. The basic process for unlocking the PEP session is shown below:

invoke-command -session $session -scriptblock { get-supportsessiontoken }

invoke-command -session $session -scriptblock { unlock-supportsession }

Figure 3.                  Unlocking the PEP session

Note: We recommend that an Azure Stack Hub Operator continues to have Dell Technologies Support or Microsoft Support monitor the progress after “breaking the glass.”

Verifying scale unit node firmware compliance

Note: You must take a valid inventory of the environment to record the “before” and “after” results of the SU node’s firmware update process. 

Procedure

1. Log in to the OpenManage Enterprise VM server.

2. Go to C:\ComplianceTools.
3. Double-click the Load Compliance Tools shortcut, which opens a PowerShell window:

4. Use the following command to check each node’s current firmware baseline:

Note: When running the Invoke-CheckFirmwareBaseline command at this step in the update process, do NOT use the “-remediate” parameter.

“Invoke-CheckFirmwareBaseline –IPAddresses <single IP address or comma separated list of scale unit node iDRAC IP addresses> –iDRACCredential (Get-Credential) -oemExtensionPath “DUPS” –ManagementServerAddress < OME VM OS IP > –ManagementServerCredential (Get-Credential)”

Note:  All characters contained with “< >” are for illustration purposes only. You must replace <xxxx> with those for your environment.

The following figure is a sample output. Items in red are the Current Version and the Available Version. If the Available Version is greater than the Current Version, those versions will need to be remediated. Pay close attention to the -IPAddresses switch, as it will change to match each scale unit node as they are updated.  

Updating scale unit node firmware

The firmware updates should be applied in a round-robin or N+1 fashion, starting with SU node #1.

Procedure

1. From the Administration Portal, select the node for remediation, and then select Drain.

2. Use the following commands in the unlocked PEP session to monitor the Drain Status and State for each SU node:

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).State }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).StatusInformation }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).DrainStatus }

Note: It is assumed that these updates are being applied in the production environment.  Some update cycles require multiple reboots. After the SU node is in Maintenance mode, load the required files to each iDRAC individually, so that the updates can proceed one node at a time. When one node completes, the Azure Stack Hub Operator resumes that node, waits for the cluster to re-balance that node, then drains the next node for maintenance until all nodes are complete. The firmware updates take about 45 minutes to run on each node. On average, the re-balance process take from 60-120 minutes per node based on the activity and load of the cluster.

3. After the drain is confirmed as complete, shut down the node through the Administration Portal.

4. Log in to the iDRAC of the SU node and go to the Job Queue tab. Delete any stale entries before starting any new jobs. 

The following figure is a sample of the Job Queue on a 13G server.

The following figure is a sample of the Job Queue on a 14G server.

5. Navigate to the page/tab in the iDRAC UI where updates are performed.
   1. For 13G systems, navigate to iDRAC Settings, and then to Update and Rollback.

2. For 14G systems, select Maintenance, and then System Update.

6. Upload the SU node’s firmware updates one at a time to the iDRAC.

Note: Upload the updates in the order they are listed in the Catalog.xml file.

The following list of firmware and the figure show updates to be applied to a 13G SU node:

• Firmware_HYPYY_WN64_3.35_A00-00.EXE
• Network_Firmware_P29N2_WN64_02.42.50.00.EXE
• Diagnostics_Application_JM78R_WN64_4239A37_4239.45.EXE
• BIOS_T9YX9_WN64_2.9.1.EXE
• SAS-Non-RAID_Firmware_8VCXF_WN64_16.17.00.03_A06_01.EXE
• Network_Firmware_Y82V5_WN64_14.23.15.04.EXE
• Firmware_HRP1V_WN64_2.25_A00-00_01.EXE

Once loaded, the updates are automatically prioritized by criticality.

7. To proceed, click Install and Reboot.

8. To view the jobs that have been created and scheduled, go to Job Queue.

9. After the jobs are submitted to the queue, monitor the update progress by launching the iDRAC Virtual Console from the iDRAC dashboard.

10. After the job complete and the last reboot takes place from the iDRAC, return to the Job Queue and validate that all the jobs are marked as Completed.

Note: When all the jobs have been reported as completed, you can select and delete the jobs from the queue.

11. If you have not done so already, on the OpenManage Enterprise VM, go back to the C:\ComplianceTools and double-click on the Load Compliance Tools shortcut, which opens the Select Load Compliance Tools PowerShell window.
12. On completion of the firmware updates, use the following command to verify the changes have taken place on the SU node:

“Invoke-CheckFirmwareBaseline –IPAddresses <single IP address of current scale unit node’s iDRAC> –iDRACCredential (Get-Credential) -oemExtensionPath “DUPS” –ManagementServerAddress < OME VM OS IP > –ManagementServerCredential (Get-Credential)”

Note: When running the Invoke-CheckFirmwareBaseline command at this step in the update process, do NOT use the “-remediate” parameter.

13. Use the following command to validate the BIOS settings for each SU node:

“Invoke-CheckBIOSSettings –IPAddresses <single IP address of current scale unit node’s iDRAC>  –iDRACCredential (Get-Credential)”

14. Use the following command to validate the iDRAC settings for each SU node:

“Invoke-CheckIDRACSettings –IPAddresses <single IP address of current scale unit node’s iDRAC> –iDRACCredential (Get-Credential)”

Note: The iDRAC settings function automatically validates and remediates, because a change to the iDRAC settings does not reboot the server.

Resuming scale unit node after completing updates

Procedure

To resume after the updates:

1. As each node is validated, use the Administration Portal to Start and Resume that SU node prior to performing a Drain operation on the next sequential node for updates.

2. Use the following commands to monitor the node status until the State value displays as Up.

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).State }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).StatusInformation }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).DrainStatus }

3. At this point, the State for each node is shown as Up, but use the following commands to monitor the Cluster Storage Jobs to ensure that the resources are properly balanced before proceeding to the Drain process on the next node in-line for updates:

“invoke-command -session $session -scriptblock { get-storagejob }”

“invoke-command -session $session -scriptblock { get-virtualdisk -cimsession s-cluster | Get-StorageJob }”

4. Repeat all the previous steps for each SU node until each node in the cluster is updated and validated as required.

Note: When all the SU nodes are updated, exit the unlocked or “Broken Glass” PEP session.