Breaking the glass

Note: “Breaking the glass” is a term used by Microsoft to describe the process for unlocking Azure Stack Hub’s Privileged Endpoint (PEP/ERCS). This process allows the use of PowerShell scripts that require elevated permissions. This process requires the assistance of Dell Technologies Support or Microsoft Support.

Before getting started with the firmware and driver updates, contact either Dell Technologies Support or Microsoft Support to unlock or “break the glass” within the PEP session.  Doing so allows the ability to monitor the update process in real time while each scale unit node is being placed into Maintenance mode.  The basic process for unlocking the PEP session is shown below:

invoke-command -session $session -scriptblock { get-supportsessiontoken }

invoke-command -session $session -scriptblock { unlock-supportsession }

Figure 12.               Unlocking the PEP session

Note: It is recommended that an Azure Stack Hub Operator continue to have Dell Technologies Support or Microsoft Support monitor the progress after “breaking the glass.”

Verifying scale unit node firmware compliance

It is important to take a valid inventory of the environment to record the “before” and “after” results of the scale unit node’s firmware update process. 

1. Log into the OME VM server.

2. Go to C:\ComplianceTools.
3. Double-click on the Load Compliance Tools shortcut, which brings up the following PowerShell window:

Figure 13.               Load Compliance Tools

4. Use the following command to check each node’s current firmware baseline:

“Invoke-CheckFirmwareBaseline –IPAddresses <single IP address or comma separated list of scale unit node iDRAC IP addresses> –iDRACCredential (Get-Credential) -oemExtensionPath “DUPS” –ManagementServerAddress < OME VM OS IP > –ManagementServerCredential (Get-Credential)”

Note:  All characters contained with “< >” are for illustration purposes only. You will need to replace <xxxx> with those for your environment.

The following screen shot is a sample output. Items in red are the Current Version and the Available Version. If the Available Version is greater than the Current Version, those versions will need to be remediated. Pay close attention to the -IPAddresses switch, as it will change to match each scale unit node as they are updated.  

Note: When running the Invoke-CheckFirmwareBaseline command at this step in the update process, it is important that you DO NOT use the “-remediate” parameter.

Figure 14.               Sample output – Current and Available versions

Updating scale unit node firmware

The firmware updates should be applied in a round-robin or N+1 fashion, starting with scale unit node #1.

1. From the Administration Portal, select the node for remediation, and then select Drain.

Figure 15.               Draining a node

2. The following commands are used within the unlocked PEP session to monitor the Drain Status and State for each scale unit node:

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).State }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).StatusInformation }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).DrainStatus }

Figure 16.               Monitoring the drain status

Note: It is assumed that these updates are being applied in the production environment.  Also, some update cycles will require multiple reboots. Once the scale unit node is in Maintenance mode, the required files will be loaded to each iDRAC individually, so the updates may proceed one node at a time. When one node completes, the Azure Stack Hub Operator will Resume that node, wait for the cluster to re-balance that node, then Drain the next node for maintenance until all nodes have been completed. The firmware updates take about 45 minutes to run on each node and on average the re-balance process take from 60-120 minutes per node based on the activity and load of the cluster.

3. Once the drain is confirmed as complete, shut down the node through the Administration Portal.

Figure 17.               Shutting down a node

4. Log into the iDRAC of the scale unit node and proceed to the Job Queue tab.  Delete any stale entries before starting any new jobs.  The following screen shot is a sample of the Job Queue on a 13G server.

Figure 18.               Job Queue screen – 13G

The following screen shot is a sample of the Job Queue on a 14G server.

Figure 19.               Job Queue screen – 14G

5. Navigate to page/tab in the iDRAC GUI where updates are performed.

For 13G systems, navigate to iDRAC Settings, and then to Update and Rollback.

Figure 20.               Firmware Update screen – 13G

For 14G systems, select Maintenance, and then System Update.

Figure 21.               System Update tab

6. Upload the scale unit node’s firmware updates one at a time to the iDRAC.

Note: Make sure to upload the updates in the order they were listed in the Catalog.xml file.

The following list of firmware and the screen shot show updates to be applied to a 13G scale unit node.

• Firmware_HYPYY_WN64_3.35_A00-00.EXE
• Network_Firmware_P29N2_WN64_02.42.50.00.EXE
• Diagnostics_Application_JM78R_WN64_4239A37_4239.45.EXE
• BIOS_T9YX9_WN64_2.9.1.EXE
• SAS-Non-RAID_Firmware_8VCXF_WN64_16.17.00.03_A06_01.EXE
• Network_Firmware_Y82V5_WN64_14.23.15.04.EXE
• Firmware_HRP1V_WN64_2.25_A00-00_01.EXE

Once loaded, the updates are automatically prioritized by criticality.

7. Click Install and Reboot to proceed.

Figure 22.               Firmware Update screen

8. After you click Install and Reboot, proceed to the Job Queue to view the jobs that have been created and scheduled.

Figure 23.               Job Queue screen

9. After the jobs have been submitted to the queue, monitor the update progress by launching the iDRAC Virtual Console from the iDRAC dashboard.

Figure 24.               iDRAC virtual console

10. Once completed and once the last reboot takes place from the iDRAC, go back to the Job Queue and validate that all the jobs are marked as Completed.

Figure 25.               Job Queue

Note: When all the jobs have been reported as completed, the jobs may be selected and deleted from the queue.

11. If you haven’t done so already on the OME VM, browse back to C:\ComplianceTools and double-click on the Load Compliance Tools shortcut, which brings up the Select Load Compliance Tools PowerShell window.
12. At the completion of the firmware updates, verify the changes have taken place on the scale unit node:

“Invoke-CheckFirmwareBaseline –IPAddresses <single IP address of current scale unit node’s iDRAC> –iDRACCredential (Get-Credential) -oemExtensionPath “DUPS” –ManagementServerAddress < OME VM OS IP > –ManagementServerCredential (Get-Credential)”

Note: When running the Invoke-CheckFirmwareBaseline command at this step in the update process, it is important that you DO NOT USE the “-remediate” parameter.

Figure 26.               Verifying changes

13. Then use the following command to validate the BIOS settings for each scale unit node:

“Invoke-CheckBIOSSettings –IPAddresses <single IP address of current scale unit node’s iDRAC>  –iDRACCredential (Get-Credential)”

Figure 27.               Validating BIOS settings

14. Then use the following command to validate the iDRAC settings for each scale unit node:

“Invoke-CheckIDRACSettings –IPAddresses <single IP address of current scale unit node’s iDRAC> –iDRACCredential (Get-Credential)”

Figure 28.               Invoke-CheckIDRACSettings

Note: The iDRAC settings function automatically validates and remediates since a change to the iDRAC settings does not reboot the server.

Resuming scale unit node after completing updates

1. As each node gets validated, use the Administration Portal to Start and Resume that Scale Unit Node prior to performing a Drain operation on the next sequential node for updates.

Figure 29.               Start/Resume

2. Use the following commands to monitor its status until the State value is equal to Up.

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).State }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).StatusInformation }

invoke-command -session $session -scriptblock { (get-clusternode -cluster s-cluster -name sac02-s1-n01).DrainStatus }

Figure 30.               Monitoring the status

3. At this point, the State for each node is shown as Up, but monitor the Cluster Storage Jobs to be sure the resources are properly balanced before proceeding to the Drain process on the next node in-line for updates.

“invoke-command -session $session -scriptblock { get-storagejob }”

“invoke-command -session $session -scriptblock { get-virtualdisk -cimsession s-cluster | Get-StorageJob }”

Figure 31.               Monitoring the cluster storage jobs

Figure 32.               Node cores and memory

Repeat all the steps for each scale unit node until each node in the cluster has been updated and validated as required.

Note: When all the Scale Unit Nodes have been updated, exit the unlocked or “Broken Glass” PEP session.