- Problem
-
Cannot add a server to Bare Metal Orchestrator if the server uses these three independent BIOS parameters: - CorrEccSmi
- DcuIpPrefetcher
- SriovGlobalEnable
(MWV-1546) - Workaround
- None
|
- Problem
-
If you target a server with a hardware profile and then disassociate the server from the hardware profile, Bare Metal Orchestrator continues to apply the hardware profile's configuration settings to the server. This means that if you reinitialize the server, the server has the hardware profile settings reapplied. (MWV-2152) - Workaround
- Create a new hardware profile for the server.
|
- Problem
-
After drives are securely erased, the iDRAC goes into an unreachable state. (MWV-2743) - Workaround
- Wait five minutes after secure erasure for the iDRAC to come back online.
|
- Problem
-
Bare Metal Orchestrator cannot contact worker nodes that are behind a network address translation (NAT). (MWV-3477) - Workaround
- None
|
- Problem
-
The BMO dashboard status bar has a system location counter that shows the number of servers that do not have location information. If you edit any of the system location fields for a server, the server is counted as having location data and the no location count is inaccurate. (MWV-3812) - Workaround
- Edit all system location fields to ensure the location count is correct.
|
- Problem
-
When accessing iDRAC names or addresses remotely, such as through a NAT server, iDRAC9 version 5.10.x.x requires manual iDRAC configuration. By default, iDRAC9 checks the HTTP or HTTPS Host Header and compares the defined DNSRacName and DNSDomainName. When the values do not match, the iDRAC refuses the HTTP or HTTPS connection. See the following article for more details and workarounds: HTTP/HTTPS FQDN Connection Failures On iDRAC9 firmware version 5.10.00.00. (MWV-4143) - Workaround
- You can also install iDRAC firmware version 5.00.20.xx, which does not contain the host header check.
|
- Problem
-
When you edit an existing CRD object, use the CRD object name in the command instead of using the -f (filename) option. Attributes edited with the bmo edit hardwareprofile -f profile.yaml command and -f option are not recognized by Bare Metal Orchestrator. (MWV-4655) - Workaround
- Dell Technologies recommends using the CRD name when editing a CRD object. For example, use the following command:
bmo edit hardwareprofile <CRD object name> |
- Problem
-
In VMware vRealize Orchestrator (vRO) version 8.3, there is a module with an expired license that makes it unusable during TCP stack deployment. (MWV-4732) - Workaround
- Use the latest version of vRO.
|
- Problem
-
If you request a TKG at the time of VMware Telco Cloud Platform (TCP) stack deployment in VMware, the TCP stack deployment completes but does not include a TKG. (MWV-4956) - Workaround
- After TCP stack deployment is complete, you can add a TKG by editing the TCP stack and defining templates and clusters.
|
- Problem
-
After you create a firmware media or OS media object and assign it an external IP address, you cannot remove the object's external IP address. If you try to remove the object's external IP address and save the changes, the address is not removed. (MWV-4976) - Workaround
- If you need to remove a firmware or OS media object's external IP address, set the object's external IP address to the Bare Metal Orchestrator VM's external IP address.
|
- Problem
-
If you try to deploy VMware Telco Cloud Platform (TCP) stack in an high availability (HA) environment, the TCP stack deployment task may time out. This is an intermittent defect and does not happen during every deployment. (MWV-5035) - Workaround
- Contact Dell Support and request that they restart the mw-stack-sku-pack service.
|
- Problem
-
When you deploy the TCP stack and if there is more than one Solid-state Drive (SSD) present in the system, TCP stack deployment may fail with one of the following error messages: error creating and mounting datastore" or "device cache contains partitions: ['vmfs']". (MWV-5856) - Workaround
- Onboard all ESXi servers on which the stack is deployed to Bare Metal Orchestrator again, and re-initiate the TCP stack deployment.
|
- Problem
-
When you onboard a server with ESXi operating system version ESXi 7.0 U2 installed on the storage volume and deploy the same version on the storage volume again, server onboarding fails. (MWV-6106) - Workaround
- Secure Erase the RAID volume disks before deploying the ESXi 7.0 U2 operating system on the server.
|
- Problem
-
Using the Bare Metal Orchestrator web user interface to upload ISO images larger than 5 GB on slow Internet connections may cause the upload to fail. (MWV-6442) - Workaround
- Upload the ISO file to the Bare Metal Orchestrator server using WinSCP or SCP. Then, log in to the Bare Metal Orchestrator server and upload the ISO using the Bare Metal Orchestrator CLI.
|
- Problem
-
After a restore operation, if the restore status is Partially Failed but the restored object count is equal to the total backed up object count, the restore operation can be considered as successful. (MWV-6498) - Workaround
- None
|
- Problem
-
Deleting a Cisco switch in Bare Metal Orchestrator does not effect on Network Services Orchestrator (NSO). (MWV-6576) - Workaround
-
If you want to delete a Cisco switch from Bare Metal Orchestrator, delete the Cisco switch from Network Services Orchestrator (NSO). |
- Problem
-
When a site deployment fails and is reinitialized, Bare Metal Orchestrator event is not generated after the site goes to "Ready" state after reinitialization. (MWV-6632) - Workaround
-
Check the status of the site manually after reinitialization. |
- Problem
-
Bare Metal Orchestrator cannot install ESXi version 7u1 operating system on the worker nodes for Dell PowerEdge R650 and R750 servers. (MWV-7255) - Workaround
-
If your BIOS has a setting for Secure Boot, set it to disabled. |
- Problem
-
When you specify BIOS attributes and BIOS firmware update at the same time during a server creation, the server goes to Failed state. (MWV-7453) - Workaround
-
Do not specify BIOS attributes and BIOS firmware update at the same time during server creation. |
- Problem
-
The ESXi operating system installation may fail with the following error message: Error reading from StdOut pipe: EOF. (MWV-7724) - Workaround
-
Reinstall the ESXi operating system. |
- Problem
-
The Current Alternate DNS Server field for the BMC does not work in the Bare Metal Orchestrator web user interface. (MWV-7748) - Workaround
-
Use the command line interface to update the alternate DNS server value. |
- Problem
-
The Bare Metal Orchestrator web user interface session timesout after all the CP nodes are powered off and then powered on. (MWV-7769) - Workaround
-
Execute the redis-slave pod which is running using the following command: kubectl exec -it -n mw-redis-system redis-slave-0 – /bin/sh Change the directory: cd to /data/appendonlydir Run the following command and exit: redis-check-aof --fix appendonly.aof.11.incr.aof Restart the redis-slave pod and the site-controller pod. |
- Problem
-
When you log in to Bare Metal Orchestrator web user interface, you may notice the following error message: {"code":400,"message":"must provide Authorization header with format `Bearer \{token} `","reason":""} (MWV-7833) - Workaround
-
After a successful Bare Metal Orchestrator installation, run the following command: bmo delete user Kyle |
- Problem
-
When you onboard a server immediately after a factory reset, the server onboarding fails. (MWV-7879) - Workaround
-
After the factory reset, wait at least five minutes before you onboard a server. |
- Problem
-
When you Repurpose a server in the Lifecycle Controller mode, and discover the same server in the Bare Metal Orchestrator, the device discovery fails with the following error message: failed to initialize BMC no sku-pack available. (MWV-8654) - Workaround
- Power on the server and discover the server again.
|
- Problem
-
In the unlikely event of a system error, the event router may restart. If the event router is restarted, events from the previous hour prior to the restart are processed again. This may cause duplicate event notifications. (MWV-9230) - Workaround
- There is currently no workaroud.
|
- Problem
-
The Ubuntu operating system installation stops without any error message. (MWV-9387) - Workaround
- Check the OS version mentioned in the ubuntu.yaml file. If the OS version is not updated or is incorrect, update the value to correct version, and create the media object again before re-installing the operating system.
|
- Problem
-
When you select incorrect attributes for any user CRUD operations in the Bare Metal Orchestrator web user interface, the appropriate error messages are not displayed. (MWV-9803) - Workaround
- Ensure that you select the correct attributes for the user CRUD operations.
|
- Problem
-
When you create a new hardware profile immediately after editing an existing hardware profile in the Bare Metal Orchestrator web user interface, you may see a warning message: This profile cannot be edited because it is currently in use by servers. (MWV-9881) - Workaround
- Refresh the Bare Metal Orchestrator web user interface screen and create a new hardware profile.
|
- Problem
-
An operating system installation using a hardware profile fails if you use the sample YAML files in the samples folder of your Bare Metal Orchestrator instance that have the vendor: dell selector defined. The YAML files are listed below. - hw-pf-exci-install-bios-mode
- hw-pf-ubuntu-hpe-install
- hw-pf-ubuntu-install
- hw-pf-ubuntu-install-bonds
(MWV-9887) - Workaround
- Remove the vendor: dell selector attribute from the sample hardware profiles before using or duplicating them. Alternatively, you can add a matching vendor: dell label to the servers to which you want the hardware profile to apply.
|
- Problem
-
Bare Metal Orchestrator stores up to a maximum of 150 MB (or approximately 150,000 pages) of events for up to 90 days, where one event is approximately one MB (size varies depending on the event content). When the maximum database capacity is reached, new events overwrite the oldest ones in the database. For larger operations that can produce more than 150,000 events, not all events related to the operation are retained. (MWV-11969) - Workaround
- There is currently no workaround.
|
- Problem
-
On a server where a drift detection prescan is performed, you can not remove the hardware profile applied to the server using the Web User Interface. (MWV-13097) - Workaround
- Edit the server using the CLI. Remove the profile label and set "scan" to "none" in the audit section. You can only perform this action on one server at a time.
|
- Problem
-
Deploying the Red Hat Linux Enterprise operating system fails on one or more servers. This may be accompanied by a "failed to get Job Object" error message in the Bare Metal Orchestrator logs. (MWV-13196) - Workaround
-
This can happen under certain conditions or hardware configurations. There is currently no workaround. |
- Problem
-
iLO firmware version-2.7.1 experiences an occasional delay of storage-controller discovery during the discovery phase (after reboot, for example). This issue has a cascading effect on Bare Metal Orchestrator RAID CRUD operations when using HPE iLO and can lead to storage inventory collection occasionally being a false-negative. The result is that an onboarded server may display as being in a Failed state. (MWV-16306) - Workaround
-
Onboard the server again to manually re-trigger inventory collection. For more information, see the Hewlett Packard Enterprise Support Center. |
- Problem
-
If installedOsConfig details are provided and PasswordAuthentication is set as no, then server onboarding fails. - Workaround
- There is currently no workaround.
|
- Problem
-
During high-load periods that consume a large amount of system resources, such as concurrent server creation (approximately 10 servers per second), a request can return an error due to a cache lock failure. The error message is "admission webhook "vserver.kb.io" denied the request: redsync: failed to acquire lock". - Workaround
- Retry the operation when there is less load on the system.
|
- Problem
-
When you create RAID volumes on a server using a PERC RAID controller, and configure RAID conversion of storage drives simultaneously in a server YAML file, the Bare Metal Orchestrator might fail to convert some drives as specified in the server YAML file. (TPDE-2382) - Workaround
- You must first configure the RAID conversion of storage drives on the server and onboard the server. Edit the server YAML file to create RAID volumes.
|
- Problem
-
If the Bare Metal Orchestrator pods are turned off due to a power outage or a reboot, running Bare Metal Orchestrator commands result in a Redis error. (TPDE-2427) - Workaround
-
Contact Dell Customer Support. |
- Problem
-
Attempting to uninstall a high availability (HA) Bare Metal Orchestrator cluster that is not in a fully functional state fails to remove the /var/lib/kubelet directory. (TPDE-2450) - Workaround
-
Reboot the Global controller (CP1) node, and the two redundant HA nodes (CP2, and CP3). When the Bare Metal Orchestrator cluster recovers, try the uninstall procedure again. Alternatively, if you have a recent snapshot of your Bare Metal Orchestrator cluster, restore the cluster from the snapshot, and then try the uninstall procedure again. |
- Problem
-
In the OS Network Settings, the host name, gateway and IP address fields should not be mandatory. (TPDE-2480) - Workaround
-
Add a space to skip the mandatory field. |
- Problem
-
Bare Metal Orchestrator installation is getting stuck frequently. (TPDE-2534) - Workaround
-
Restart Bare Metal Orchestrator uninstallation. |
- Problem
-
Some pods do not get restarted after certificate renewal. This causes some pods to use old certificates while other pods use the new certificates. Hence, the authority of the certificate cannot be verified. (TPDE-2623) - Workaround
-
Restart all pods that require renewed certificates for communication. |
- Problem
-
Installing Ubuntu 18.04 LTS or Ubuntu 20.04 LTS can fail on HPE servers if the RAID volume is created and deleted on the HPE server. This can cause the RAID volume to appear as separate disks instead of appearing as a boot option. So when Bare Metal Orchestrator inventories the server, an improper RAID volume ID of 1 gets populated in the kickstart file. (TPDE-2628) - Workaround
-
Before installing the operating system on an on-boarded HPE server, use the get server command or the Bare Metal Orchestrator web interface to check that the correct RAID volume storage details are present and ensure the storage volume ID does not equal 1. |
- Problem
-
For 16G servers, when upgrading from ESXi 7.0 update 1 to 7.0 update 2 with BOSS N1 controller, the OS upgrade is taking longer than expected time. (TPDE-2631) - Workaround
-
None (the upgrade succeeds without any issue.) |
- Problem
-
Using Bare Metal Orchestrator to decommission a Dell server could fail due to a secure erase failure. (TPDE-2639) - Workaround
-
There is currently no workaround. Contact your Dell Support representative if you encounter this issue. |
- Problem
-
The Bare Metal Orchestrator logs record a status of Rejected instead of Completed after successfully running certain CLI commands, such as bmo get backups . (TPDE-2641) - Workaround
-
There is currently no workaround. |
- Problem
-
After using the Bare Metal Orchestrator web UI to change the backup location from the public URL to an S3 URL, the CLI still displays the public URL. (TPDE-2647) - Workaround
-
Use Backup Location field in the Bare Metal Orchestrator web user interface always to view the backup location. |
- Problem
-
Red Hat Enterprise Linux OS installation fails on 16G Dell PowerEdge R760 rack server with Dell HBA355i Fnt (Embedded) storage controller. (TPDE-2650) - Workaround
-
None (Red Hat Enterprise Linux OS installation works fine with HBA355i Adaptor.) |
- Problem
-
When you do a bulk discovery of more than 1000 servers using a CSV file in an high availability (HA) environment, you might see an admission webhook error message in the Bare Metal Orchestrator web user interface. (TPDE-2678) - Workaround
-
You can ignore the error message since the servers will be onboarded as expected. |
- Problem
-
The CLI commands to renew one or more certificates results in the following error, "Failed to parse arg. Reason : Arguments does not contain name". (TPDE-2688) - Workaround
- In the CLI, when renewing a certificate for a single resource, run the following command:
bmo renew cert <name> -c <name> -n <namespace> In the CLI, when renewing all certificates for all resources, run the following command: bmo renew certs all -c all -n all |
- Problem
-
When you onboard more than 75,000 servers in the Bare Metal Orchestrator, the backup operation may fail sometimes due to out of memory error. (TPDE-2737) - Workaround
-
Contact Dell Support and request to update the memory limits in Velero. |
- Problem
-
When you delete all the servers in a site and delete the site using Bare Metal Orchestrator CLI, the Bare Metal Orchestrator may fail to delete the site. (TPDE-2771) - Workaround
-
Contact your Dell Support representative if you encounter this issue. |
- Problem
-
When you apply a Hardware profile, the Bare Metal Orchestrator fails to validate the existence of a firmware media file. The related error message is not shown in the Bare Metal Orchestrator web user interface. (TPDE-2837) - Workaround
-
Before applying the hardware profile, ensure that the attributes are updated correctly and all the firmware media files are uploaded to the web server. |
- Problem
-
When uninstalling Bare Metal Orchestrator, the remote sites fail because of PersistentVolumeClaims (PVCs) that are not removed from the remote worker nodes during the uninstall procedure. (TPDE-2881) - Workaround
-
After the uninstall, run the following command on all nodes in cluster: rm -rf /opt/local-path-provisioner/* |
- Problem
-
Unable to perform any operation on the server due to the server being stuck in the busy state. (TPDE-2891, TPDE-2871) - Workaround
- Change the server state to failed, reinitialize the server to make it ready, or delete it and onboard it again.
- In the CLI, to change the server state, edit the server YAML file, add the attribute resetBusyState under labels as shown below and set it to true.
apiVersion: mw.dell.com/v3 kind: Server metadata: name: dell21 namespace: metalweaver labels: LC: common model: r740 site: gc vendor: dell resetBusyState: "true" spec: # Add fields here bmcEndPoint: "https://<BMC-IP>" userName: root password: <REPLACE_THIS> ... For information about server tasks, see the Bare Metal Orchestrator Command Line Interface Guide. - If you are using APIs, send a server patch request to add the resetBusyState attribute as shown below:
curl --location --request PATCH 'https://<cluster-ip>/api/v3/tenant/{tenant_name}/resources/servers/{servername}' \ --header 'Content-Type: application/json+patch' \ --header 'Accept: application/json+patch' \ --header 'Authorization: Bearer $token --data '[ { "op": "add", "value": "true", "path": "/metadata/labels/resetBusyState" } ]' For more information about server APIs, see the Bare Metal Orchestrator API Guide available on the Developer portal. |
- Problem
-
When backing up the Bare Metal Orchestrator cluster using the Bare Metal Orchestrator web user interface, the status field does not display the number of items. (TPDE-2912) - Workaround
-
There is currently no work around. |
- Problem
-
After creating a ETCD backup, when you run the describe backup command kubectl describe backup envd -n velero, the backup creation fails. (TPDE-2949) - Workaround
-
Run the following command: kubectl describe backups.velero.io envd -n velero |
- Problem
-
When you edit a hardware profile that is already applied on the servers in the Bare Metal Orchestrator web user interface, you might see the following warning: This profile cannot be edited because it is currently in use by servers . (TPDE-3001) - Workaround
- Edit the hardware profile in the Bare Metal Orchestrator CLI.
|
- Problem
-
Audit logs for maintenance mode are not generated in the OpenSearch dashboard. (TPDE-3058) - Workaround
-
Use the CLI to view the audit logs from the api-svc pod. Run the following command: kubectl logs -f -l app=mw-api-svc -n gc |
- Problem
-
When you deploy the VMware TKG stack in Bare Metal Orchestrator, the TKG cluster creation fails with the following error message: "rpc error: code = Unknown desc = error while getting token. unexpected status code 401" . (TPDE-3073) - Workaround
-
Deploy the TKG clusters using VMware Telco Cloud Automation (TCA) web user interface. |
- Problem
-
In Firmware Media Facade API, when the content type identifier in the request header is not 'application/json', instead of providing the 415 error code, it considers the content as 'application/json' and proceeds with the request. (TPDE-3092) - Workaround
-
None |
- Problem
-
When you list the contents of a specific folder in the web server using the bmo get fs CLI command or perform a firmware update, the operation fails with the following error message: "x509: invalid signature: parent certificate cannot sign this kind of certificate". (TPDE-3128) - Workaround
- Restart nginx-server and server-controller pods in the Global Controller.
|
- Problem
-
Devices are reported as non-compliant when the server contains more than one CPU. (TPDE-3129) - Workaround
-
Set TotalCores to the core count of the first CPU. |
- Problem
-
The Edit Hardware Profile window of the Bare Metal Orchestrator web user interface will show RAID Settings as configured by default. (TPDE-3132) - Workaround
- None
|
- Problem
-
The hardware profile fails to configure RAID storage on a server if that server has a combination of a PowerEdge H755 PERC controller and an NVMe controller installed. (TPDE-3964) - Workaround
-
Do not install a mix of PERC and NVMe controllers on the same server. |