Installing VMware vSphere ESXi 7.0 on the Dell EMC R6515 servers
We followed the steps below to install VMware vSphere® ESXi 7.0 on four Dell EMC PowerEdge R6515 servers onto which we deployed a VMware vSAN™ cluster. We also installed ESXi 7.0 Update 1 on the four Dell EMC PowerEdge R6515 servers for to upgrade the OS. Our testbed also included an infrastructure server that hosted our VMware vCenter®, Windows jump system, and control system.
Changing System Profile to Performance
- Boot the server.
- At the POST menu, select F2 for System Setup.
- Select System BIOS.
- Select System Profile Settings.
- Click the System Profile drop-down menu, and select Performance.
- Click Back.
- Click Finish.
- Click Finish again, and reboot the server.
Installing vSphere ESXi 7.0
- Attach the installation media.
- Boot the server.
- At the POST menu, select F11 for Boot Manager.
- Select One-shot UEFI Boot Menu.
- Select the installation media.
- At the VMware Installer screen, press Enter.
- At the EULA screen, press F11 to Accept and Continue.
- Under Storage Devices, select the appropriate virtual disk, and press Enter.
- Select US as the keyboard layout, and press Enter.
- Enter the root password twice, and press Enter.
- To start the installation, press F11.
- After the server reboots, press F2, and enter root credentials.
- Select Configure Management Network, and press Enter.
- Select the appropriate network adapter, and click OK.
- Select IPv4 settings and enter the desired IP address, subnet mask, and gateway for the server.
- Select OK, and restart the management network.
- Repeat steps 1 through 16 on the rest of the servers.
Installing vSphere ESXi 7.0 Update 1
- Attach the installation media.
- Boot the server.
- At the POST menu, select F11 for Boot Manager.
- Select One-shot UEFI Boot Menu.
- Select the installation media.
- At the VMware Installer screen, press Enter.
- At the EULA screen, press F11 to Accept and Continue.
- Under Storage Devices, select the appropriate virtual disk, and press Enter.
- Verify that Upgrade ESXi, preserve VMFS datastore is selected, and press Enter.
- Select US as the keyboard layout, and press Enter.
- Enter the root password twice, and press Enter.
- Press F11 to start the installation.
- After the server reboots, press F2, and enter root credentials.
- Repeat steps 1 through 13 on the other three servers.
Deploying vCenter 7.0 for SUTs
- On a Windows machine or VM, locate the VMware-VCSA installer image.
- Mount the image, navigate to the vcsa-ui-installer folder, and double-click win32.
- Double-click installer.exe
- Click Install.
- Click Next.
- Accept the terms of the license agreement, and click Next.
- Leave the default “vCenter Server with an Embedded Platform Services Controller” selected, and click Next.
- Enter the FQDN or IP address of the host onto which you will deploy the vCenter Server Appliance.
- Provide the servers credentials, and click Next.
- At the Configure Network Settings page, configure the network settings for your environment, and click Next.
- Review your settings, and click Finish.
- When the deployment completes, click Next.
- At the Introduction page, click Next.
- At the Appliance configuration page, select the time synchronization mode and SSH access settings, and click Next. We used a local NTP server and enabled SSH.
- Select Create a new SSO domain.
- Provide a password, and confirm it.
- Provide an SSO Domain name and SSO Site name, and click Next.
- At the CEIP page, click Next.
- At the Ready to complete page, click Finish.
- When the installation completes, click Close.
- Using the vSphere web client, log into the vCenter server using the credentials you previously provided.
Creating a cluster in vCenter and adding hosts
Creating the vSAN cluster
- Once you have logged into the vCenter, navigate to Hosts and Clusters.
- Select the primary site management vCenter.
- Right-click the vCenter object, and select New Datacenter…
- Enter a name for the new data center, and click OK.
- Right-click the new data center, and click New Cluster…
- Enter vSAN as the name for the new cluster.
- Click OK.
Adding the hosts to the vSAN cluster
- Once you have created the cluster, right-click the cluster, and click Add Host.
- Enter the FQDN or IP address of the first host, and click Next.
- Enter the root credentials for the server, and click Next.
- To accept the server’s certificate, click Yes.
- Review the server details, and click Next.
- Assign the desired license, and click Next.
- Disable Lockdown mode, and click Next.
- Click Finish.
- Repeat steps 1 through 8 for the remaining vSAN hosts and the client host.
Configuring the vSAN cluster
Configuring the vSAN networking
- Log into the vCenter web client, and navigate to Hosts and Clusters.
- Expand the primary site management vCenter, and select the client host.
- Click the Configure tab.
- Select VMkernel adapters, and click Add Networking.
- Select Virtual Machine Port Group for a Standard Switch, and click Next.
- Select New standard switch, change MTU (Bytes) to 9000, and click Next.
- Click Add adapters, and select your desired physical adapter. We used a 25GbE connection.
- Choose a network label, and click Next. We labeled our client traffic network Priv_Net.
- Click Finish.
- In the Hosts and Clusters pane, expand the vSAN cluster.
- Select the first host, and navigate to the Configure tab.
- Under Networking, select VMkernel adapters, and click Add Networking.
- Select VMkernel Network Adapter, and click Next.
- Select New standard switch, change MTU (Bytes) to 9000, and click Next.
- Click Add adapters, and select your desired physical adapter. We used a 25GbE connection.
- Type vSAN for the Network label, input your VLAN ID (if you’re using one), and select vSAN from the Available services below. Click Next.
- Choose your network settings, and click Next.
- Click Finish.
- Click Add Networking.
- Select VMkernel Network Adapter, and click Next.
- Choose Select an existing standard switch, browse to the vSwitch you created in step 14, click OK, and click Next.
- Type vMotion for the Network label, input your VLAN ID (if you’re using one), and select vMotion from the Available services below. Click Next.
- Choose your network settings, and click Next.
- Click Finish.
- Click Add Networking.
- Select Virtual Machine Port Group for a Standard Switch, and click Next.
- Choose Select an existing standard switch, browse to the vSwitch you created in step 14, click OK, and click Next.
- Choose a network label, and click Next. The label should match the client traffic network label you created in step 8. We used the label Priv_Net.
- Click Finish.
- Repeat steps 11 through 29 for the remaining hosts in the vSAN cluster.
Turning on the vSAN service
- Log into the vCenter web client, and navigate to Hosts and Clusters.
- Select the vSAN cluster, and click Configure.
- Under vSAN, click Services.
- Click Turn on VSAN.
Adding the MongoDB 10GbE network connection.
- Click the ESXi host.
- Click the Actions drop-down menu.
- Select Add Networking.
- For Select connection type, select Virtual Machine Port Group for a Standard Switch, and click Next.
- Select An existing standard switch, and click Browse…
- Select vSwitch1, click OK, and click Next.
- Under Network label, name it MongoDB.
- For VLAN ID, select None (0), and click Next.
- Click Finish.
Configuring the MongoDB Server VMs
Installing CentOS Linux 8
- In vCenter Server, right-click ESXi host, and select New Virtual Machine.
- Select Create a new virtual machine, and click Next.
- On the Virtual machine name line, give the machine a name (for example: Mongo-01a).
- Assign to the appropriate data center, and click Next.
- Assign to the appropriate ESXi host, and click Next.
- Assign to the correct datastore, and click Next.
- Select compatibility, and click Next.
- Select Linux, select CentOS 8 (64-bit) for the guest OS information, and click Next.
- On the Customize hardware section, add/change the following:
- CPU: 16
- Memory: 64
- Hard Drive: 30GB
- New Network: VM Network
- New Network: MongoDB
- New CD/DVD Drive: Datastore ISO File (CentOS-8.3.2011-x86_64-boot.iso)
- Click Next, and click Finish.
- Power on, and boot to the CentOS Linux 8 installation media using CentOS-8.3.2011-x86_64-boot.iso.
- Select Install CentOS Linux 8.
- Choose English, and click Continue.
- Under Installation Destination, select the desired disk to install the OS.
- Under Storage Configuration, select Custom, and click Done.
- Select Click here to create them automatically.
- Remove the /home partition if it exists.
- Expand the swap partition to 4GB.
- Assign all remaining free space to the / partition.
- Click Done.
- Click Accept Changes.
- Select Kdump.
- Uncheck Enable kdump, and click Done.
- Select Network & Hostname.
- Enter the desired hostname for the system.
- Turn on the desired network ports, and click Configure.
- On the General tab, select Connect automatically with priority.
- On the IPv4 Settings tab, click the Method drop-down menu, and select Manual.
- Under Addresses, click Add, and enter the desired static IP information for the server.
- Enter the desired DNS information.
- Click Save, and click Done.
- Select Date & Time, and ensure the correct date, time, and time zone are set.
- To add your NTP server, click the cog next to the Network Time On/Off switch.
- Add the IP address of your NTP server, and click +.
- Uncheck all other NTP servers.
- Click OK.
- Click Done.
- Click Software Selection.
- Choose the Base Environment of Server with GUI.
- Click Done.
- Click Begin Installation.
- Select Root Password.
- Enter the desired root password, and click Done.
- When the installation completes, select Reboot to restart the server.
- Click Licensing Information.
- Check I accept the license agreement, and click Done.
- Click Finish Configuration.
- At the Welcome screen, click Next.
- At the Privacy screen, click Next.
- At the Online Accounts screen, click Skip.
- At the About You screen, add a user, and click Next.
- At the Password screen, add a password, and click Next.
- Click Start Using CentOS Linux.
- Log onto the server GUI as the user you created during the setup.
- Close the Getting Started window.
- Reboot the system.
Configuring CentOS Linux 8
- Log into the server GUI as the user you created during the setup.
- Open a terminal window.
- Issue the su command to log in as root, and enter the root password:
su root - Disable the firewall by typing the following commands:
systemctl stop firewalld
systemctl disable firewalld - Edit the SELinux config file by using the vi text editor:
vi /etc/selinux/config
SELINUX=permissive - Save the SELinux config file by pressing the Esc key, and typing the following into the vi text editor:
:wq! - Install the EPEL Repository by typing the following into a terminal window:
yum install epel-release -y - Update the host:
yum update -y - Power off the host.
Installing MongoDB
- Log into vCenter Server and assign a second disk to the VM that is 100 GB in size.
- Power on the VM.
- Log into the VM.
- To disable Transparent HugePages within a virtual guest machine, create a new tuned config file by typing the following commands:
mkdir /etc/tuned/no-thp
vi /etc/tuned/no-thp/tuned.conf - With the blank tuned.conf file open, write the following into the file, and save:
[main]
include=virtual-guest
[vm]
transparent_hugepages=never - To disable Transparent HugePages on the OS, follow the procedure on MongoDB’s website:
https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/ - Add all host information for all systems into the /etc/hosts file.
- Reboot the server.
- Create the mogodb-org.repo file.
touch /etc/yum.repos.d/mongodb-org.repo - Populate the mongodb-org.repo file:
vim /etc/yum.repos.d/mongodb-org.repo - Add the following:
[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/8Server/mongodb-org/4.4/x86_64/ gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc: - Install MongoDB:
yum install -y mongodb-org - Make mongod start with the operating system by typing the following command:
echo "mongod --config=/etc/mongod.conf" >> /etc/rc.d/rc.local - Add the database drive to the VM, and create the database folders for MongoDB by typing the following commands:
mkfs.xfs /dev/sdb mkdir /data
mount /dev/sdb /data
echo '/dev/sdb /data xfs defaults 0 0' >> /etc/fstab mkdir -p /data/db1
chown mongod /data/db1 chgrp mongod /data/db1 - Open the mongod configuration file by typing the following command:
Vi /etc/mongod.conf - Inside the mongod configuration file, make the following changes:
storage:
dbPath: /data/db1
journal:
enabled: true
engine: "wiredTiger"
net:
port: 27017
bindIp: 192.168.X.X #Enter the IP address for the 10Gb NIC IP Address - Save the mongod configuration file.
- Start your mongod server by typing the following command:
mongod --config=/etc/mongod.confI - Reboot the server, and verify that the daemons load properly:
systemctl status mongod - Update the system:
yum -y update - Power off the system.
- Clone the system 11 times using vCenter Server.
- Boot the new systems, and modify their IP address and hostnames to be unique and match the /etc/hosts file.
- Modify the /etc/mongod.conf file to reflect the correct IP address on the bindIp line.
Setting MongoDB VM NUMA nodes*
We enabled four NUMA nodes per server, and then we spread out the NUMA nodes to one per VM to balance the load.
- Log into vCenter Server.
- Select the MongoDB VM.
- Right-click, and select Edit Settings.
- Click the VM Options tab.
- Expand Advanced, and click EDIT CONFIGURATION.
- Click the ADD CONFIGURATION PARAMS button.
- Under Name, type numa.nodeAffinity
- Under Value, depending on which NUMA node you’re assigning, enter 0, 1, 2, or 3.
- Click OK, and click OK again.
- Repeat steps 2 through 9 for each SUT and MongoDB VM, balancing them across SUTs and VMs.
*Using nodeaffinity in a production environment should be done with caution and can lead to unexpected side effects. Please see the VMware Resource Management Guide – Potential Issues with CPU Affinity for details. https://docs.vmware.com/en/VMware-vSphere/7.0/ com.vmware.vsphere.resmgmt.doc/GUID-199F1ADD-3B9B-490C-8F0C-440B53577B3A.html.
Installing ESXi 7.0 on the four YCSB infrastructure servers
We followed the steps below to install ESXi 7.0 on four YCSB infrastructure servers. We also installed ESXi 7.0 Update 1 on the four YCSB infrastructure servers to upgrade the OS.
Change System Profile to Performance
- Boot the server.
- At the POST menu, select F2 for System Setup.
- Select System BIOS.
- Select System Profile Settings.
- Click the System Profile drop-down menu, and select Performance.
- Click Back.
- Click Finish.
- Click Finish again, and reboot the server.
Installing vSphere ESXi 7.0
- Attach the installation media.
- Boot the server.
- At the POST menu, select F11 for Boot Manager.
- Select One-shot UEFI Boot Menu.
- Select the installation media.
- At the VMware Installer screen, press Enter.
- At the EULA screen, press F11 to Accept and Continue.
- Under Storage Devices, select the appropriate virtual disk, and press Enter.
- Select US as the keyboard layout, and press Enter.
- Enter the root password twice, and press Enter.
- To start the installation, press F11.
- After the server reboots, press F2, and enter root credentials.
- Select Configure Management Network, and press Enter.
- Select the appropriate network adapter, and click OK.
- Select IPv4 settings and enter the desired IP address, subnet mask, and gateway for the server.
- Select OK and restart the management network.
- Repeat steps 1 through 16 on the rest of the servers.
Installing vSphere ESXi 7.0 Update 1
- Attach the installation media.
- Boot the server.
- At the POST menu, select F11 for Boot Manager.
- Select One-shot UEFI Boot Menu.
- Select the installation media.
- At the VMware Installer screen, press Enter.
- At the EULA screen, press F11 to Accept and Continue.
- Under Storage Devices, select the appropriate virtual disk, and press Enter.
- Verify that Upgrade ESXi, preserve VMFS datastore is selected, and press Enter.
- Select US as the keyboard layout, and press Enter.
- Enter the root password twice, and press Enter.
- Press F11 to start the installation.
- After the server reboots, press F2, and enter root credentials.
- Repeat steps 1 through 13 on the rest of the servers.
Adding the MongoDB 10GbE network connection
- Click the ESXi host.
- Click the Actions drop-down menu.
- Select Add Networking.
- For Select connection type, select Virtual Machine Port Group for a Standard Switch, and click Next.
- Select An existing standard switch, and click Browse…
- Select vSwitch1, click OK, then click Next.
- Under Network label, name it MongoDB.
- For VLAN ID select None (0), and click Next.
- Click Finish.
Installing and configuring the YCSB driver VMs
We installed CentOS Linux 8 onto each VM, and then installed YCSB on each driver VM. We created 16 YCSB VMs per infrastructure server, for a total of 64 VMs. We created the MongoDB database using YCSB.
Installing CentOS Linux 8
- In vCenter Server, right-click ESXi host, and select New Virtual Machine.
- Select Create a new virtual machine, and click Next.
- On the Virtual machine name line, give the machine a name (for example: YCSB-01).
- Assign to the appropriate data center, and click Next.
- Assign to the appropriate ESXi host, and click Next.
- Assign to the correct datastore, and click Next.
- Select compatibility, and click Next.
- Select Linux, select CentOS 8 (64-bit) for the guest OS information, and click Next.
- On the Customize hardware section, add/change the following:
- CPU: 16
- Memory: 64
- Hard Drive: 30GB
- New Network: VM Network
- New Network: MongoDB
- New CD/DVD Drive: Datastore ISO File (CentOS-8.3.2011-x86_64-boot.iso)
- Click Next, and click Finish.
- Power on, and boot to the CentOS Linux 8 installation media using CentOS-8.3.2011-x86_64-boot.iso.
- Select Install CentOS Linux 8.
- Choose English, and click Continue.
- Under Installation Destination, select the desired disk to install the OS.
- Under Storage Configuration, select Custom, and click Done.
- Select Click here to create them automatically.
- Remove the /home partition if it exists.
- Expand the swap partition to 4GB.
- Assign all remaining free space to the / partition.
- Click Done.
- Click Accept Changes.
- Select Kdump.
- Uncheck Enable kdump, and click Done.
- Select Network & Hostname.
- Enter the desired hostname for the system.
- Turn on the desired network ports, and click Configure.
- On the General tab, select Connect automatically with priority.
- On the IPv4 Settings tab, click the Method drop-down menu, and select Manual.
- Under Addresses, click Add, and enter the desired static IP information for the server.
- Enter the desired DNS information.
- Click Save, and click Done.
- Select Date & Time, and ensure the correct date, time, and time zone are set.
- To add your NTP server, click the cog next to the Network Time On/Off switch.
- Add the IP address of your NTP server, and click +.
- Uncheck all other NTP servers.
- Click OK.
- Click Done.
- Click Software Selection.
- Choose the Base Environment of Server with GUI.
- Click Done.
- Click Begin Installation.
- Select Root Password.
- Enter the desired root password, and click Done.
- When the installation completes, select Reboot to restart the server.
- Click Licensing Information.
- Check I accept the license agreement, and click Done.
- Click Finish Configuration.
- At the Welcome screen, click Next.
- At the Privacy screen, click Next.
- At the Online Accounts screen, click Next.
- At the About You screen, add a user, and click Next.
- At the Password screen, add a password, and click Next.
- Click Start Using CentOS Linux.
- Log onto the server GUI as the user you created during the setup.
- Close the Getting Started window.
- Reboot the system.
Configuring CentOS Linux 8
- Log onto the server GUI as the user you created during the setup.
- Open a terminal window.
- Issue the su command to log in as root, and enter the root password:
su root - Disable the firewall by typing the following commands:
systemctl stop firewalld
systemctl disable firewalld - Edit the SELinux config file by using the vi text editor:
vi /etc/selinux/config
SELINUX=permissive - Save the SELinux config file by typing pressing the Esc key, and typing the following into the vi text:
:wq! - Install the EPEL Repository by typing the following into a terminal window:
yum install epel-release -y - Update the host:
yum update -y - Power off the host.
Installing YCSB
- Power on the VM.
- Log into the VM.
- Install Java-devel, wget, and curl by issuing the following command:
yum -y install java-devel wget curl - Install Maven with the following commands:
wget " http://apache.cs.utah.edu/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz "
sudo tar xzf apache-maven-*-bin.tar.gz -C /usr/local
cd /usr/local
sudo ln -s apache-maven-* maven
echo export M2_HOME=/usr/local/maven >> /etc/profile.d/maven.sh
echo export PATH=${M2_HOME}/bin:${PATH} >> /etc/profile.d/maven.sh - Download and unpack YCSB into your driver with the following commands:
curl -O --location https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar. gz
tar xfvz ycsb-0.17.0.tar.gz - Repeat steps 1 to 5 on the remaining 63 YCSB driver VMs.
Creating the MongoDB database using YCSB
- Log into the YCSB driver VM.
- Run the following command to create a roughly 32GB database for MongoDB:
Note: Use the fully qualified path to the binaries and config files. For <mongodb-server>, add the 10Gb IP address for the MongoDB server.
<install-dir>ycsb-0.17.0/bin/ycsb load mongodb -s -P <install-dir>ycsb-0.17.0/workloads/workloada
-threads 64 -p mongodb.url=mongodb://<mongodb-server>:27017/ycsb?w=1 -p recordcount= 27000000 - Repeat steps 1 and 2 to create the database on the other 15 MongoDB server VMs, changing the IP address for each.
Running the YCSB benchmark
We created a central control VM to launch tests simultaneously across the 64 YCSB drivers. We created two scripts: one that prepared the environment for testing and one that started the workloads. We describe their function in the following two sections.
Preparing for the test
The prepare script logged into each of the 16 MongoDB server VMs and issued the following commands:
blockdev --setra 512 /dev/sdb
sync; echo 3 > /proc/sys/vm/drop_caches
systemctl stop mongod
systemctl start mongod
The prepare script logged into each of the 64 YCSB driver VMs and issued the following command:
sync; echo 3 > /proc/sys/vm/drop_caches
Running the test
We performed tests with 4 YCSB driver VMs running workloads against each MongoDB server VM simultaneously. We tuned each YCSB client driver VM to run 64 threads, and they performed 30,000,000 operations per thread.
The run script logged into each of the 64 YCSB driver VMs and issued the following commands, where <mongodb-server> would be set appropriately to balance across the environment:
<install-dir>/ycsb-0.17.0/bin/ycsb run mongodb-async -s -P <install-dir>/ycsb-0.17.0/ workloads/workloadc -threads 64 -p mongodb.url=mongodb://<mongodb-server>:27017/ycsb?w=1 -p operationcount=30000000 -p mongodb.upsert=truesystemctl stop mongod
Upon completion, the script gathers logs into a central folder for analysis. We recorded these results.
This project was commissioned by Dell Technologies.