Home > Storage > ObjectScale and ECS > Product Documentation > Dell ECS with NGINX (OpenResty) > Installation and setup
Components configured in this example to employ ECS with OpenResty include a server to host OpenResty running Ubuntu 16.04 LTS, OpenResty (source, NGINX core, libraries, LuaJIT, and additional modules), Domain Name System (DNS) and an ECS U300 appliance with ECS 3.0HF1 installed. Ubuntu and Windows client servers were used to validate OpenResty with ECS deployment for both object and file access. S3 Browser for Windows and Linux curl was used to validate the setup of OpenResty with ECS for object access and a regular Linux mount command was used to test NFS. Figure 5 illustrates the components in this example.
A physical server (bare metal) or a virtual machine can be used for deployment of OpenResty. The server should be sized (CPU, memory, network cards, and so on) based on the following criteria:
A UNIX or Windows operating system is also installed on the server or virtual machine. See the specific operating system website to get minimum server requirements. OpenResty can also be encapsulated using container technology such as Docker for flexibility and automation in deployment.
As mentioned, OpenResty is an open-source web platform that bundles NGINX core, LuaJIT, and other modules and libraries contributed by the developer community. For this example, certain versions of the OpenResty components and a build (compilation) from the sources are required in order to use third-party modules for ECS health checks, monitoring, and metrics collection. Table 2 provides the components, versions and download locations of OpenResty, modules, and Linux libraries and tools used in this sample deployment.
Type | Components | Download Location |
OpenResty | OpenResty 1.11.2.2 | https://openresty.org/download/openresty-1.11.2.2.tar.gz |
NGINX 1.11.5 | http://nginx.org/download/nginx-1.11.5.tar.gz | |
LUARocks 2.3.0 | http://luarocks.org/releases/luarocks-2.3.0.tar.gz | |
OpenSSL 1.0.2j | https://www.openssl.org/source/openssl-1.0.2j.targ.gz | |
PCRE 8.39 | https://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre- 8.39.tar.gz | |
3rd Party Modules for Monitoring | https://github.com/vozlt/nginx-module-sts/archive/master.zip | |
https://github.com/vozlt/nginx-module-stream-sts/archive/master.zip | ||
https://github.com/vozlt/nginx-module-vts/archive/master.zip | ||
3rd Party Modules for Health Checks | https://github.com/lusis/ngx_stream_upstream_check_module/archive /master.zip | |
Libraries and Tools | build-essential | Installed from Ubuntu Repositories (apt-get install) |
Lua Scripts | prometheus.lua | Developed by the Dell DevOps team and attached to this white paper (nginx-sample.zip) |
The basic steps to install OpenResty include:
Several libraries and tools are required to setup and build OpenResty. Figure 6 provides a sample command to install these libraries and tools.
# apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
libgd-dev \
libgeoip-dev \
libncurses5-dev \
libperl-dev \
libreadline-dev \
libxslt1-dev \
make \
perl \
unzip \
zlib1g-dev \
iproute \
iptables \
wget \
apt-utils
OpenResty has limited built in modules related to health checks, monitoring, and metric collection (read latency, write latency, and so on). Several modules from the Github repositories are available to provide this additional functionality and include:
Figure 7 shows sample commands to download and unzip these modules. The normal directory location to place these modules is in “/usr/local/src”.
# cd /usr/local/src
# wget -O nginx-module-sts-master.zip https://github.com/vozlt/nginx-module- sts/archive/master.zip
# unzip nginx-module-sts-master.zip
# wget -O nginx-module-stream-sts-master.zip https://github.com/vozlt/nginx-module-stream- sts/archive/master.zip
# unzip nginx-module-stream-sts-master.zip
# wget -O nginx-module-vts-master.zip https://github.com/vozlt/nginx-module- vts/archive/master.zip
# unzip nginx-module-vts-master.zip
# unzip ngx_stream_upstream_check_module.zip
# ls -l *.zip
-rw-r--r-- 1 root root 30266 Jun 1 10:49 nginx-module-stream-sts-master.zip
-rw-r--r-- 1 root root 357012 Jun 1 10:51 nginx-module-sts-master.zip
-rw-r--r-- 1 root root 396273 Jun 1 10:29 nginx-module-vts-master.zip
-rw-r--r-- 1 root root 154616 Jun 1 10:29 ngx_stream_upstream_check_module.zip
Additional definitions are needed for the ngx_stream_upstream_check_module-master module so a patch is executed to modify the modules with these new definitions. This patch is attached to this white paper. Figure 8 shows an example of how this patch is applied and output when applied.
# cd ngx_stream_upstream_check_module-master
# patch -p1 < /tmp/stream_upstream_module.patch
patching file ngx_stream_upstream_check_module.c
patching file ngx_stream_upstream_check_module.h
The OpenResty bundled sources in the main website does not include certain versions of NGINX, Lua, PCRE (Perl Compatible Regular Expression), and OpenSSL components that are needed for some of the Lua scripts and third party add-on modules. Figure 9 illustrates how to replace the bundled version of OpenResty with the necessary versions needed for this sample deployment.
# cd /tmp
# curl -kfSL https://www.openssl.org/source/openssl-1.0.2j.tar.gz -o openssl-1.0.2j.tar.gz
# tar xzf openssl-1.0.2j.tar.gz
# curl -kfSL https://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.39.tar.gz -o pcre-8.39.tar.gz
# tar xzf pcre-8.39.tar.gz
# curl -kfSL https://openresty.org/download/openresty-1.11.2.2.tar.gz -o openresty- 1.11.2.2.tar.gz
# tar xzf openresty-1.11.2.2.tar.gz
# curl -kfSL http://nginx.org/download/nginx-1.11.5.tar.gz -o nginx-1.11.5.tar.gz
# tar xzf nginx-1.11.5.tar.gz
Check for failures or errors during download and when unpacking the tarballs of the OpenResty components. To verify if you have downloaded correctly, Figure 10 has the sizes of the files after download. Depending on the version or release downloaded, these sizes may differ.
# ls -l /tmp/*.tar.gz
-rw-r--r-- 1 root root 956517 Jun 1 10:34 nginx-1.11.5.tar.gz
-rw-r--r-- 1 root root 4244348 Jun 1 10:34 openresty-1.11.2.2.tar.gz
-rw-r--r-- 1 root root 5307912 Jun 1 10:34 openssl-1.0.2j.tar.gz
-rw-r--r-- 1 root root 2062258 Jun 1 10:34 pcre-8.39.tar.gz
For the proper functioning of the ngx_stream_upstream_check_module, a patch needs to be applied to NGINX. Also, the nginx-1.11.5 version needs to be copied over to the OpenResty bundled directory. Because version 1.11.5 is used, NGINX 1.11.2 is removed from the bundle. Figure 11 provides the commands to do this.
# cd nginx-1.11.5/
# patch -p0 < /tmp/patch-1.11.x.patch
patching file src/stream/ngx_stream_upstream_hash_module.c Hunk #3 succeeded at 558 (offset 3 lines).
patching file src/stream/ngx_stream_upstream_least_conn_module.c Hunk #2 succeeded at 146 (offset -1 lines).
patching file src/stream/ngx_stream_upstream_round_robin.c Hunk #2 succeeded at 102 with fuzz 1 (offset 1 line).
Hunk #3 succeeded at 462 with fuzz 1 (offset 11 lines).
Hunk #4 succeeded at 562 (offset 4 lines).
patching file src/stream/ngx_stream_upstream_round_robin.h
# cd /tmp
# mv nginx-1.11.5 /tmp/openresty-1.11.2.2/bundle/
# rm -rf /tmp/openresty-1.11.2.2/bundle/nginx-1.11.2
Afterwards, the bundle should now include the nginx-1.11.5 as highlighted in Figure 12.
With all the sources, add-on modules, patches, and libraries downloaded and setup, the last step is to build and install the OpenResty binaries in their appropriate directories. Before a “make” is initiated to build OpenResty, a “configure” is performed to set the flags and include all modules that are part of the OpenResty binary. Figure 13 illustrates the configure command with the options, and the build and install commands. If an “Error” is encountered during configuration, check that all libraries and other dependencies in the previous steps have been installed in the correct directories. An appropriate error message is printed out to indicate the reason for the error. A successful “configure” outputs the following when completed:
Type the following commands to build and install:
make
make install
The “make” and “make install” commands should also not produce an Error message and an error message is in the output to provide some clue about the error. Pre-built packages are available and can be downloaded; however, the third party modules used in this example are not included.
# cd /tmp/openresty-1.11.2.2/
# ./configure -j1 --with-openssl=/tmp/openssl-1.0.2j --with-pcre=/tmp/pcre-8.39 --with-file-aio
--with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with- http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with- http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with- http_random_index_module --with-http_realip_module --with-http_secure_link_module --with- http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module
--with-http_v2_module --with-http_xslt_module=dynamic --with-ipv6 --with-mail --with- mail_ssl_module --with-md5-asm --with-pcre-jit --with-sha1-asm --with-stream --with- stream_ssl_module --with-threads --add-module=/usr/local/src/nginx-module-sts-master --add- module=/usr/local/src/nginx-module-stream-sts-master --add-module=/usr/local/src/nginx-module- vts-master --add-module=/usr/local/src/ngx_stream_upstream_check_module-master
In this example, a DNS is set up on a Windows 2012 server and accessible from the server hosting OpenResty. Table 3 shows the DNS entries created. Adding DNS entries allows mapping of “names” to IP addresses. In this example, DNS is used as a mechanism for translating the object protocol (S3 or Swift) the client is using and allows OpenResty to direct request to the appropriate pool of ECS nodes based on the protocol name. The names associated with each object protocol are mapped to one IP address associated with OpenResty and translate it to a pool of ECS nodes. An “A-record “ is created in DNS which maps a name to the IP address of the OpenResty server and CNAME provides an alias for each protocol.
DNS Record | Record Type | Record Data | Comments |
os.ecstme.org | A | 10.246.150.210 | NGINX external IP Address and also used for S3 protocol access (os=object store) |
*.os.ecstme.org | CNAME | os.ecstme.org | Used for S3 virtually hosted buckets, that is, mybucket.os.ecstme.org |
swift.ecstme.org | CNAME | os.ecstme.org | Endpoint for clients using the Swift protocol |
From the Windows Server, start up the DNS Manager and add DNS entries of “New Host” for A-Record and “New Alias” in your domain zone as described in Table 3. Sample screenshots of this appear in Figure 14 and Figure 15.