We need from our cluster operating system to manage the hardware infrastructure and provide the interface to applications, consisting of a plethora of pre-processing, analytical, and machine learning frameworks and services. Kubernetes custom resources, and operators allow this conveniently and natively.
The most notable and demanded services are:
A few dozen other purposely built libraries and frameworks can be easily added to the system using Docker deployment options — Docker® images.
Intel offers libraries and frameworks with software optimizations to deliver faster AI training and inference with their world-class Intel Xeon Scalable processors. These include TensorFlow, MxNet, PaddlePaddle, Caffe and PyTorch.
Integrating these into enterprise a data science and machine learning solution enables users to deploy a reliable, powerful solution with better performance quickly. The Intel distribution for Python for example, accelerates AI-related Python libraries such as XGBoost, NumPy, SciPy, and scikit-learn with integrated Intel Performance Libraries such as Intel MKL to deliver faster AI training and inference.