.. _cluster-installation: Guide to Typical Cluster Installation --------------------------------------------------------------- This section is all about deploying FindFace Enterprise Server in a cluster environment. .. tip:: If after having read this section, you still have questions, do not hesitate to contact our experts by support@ntechlab.com. The reasons for deploying FindFace Enterprise Server in a cluster are the following: * The necessity to distribute the video processing high load. * The necessity to process video streams from a group of cameras in the place of their physical location. .. note:: The most common use cases where such need comes to the fore are hotel chains, chain stores, several security checkpoints in the same building, etc. * The necessity to distribute the biometric sample extraction high load. * A large number of faces to search through that requires the implementation of a distributed face database. Before you start the deployment, outline your system architecture, depending on its load and allotted resources (see :ref:`requirements-system`). The most common distributed scheme is as follows: * One principal server with the following components: ``findface-ntls``, ``findface-sf-api``, ``findface-video-manager``, ``findface-upload``, ``findface-video-worker``, ``findface-extraction-api``, ``findface-tarantool-server``, and third-parties. * Several additional video processing servers with installed ``findface-video-worker``. * (If needed) Several additional biometric servers with installed ``findface-extraction-api``. * (If needed) Additional database servers with multiple Tarantool shards. This section describes the most common distributed deployment. In high load systems, it may also be necessary to distribute the API processing (``findface-sf-api`` and ``findface-video-manager``) across several additional servers. In this case, refer to :ref:`custom-installation`. To deploy FindFace Enterprise Server in a cluster environment, follow the steps below: .. contents:: :local: Deploy Principal Server ^^^^^^^^^^^^^^^^^^^^^^^^^^ To deploy the principal server as part of a distributed architecture, do the following: #. On the designated physical server, :ref:`install ` FindFace Enterprise Server from the console installer as follows: * Product to install: ``FindFace Server``. * Installation type: ``Single server, multiple video workers``. In this case, FindFace Enterprise Server will be installed and configured to interact with additional remote ``findface-video-worker`` instances. * Type of the ``findface-video-worker`` acceleration (on the principal server): CPU or GPU, subject to your hardware configuration. * Type of the ``findface-extraction-api`` acceleration (on the principal server): CPU or GPU, subject to your hardware configuration. After the installation is complete, the following output will be shown on the console: .. tip:: Be sure to save this data: you will need it later. .. code:: ############################################################################# # Installation is complete # ############################################################################# - upload your license to http://127.0.0.1:3185/ - FindFace SF-API address: http://172.20.77.78:18411/ - FindFace VideoManager address: http://172.20.77.78:18411/ #. Upload the FindFace Enterprise Server license file via the ``findface-ntls`` web interface ``http://:3185``. .. note:: The IP address is shown in the links to the FindFace web services in the following way: as an external IP address if the host belongs to a network, or ``127.0.0.1`` otherwise. #. Allow the licensable services to access the ``findface-ntls`` license server from any IP address, To do so, open the ``/etc/findface-ntls.cfg`` configuration file and set ``listen = 0.0.0.0:3133``. .. code:: sudo vi /etc/findface-ntls.cfg # Listen address of NTLS server where services will connect to. # The format is IP:PORT # Use 0.0.0.0:PORT to listen on all interfaces # This parameter is mandatory and may occur multiple times # if you need to listen on several specific interfaces or ports. listen = 0.0.0.1:3133 Deploy Video Processing Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On an additional video processing server, install only a ``findface-video-worker`` instance following the :ref:`step-by-step instructions `. Answer the installer questions as follows: * Product to install: FindFace Video Worker. * Type of the ``findface-video-worker`` acceleration: CPU or GPU, subject to your hardware configuration. * FindFace Enterprise Server IP address: IP address of the principal server. After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``FindFace Video Worker`` on other hosts without having to answer the questions again, by executing: .. code:: sudo ./.run -f /tmp/.json .. note:: After the installation, specify the ``findface-ntls`` and/or ``findface-video-manager`` IP addresses in the ``findface-video-worker`` configuration file. .. code:: sudo vi /etc/findface-video-worker-cpu.ini sudo vi /etc/findface-video-worker-gpu.ini In the ``ntls-addr`` parameter, specify the ``findface-ntls`` host IP address. .. code:: ntls-addr=127.0.0.1:3133 In the ``mgr-static`` parameter, specify the ``findface-video-manager`` host IP address, which provides ``findface-video-worker`` with settings and the video stream list. .. code:: mgr-static=127.0.0.1:18811 Deploy Biometric Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On an additional biometric server, install only a ``findface-extraction-api`` instance from the console installer. Answer the installer questions as follows: * Product to install: ``FindFace Server``. * Installation type: ``Fully customized installation``. * FindFace Enterprise Server components to install: ``findface-extraction-api`` and ``findface-data``. To make a selection, first, deselect all the listed components by entering ``-*`` in the command line. Select ``findface-extraction-api`` and ``findface-data`` by entering their sequence number (keyword): ``1 7``. Enter ``done`` to save your selection and proceed to another step. * Type of ``findface-extraction-api`` acceleration: CPU or GPU. * Modification of the ``findface-extraction-api`` configuration file: specify the IP address of the ``findface-ntls`` server. * Neural network models to install: CPU or GPU model for face biometrics (mandatory), and (optional) CPU/GPU models for gender, age, emotions, glasses and/or beard recognition. To make a selection, first, deselect all the listed models by entering ``-*`` in the command line. Select required models by entering their sequence number (keyword), for example, ``8 2`` to select the GPU-models for biometric sample extraction and age recognition. Enter ``done`` to save your selection and proceed to another step. Be sure to choose the right acceleration type for each model, matching the acceleration type of ``findface-extraction-api``: CPU or GPU. Be aware that ``findface-extraction-api`` on CPU can work only with CPU-models, while ``findface-extraction-api`` on GPU supports both CPU- and GPU-models. See :ref:`face-features` for details. The following models are available: +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | Face feature | Acceleration | Package | +==================+==============+=======================================================================================================+ | face | CPU | findface-data-grapefruit-160-cpu_3.0.0_amd64.deb, findface-data-grapefruit-480-cpu_3.0.0_amd64.deb | | (biometry) +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-grapefruit-160-gpu_3.0.0_amd64.deb, findface-data-grapefruit-480-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | age | CPU | findface-data-age.v1-cpu_3.0.0_amd64.deb | | +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-age.v1-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | gender | CPU | findface-data-gender.v2-cpu_3.0.0_amd64.deb | | +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-gender.v2-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | emotions | CPU | findface-data-emotions.v1-cpu_3.0.0_amd64.deb | | +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-emotions.v1-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | glasses3 | CPU | findface-data-glasses3.v0-cpu_3.0.0_amd64.deb | | +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-glasses3.v0-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ | beard | CPU | findface-data-beard.v0-cpu_3.0.0_amd64.deb | | +--------------+-------------------------------------------------------------------------------------------------------+ | | GPU | findface-data-beard.v0-gpu_3.0.0_amd64.deb | +------------------+--------------+-------------------------------------------------------------------------------------------------------+ After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``findface-extraction-api`` on other hosts without having to answer the questions again. .. code:: sudo ./.run -f /tmp/.json After all the biometric servers are deployed, distribute load across them by using a :ref:`load balancer `. .. _load-balancing: Distribute Load across Biometric Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To distribute load across several biometric servers, you need to set up load balancing. The following step-by-step instructions demonstrate how to set up NGINX load balancing in a round-robin fashion for 3 ``findface-extraction-api`` instances located on different physical hosts: one on the FindFace Enterprise Server principal server (``172.168.1.9``), and 2 on additional remote servers (``172.168.1.10``, ``172.168.1.11``). Should you have more biometric servers in your system, load-balance them by analogy. .. tip:: You can use any load balancer according to your preference. Please refer to the relevant official documentation for guidance. To set up load balancing, do the following: #. Designate the FindFace Enterprise Server principal server (recommended) or any other server with NGINX as a gateway to all the biometric servers. .. important:: You will have to specify the gateway server IP address when configuring the FindFace Enterprise Server :ref:`network `. .. tip:: You can install NGINX as such: .. code:: sudo apt update sudo apt install nginx #. On the gateway server, create a new NGINX configuration file. .. code:: sudo vi /etc/nginx/sites-available/extapi #. Insert the following entry into the newly created configuration file. In the ``upstream`` directive (``upstream extapibackends``), substitute the exemplary IP addresses with the actual IP addresses of the biometric servers. In the ``server`` directive, specify the gateway server listening port as ``listen``. You will have to enter this port when configuring the FindFace Enterprise Server :ref:`network `. .. code:: upstream extapibackends { server 172.168.1.9:18666; ## ``findface-extraction-api`` on principal server server 172.168.1.10:18666; ## 1st additional extraction server server 127.168.1.11:18666; ## 2nd additional extraction server } server { listen 18667; server_name extapi; client_max_body_size 64m; location / { proxy_pass http://extapibackends; proxy_next_upstream error; } access_log /var/log/nginx/extapi.access_log; error_log /var/log/nginx/extapi.error_log; } #. Enable the load balancer in NGINX. .. code:: sudo ln -s /etc/nginx/sites-available/extapi /etc/nginx/sites-enabled/ #. Restart nginx. .. code:: sudo service nginx restart #. On the principal server and each additional biometric server, open the ``/etc/findface-extraction-api.ini`` configuration file. Substitute localhost in the ``listen`` parameter with the relevant server address that you have specified in ``upstream extapibackends`` (``/etc/nginx/sites-available/extapi``) before. In our example, the address of the 1st additional extraction server has to be substituted as such: .. code:: sudo vi /etc/findface-extraction-api.ini listen: 172.168.1.10:18666 #. Restart the ``findface-extraction-api`` on the principal server and each additional biometric server. .. code:: sudo systemctl restart findface-extraction-api.service The load balancing is now successfully set up. Be sure to specify the actual gateway server IP address and listening port, when configuring the FindFace Enterprise Server :ref:`network `. Distribute Database ^^^^^^^^^^^^^^^^^^^^^^^^^ The ``findface-tarantool-server`` component connects the Tarantool database and the ``findface-sf-api`` component, transferring search results from the database to ``findface-sf-api`` for further processing. To increase search speed, multiple ``findface-tarantool-server`` shards can be created on each Tarantool host. Their running concurrently leads to a remarkable increase in performance. Each shard can handle up to approximately 10,000,000 faces. When deploying ``findface-tarantool-server`` from the console installer, shards are created automatically given the server hardware. To distribute the face database, install only a ``findface-tarantool-server`` instance on each additional database server. Answer the installer questions as follows: * Product to install: ``FindFace Server``. * Installation type: ``Fully customized installation``. * FindFace Enterprise Server components to install: ``findface-tarantool-server``. To make a selection, first, deselect all the listed components by entering ``-*`` in the command line. Select ``findface-tarantool-server`` by entering its sequence number (keyword): ``13``. Enter ``done`` to save your selection and proceed to another step. After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``findface-tarantool-server`` on other hosts without having to answer the questions again. .. code:: sudo ./.run -f /tmp/.json As a result of the installation, ``findface-tarantool-server`` shards will be automatically installed in the amount of ``N = max(min(mem_mb // 2000, cpu_cores), 1)``, i.e., equal to the RAM size in MB divided by 2000, or the number of CPU physical cores (but at least 1 shard). Be sure to specify the shards IP addresses and ports, when configuring the FindFace Enterprise Server :ref:`network `. To learn the port numbers, execute on each database server: .. code:: sudo cat /etc/tarantool/instances.enabled/*shard* | grep -E ".start|(listen =)"` You will get the following result: .. code:: listen = '127.0.0.1:33001', FindFace.start("127.0.0.1", 8101, { listen = '127.0.0.1:33002', FindFace.start("127.0.0.1", 8102, { You can find the port number of a shard in the ``FindFace.start`` section, for example, ``8101``, ``8102``, etc. .. _network: Configure Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^ After all the FindFace Enterprise Server components are deployed, configure their interaction over the network. Do the following: #. Open the ``findface-sf-api`` configuration file: .. code:: sudo vi /etc/findface-sf-api.ini Specify the following parameters: +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | Parameter | Description | +=============================================+==========================================================================================================+ | ``extraction-api`` -> ``extraction-api`` | IP address and listening port of the :ref:`gateway biometric server ` with set up | | | load balancing. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | ``storage-api`` -> ``shards`` -> ``master`` | IP address and port of the ``findface-tarantool-server`` master shard. Specify each shard by analogy. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | ``upload_url`` | WebDAV NGINX path to send original images, thumbnails and normalized face images to the | | | ``findface-upload`` service. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ .. code:: ... extraction-api: extraction-api: http://172.168.1.9:18667 ... webdav: upload-url: http://127.0.0.1:3333/uploads/ ... storage-api: ... shards: - master: http://172.168.1.9:8101/v2/ slave: '' - master: http://172.168.1.9:8102/v2/ slave: '' - master: http://172.168.1.12:8101/v2/ slave: '' - master: http://172.168.1.12:8102/v2/ slave: '' - master: http://172.168.1.13:8102/v2/ slave: '' - master: http://172.168.1.13:8102/v2/ slave: '' #. Open the ``findface-facerouter`` configuration file. Specify the IP address of the ``findface-sf-api`` host. .. code:: sudo vi /etc/findface-facerouter.py sfapi_url = 'http://localhost:18411' #. Open the ``findface-video-manager`` configuration file. In the ``router_url`` parameter, specify the IP address and port of the ``findface-facerouter`` host to receive detected faces from ``findface-video-worker``. .. code:: sudo vi /etc/findface-video-manager.conf ... router_url: http://127.0.0.1:18820/v0/frame The FindFace Enterprise Server components interaction is now set up.