.. _cluster-installation: Guide to Typical Cluster Installation --------------------------------------------------------------- This section is all about deploying FindFace Security in a cluster environment. .. tip:: If after having read this section, you still have questions, do not hesitate to contact our experts by support@ntechlab.com. The reasons for deploying FindFace Security in a cluster are the following: * Necessity to distribute the video processing high load. * Necessity to process video streams from a group of cameras in the place of their physical location. .. note:: The most common use cases where such need comes to the fore are hotel chains, chain stores, several security checkpoints in the same building, etc. .. seealso:: :ref:`video-allocation` * Necessity to distribute the biometric sample extraction high load. * Large number of faces to search through, that requires implementation of a distributed face database. Before you start the deployment, outline your system architecture, depending on its load and allotted resources (see :ref:`requirements`). The most common distributed scheme is as follows: * One principal server with the following components: ``findface-ntls``, ``findface-security``, ``findface-sf-api``, ``findface-video-manager``, ``findface-upload``, ``findface-video-worker``, ``findface-extraction-api``, ``findface-tarantool-server``, and third-parties. * Several additional video processing servers with installed ``findface-video-worker``. * (If needed) Several additional biometric servers with installed ``findface-extraction-api``. * (If needed) Additional database servers with multiple Tarantool shards. This section describes the most common distributed deployment. In high load systems, it may also be necessary to distribute the API processing (``findface-sf-api`` and ``findface-video-manager``) across several additional servers. In this case, refer to :ref:`custom-installation`. To deploy FindFace Security in a cluster environment, follow the steps below: .. contents:: :local: Deploy Principal Server ^^^^^^^^^^^^^^^^^^^^^^^^^^ To deploy the principal server as part of a distributed architecture, do the following: #. On the designated physical server, :ref:`install ` FindFace Security from installer as follows: * Product to install: ``FindFace Security``. * Installation type: ``Single server, multiple video workers``. In this case, FindFace Security will be installed and configured to interact with additional remote ``findface-video-worker`` instances. * Type of the ``findface-video-worker`` acceleration (on the principal server): CPU or GPU, subject to your hardware configuration. * Type of the ``findface-extraction-api`` acceleration (on the principal server): CPU or GPU, subject to your hardware configuration. After the installation is complete, the following output will be shown on the console: .. code:: ############################################################################# # Installation is complete # ############################################################################# - upload your license to http://172.20.77.17/#/license/ - user interface: http://172.20.77.17/ superuser: admin password: admin documentation: http://172.20.77.17/doc/ #. Upload the FindFace Security license file via the main web interface ``http:///#/license``. To access the web interface, use the provided ``admin`` credentials. .. note:: The host IP address is shown in the links to FindFace web services in the following way: as an external IP address if the host belongs to a network, or ``127.0.0.1`` otherwise. .. important:: Do not disclose the superuser (Super Administrator) credentials to others. To administer the system, create a new user with the administrator privileges. Whatever the role, Super Administrator cannot be deprived of its rights. #. Allow the licensable services to access the ``findface-ntls`` license server from any IP address, To do so, open the ``/etc/findface-ntls.cfg`` configuration file and set ``listen = 0.0.0.0:3133``. .. code:: sudo vi /etc/findface-ntls.cfg ## Address to accept incoming client connections (IP:PORT) ## type:string env:CFG_LISTEN longopt:--listen listen = 0.0.0.1:3133 Deploy Video Processing Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On an additional video processing server, install only a ``findface-video-worker`` instance following the :ref:`step-by-step instructions `. Answer the installer questions as follows: * Product to install: FindFace Video Worker. * Type of the ``findface-video-worker`` acceleration: CPU or GPU, subject to your hardware configuration. * FindFace Security IP address: IP address of the principal server. After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``FindFace Video Worker`` on other hosts without having to answer the questions again, by executing: .. code:: sudo ./findface-security-and-server-4.4.run -f /tmp/.json .. note:: If ``findface-ntls`` and/or ``findface-video-manager`` are installed on a different host than that with ``findface-security``, specify their IP addresses in the ``findface-video-worker`` configuration file after the installation. .. code:: sudo vi /etc/findface-video-worker-cpu.ini sudo vi /etc/findface-video-worker-gpu.ini In the ``ntls-addr`` parameter, specify the ``findface-ntls`` host IP address. .. code:: ntls-addr=127.0.0.1:3133 In the ``mgr-static`` parameter, specify the ``findface-video-manager`` host IP address, which provides ``findface-video-worker`` with settings and the video stream list. .. code:: mgr-static=127.0.0.1:18811 Deploy Biometric Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ On an additional biometric server, install only a ``findface-extraction-api`` instance from the console installer. Answer the installer questions as follows: * Product to install: ``FindFace Security``. * Installation type: ``Fully customized installation``. * FindFace Security components to install: ``findface-extraction-api`` and ``findface-data``. To make a selection, first deselect all the listed components by entering ``-*`` in the command line, then select ``findface-extraction-api`` and ``findface-data`` by entering their sequence number (keyword): ``1 7``. Enter ``done`` to save your selection and proceed to another step. * Type of ``findface-extraction-api`` acceleration: CPU or GPU. * Modification of the ``findface-extraction-api`` configuration file: specify the IP address of the ``findface-ntls`` server. * Neural network models to install: CPU or GPU model for face biometrics (mandatory), and (optional) CPU/GPU models for gender, age, emotions, glasses, beard, and face mask recognition. To make a selection, first deselect all the listed models by entering ``-*`` in the command line, then select required models by entering their sequence number (keyword), for example, ``8 2`` to select the GPU-models for biometric sample extraction and age recognition. Enter ``done`` to save your selection and proceed to another step. Be sure to choose the right acceleration type for each model, matching the acceleration type of ``findface-extraction-api``: CPU or GPU. Be aware that ``findface-extraction-api`` on CPU can work only with CPU-models, while ``findface-extraction-api`` on GPU supports both CPU- and GPU-models. See :ref:`face-features` for details. The following models are available: +------------------+--------------+------------------------------------------------+ | Face feature | Acceleration | Package | +==================+==============+================================================+ | face | CPU | findface-data-ifruit-320-cpu_3.0.0_all.deb | | (biometry) | | findface-data-ifruit-160-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-ifruit-320-gpu_3.0.0_all.deb | | | | findface-data-ifruit-160-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | age | CPU | findface-data-age.v1-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-age.v1-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | gender | CPU | findface-data-gender.v2-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-gender.v2-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | emotions | CPU | findface-data-emotions.v1-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-emotions.v1-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | glasses3 | CPU | findface-data-glasses3.v0-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-glasses3.v0-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | beard | CPU | findface-data-beard.v0-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-beard.v0-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ | face mask | CPU | findface-data-medmask3.v2-cpu_3.0.0_all.deb | | +--------------+------------------------------------------------+ | | GPU | findface-data-medmask3.v2-gpu_3.0.0_all.deb | +------------------+--------------+------------------------------------------------+ After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``findface-extraction-api`` on other hosts without having to answer the questions again. .. code:: sudo ./findface-security-and-server-4.4.run -f /tmp/.json After all the biometric servers are deployed, distribute load across them by using a :ref:`load balancer `. .. _load-balancing: Distribute Load across Biometric Servers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To distribute load across several biometric servers, you need to set up load balancing. The following step-by-step instructions demonstrate how to set up ``nginx`` load balancing in a round-robin fashion for 3 ``findface-extraction-api`` instances located on different physical hosts: one on the FindFace Security principal server (``172.168.1.9``), and 2 on additional remote servers (``172.168.1.10``, ``172.168.1.11``). Should you have more biometric servers in your system, load-balance them by analogy. .. tip:: You can use any load balancer according to your preference. Please refer to the relevant official documentation for guidance. To set up load balancing, do the following: #. Designate the FindFace Security principal server (recommended) or any other server with nginx as a gateway to all the biometric servers. .. important:: You will have to specify the gateway server IP address when configuring the FindFace Security :ref:`network `. .. tip:: You can install nginx as such: .. code:: sudo apt update sudo apt install nginx #. On the gateway server, create a new nginx configuration file. .. code:: sudo vi /etc/nginx/sites-available/extapi #. Insert the following entry into the newly created configuration file. In the ``upstream`` directive (``upstream extapibackends``), substitute the exemplary IP addresses with the actual IP addresses of the biometric servers. In the ``server`` directive, specify the gateway server listening port as ``listen``. You will have to enter this port when configuring the FindFace Security :ref:`network `. .. code:: upstream extapibackends { server 172.168.1.9:18666; ## ``findface-extraction-api`` on principal server server 172.168.1.10:18666; ## 1st additional extraction server server 127.168.1.11:18666; ## 2nd additional extraction server } server { listen 18667; server_name extapi; client_max_body_size 64m; location / { proxy_pass http://extapibackends; proxy_next_upstream error; } access_log /var/log/nginx/extapi.access_log; error_log /var/log/nginx/extapi.error_log; } #. Enable the load balancer in nginx. .. code:: sudo ln -s /etc/nginx/sites-available/extapi /etc/nginx/sites-enabled/ #. Restart nginx. .. code:: sudo service nginx restart #. On the principal server and each additional biometric server, open the ``/etc/findface-extraction-api.ini`` configuration file. Substitute localhost in the ``listen`` parameter with the relevant server address that you have specified in ``upstream extapibackends`` (``/etc/nginx/sites-available/extapi``) before. In our example, the address of the 1st additional extraction server has to be substituted as such: .. code:: sudo vi /etc/findface-extraction-api.ini listen: 172.168.1.10:18666 #. Restart the ``findface-extraction-api`` on the principal server and each additional biometric server. .. code:: sudo systemctl restart findface-extraction-api.service The load balancing is now successfully set up. Be sure to specify the actual gateway server IP address and listening port, when configuring the FindFace Security :ref:`network `. Distribute Database ^^^^^^^^^^^^^^^^^^^^^^^^^ The ``findface-tarantool-server`` component connects the Tarantool database and the ``findface-sf-api`` component, transferring search results from the database to ``findface-sf-api`` for further processing. To increase search speed, multiple ``findface-tarantool-server`` shards can be created on each Tarantool host. Their running concurrently leads to a remarkable increase in performance. Each shard can handle up to approximately 10,000,000 faces. When deploying ``findface-tarantool-server`` from installer, shards are created automatically given the server hardware. To distribute the face database, install only a ``findface-tarantool-server`` instance on each additional database server. Answer the installer questions as follows: * Product to install: ``FindFace Security``. * Installation type: ``Fully customized installation``. * FindFace Security components to install: ``findface-tarantool-server``. To make a selection, first deselect all the listed components by entering ``-*`` in the command line, then select ``findface-tarantool-server`` by entering its sequence number (keyword): ``13``. Enter ``done`` to save your selection and proceed to another step. After that, the installation process will automatically begin. The answers will be saved to a file ``/tmp/.json``. Use this file to install ``findface-tarantool-server`` on other hosts without having to answer the questions again. .. code:: sudo ./findface-security-and-server-4.4.run -f /tmp/.json As a result of the installation, ``findface-tarantool-server`` shards will be automatically installed in the amount of ``N = min(max(min(mem_mb // 2000, cpu_cores), 1), 16 * cpu_cores)``. I.e., it is equal to the RAM size in MB divided by 2000, or the number of CPU physical cores (but at least one shard), or the number of CPU physical cores multiplied by 16, should the first obtained value be greater. Be sure to specify the shards IP addresses and ports, when configuring the FindFace Security :ref:`network `. To learn the port numbers, execute on each database server: .. code:: sudo cat /etc/tarantool/instances.enabled/*shard* | grep -E ".start|(listen =)"` You will get the following result: .. code:: listen = '127.0.0.1:33001', FindFace.start("127.0.0.1", 8101, { listen = '127.0.0.1:33002', FindFace.start("127.0.0.1", 8102, { You can find the port number of a shard in the ``FindFace.start`` section, for example, ``8101``, ``8102``, etc. .. _network: Configure Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^ After all the FindFace Security components are deployed, configure their interaction over the network. Do the following: #. Open the ``/etc/findface-sf-api.ini`` configuration file: .. code:: sudo vi /etc/findface-sf-api.ini Specify the following parameters: +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | Parameter | Description | +=============================================+==========================================================================================================+ | ``extraction-api`` -> ``extraction-api`` | IP address and listening port of the :ref:`gateway biometric server ` with set up | | | load balancing. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | ``storage-api`` -> ``shards`` -> ``master`` | IP address and port of the ``findface-tarantool-server`` master shard. Specify each shard by analogy. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ | ``upload_url`` | WebDAV NginX path to send original images, thumbnails and normalized face images to the | | | ``findface-upload`` service. | +---------------------------------------------+----------------------------------------------------------------------------------------------------------+ .. code:: ... extraction-api: extraction-api: http://172.168.1.9:18667 ... webdav: upload-url: http://127.0.0.1:3333/uploads/ ... storage-api: ... shards: - master: http://172.168.1.9:8101/v2/ slave: '' - master: http://172.168.1.9:8102/v2/ slave: '' - master: http://172.168.1.12:8101/v2/ slave: '' - master: http://172.168.1.12:8102/v2/ slave: '' - master: http://172.168.1.13:8102/v2/ slave: '' - master: http://172.168.1.13:8102/v2/ slave: '' #. Open the ``/etc/findface-security/config.py`` configuration file. .. code:: sudo vi /etc/findface-security/config.py Specify the following parameters: +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | Parameter | Description | +===================================+==========================================================================================================+ | ``SERVICE_EXTERNAL_ADDRESS`` | FindFace Security IP address or URL prioritized for the Genetec integration and webhooks. | | | Once this parameter not specified, the system uses ``EXTERNAL_ADDRESS`` for these purposes. | | | To use Genetec and webhooks, be sure to specify at least one of those parameters: | | | ``SERVICE_EXTERNAL_ADDRESS``, ``EXTERNAL_ADDRESS``. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``EXTERNAL_ADDRESS`` | (Optional) IP address or URL that can be used to access the FindFace Security web interface. | | | Once this parameter not specified, the system auto-detects it as the external IP address. To access | | | FindFace Security, you can use both the auto-detected and specified IP addresses. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``VIDEO_DETECTOR_TOKEN`` | To authorize the video face detection module, come up with a token and specify it here. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``VIDEO_MANAGER_ADDRESS`` | IP address of the ``findface-video-manager`` host. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``NTLS_HTTP_URL`` | IP address of the ``findface-ntls`` host. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``ROUTER_URL`` | External IP address of the ``findface-security`` host that will receive detected faces from the | | | ``findface-video-worker`` instance(s). | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``SF_API_ADDRESS`` | IP address of the ``findface-sf-api`` host. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ | ``EXTRACTION_API`` | IP address and listening port of the :ref:`gateway biometric server ` with set up | | | load balancing. | +-----------------------------------+----------------------------------------------------------------------------------------------------------+ .. code:: sudo vi /etc/findface-security/config.py ... # SERVICE_EXTERNAL_ADDRESS prioritized for webhooks and genetec SERVICE_EXTERNAL_ADDRESS = 'http://localhost' EXTERNAL_ADDRESS = 'http://127.0.0.1' ... FFSECURITY = { 'VIDEO_DETECTOR_TOKEN': '7ce2679adfc4d74edcf508bea4d67208', ... 'EXTRACTION_API': 'http://172.168.1.9:18667/', 'VIDEO_MANAGER_ADDRESS': 'http://127.0.0.1:18810', ... 'NTLS_HTTP_URL': 'http://127.0.0.1:3185', 'ROUTER_URL': 'http://172.168.1.9', ... 'SF_API_ADDRESS': 'http://127.0.0.1:18411', ... } The FindFace Security components interaction is now set up. .. important:: To preserve the FindFace Security compatibility with the installation environment, we highly recommend you to disable the Ubuntu automatic update. In this case, you will be able to update your OS manually, fully controlling which packages to update. To disable the Ubuntu automatic update, execute the following commands: .. code:: sudo apt-get remove unattended-upgrades sudo systemctl stop apt-daily.timer sudo systemctl disable apt-daily.timer sudo systemctl disable apt-daily.service sudo systemctl daemon-reload