.. _architecture:

*************************************
Architecture
*************************************

Though you mostly interact with FindFace Multi through its web interface, be sure to take a minute to learn the FindFace Multi architecture. This knowledge is essential for the FindFace Multi deployment, integration, maintenance, and troubleshooting.

.. rubric:: In this chapter:

.. contents::
   :local:

Recognition Process. Objects of Recognition
====================================================

FindFace Multi can recognize the following objects and their attributes:

* human faces
* human bodies (silhouettes)
* vehicles

FindFace Multi detects an object in the photo or video and prepares its image through normalization. The normalized image is then used for extracting the object's feature vector (an n-dimensional vector of numerical features that represent the object). Object feature vectors are stored in the database and further used for verification and identification purposes.

.. _docker-platform:

Docker Platform
======================

FindFace Multi is deployed in Docker, a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. Each FindFace Multi service runs in a Docker container.


Architectural Elements
===============================

FindFace Multi consists of the following fundamental architectural elements:

* FindFace core, a cutting-edge AI-based recognition technology that can be used as a separate product `FindFace Server <https://docs.ntechlab.com/projects/ntech-lab-ffserver-core-en/en/12.240830.2/>`_. 
* FindFace Multi, which is a turnkey application module for FindFace Server.

The FindFace core internal architecture allows for embedding :ref:`architecture-vms`, an additional functionality that records, stores, and plays back video data from cameras.


Architecture scheme
------------------------

|architecture_en|

.. |architecture_en| image:: /_static/architecture_en.png

.. |architecture_ru| image:: /_static/architecture_ru.png


FindFace core
-------------------------

There is a separate product `FindFace Server <https://docs.ntechlab.com/projects/ntech-lab-ffserver-core-en/en/12.240830.2/>`_ inside the FindFace core. It includes the following components:

+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+
| Component                 | Ports in use | Description                                                                             | Vendor                                     |
+===========================+==============+=========================================================================================+============================================+
| findface-extraction-api   | 18666        | A service that uses neural networks to detect an object in an image and extract its     | NtechLab own deployment                    |
|                           |              | feature vector. It also recognizes object attributes                                    |                                            |
|                           |              | (for example, gender, age, emotions, beard, glasses, face mask - for face objects).     |                                            |
|                           |              | CPU- or GPU-acceleration. Read more about the :ref:`extraction-api-config` service.     |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-sf-api           | 18411        | A service that implements the internal HTTP API for object detection and recognition.   |                                            |
|                           |              | Read more about the :ref:`sf-api-config` service.                                       |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-tarantool-server | 32001,       | A service that provides interaction between the ``findface-sf-api`` service and the     |                                            |
|                           | shard ports  | feature vector database (the Tarantool-powered database that stores object              |                                            |
|                           | (default     | feature vectors. Read more about the :ref:`tarantool-server-config` service.            |                                            |
|                           | 330xx, 81xx) |                                                                                         |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-upload           | 3333         | NginX-based web server used as storage for original images, thumbnails, and             |                                            |
|                           |              | normalized object images. If :ref:`architecture-vms` is installed, ``findface-upload``  |                                            |
|                           |              | also stores video data from cameras. Read more about the :ref:`upload-config` service.  |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-facerouter       | 18820        | A service used to define processing directives for detected objects. In FindFace        |                                            |
|                           |              | Multi, its functions are performed by ``findface-multi-legacy``                         |                                            |
|                           |              | (see :ref:`application-module`).                                                        |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-video-manager    | 18810,       | A service, part of the video object detection module, that is used for managing the     |                                            |
|                           | 18811        | video object detection functionality, configuring the video object detector             |                                            |
|                           |              | settings and specifying the list of to-be-processed video streams. Read more about      |                                            |
|                           |              | the :ref:`findface-video-manager <video-config>` service.                               |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-video-worker     | 18999        | A service, part of the video object detection module, that recognizes an object in      |                                            |
|                           |              | the video and posts its normalized image, full frame, and metadata (such as             |                                            |
|                           |              | detection time) to the ``findface-facerouter`` service for further processing           |                                            |
|                           |              | according to given directives. If :ref:`Video Recorder <architecture-vms>` is           |                                            |
|                           |              | enabled, ``findface-video-worker`` sends video over to ``findface-video-storage``.      |                                            |
|                           |              | Provides :ref:`face liveness detection <liveness>` if enabled. CPU- or GPU-acceleration.|                                            |
|                           |              | Read more about the :ref:`findface-video-worker <video-config>` service.                |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-ntls             | 443          | A license server that interfaces with the NtechLab Global License Server, a USB         |                                            |
|                           | (TCP),       | dongle, or hardware fingerprint to verify the :ref:`license <licensing-principles>`     |                                            |
|                           | 3133, 3185   | of your FindFace Multi instance. Licensing via a proxy server is supported. Read more   |                                            |
|                           |              | about the :ref:`ntls-config` service.                                                   |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-deduplicator     | 18310        | The service compares feature vectors; used for deduplication of episodes and cluster    |                                            |
|                           |              | events.                                                                                 |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+                                            |
| findface-liveness-api     | 18301        | Besides the embedded functionality provided by ``findface-video-worker``, face          |                                            |
|                           |              | liveness detection can also be harnessed as a standalone service                        |                                            |
|                           |              | ``findface-liveness-api``. The service takes a specific number of frames from a         |                                            |
|                           |              | video chunk and returns the best quality face, and decimal liveness result averaged     |                                            |
|                           |              | across the taken frames. The service is also involved                                   |                                            |
|                           |              | in the course of user :ref:`authentication <authentication>` by face. CPU- or           |                                            |
|                           |              | GPU-decoding is supported.                                                              |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+
| Tarantool                 | Shard ports  | A third-party software that implements the feature vector database that stores          | `Tarantool <https://tarantool.io/en/>`_    |
|                           | (default     | extracted object feature vectors and identification events. The system data,            |                                            |
|                           | 330xx, 81xx) | records, user accounts, and camera settings are stored in PostgreSQL                    |                                            |
|                           |              | (part of the FindFace Multi application module).                                        |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+
| etcd                      | 2379         | A third-party software that implements a distributed key-value store for                | `etcd <https://github.com/etcd-io/etcd>`_  |
|                           |              | ``findface-video-manager``. Used as a coordination service in the distributed           |                                            |
|                           |              | system, providing the video object detector with fault tolerance.                       |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+
| NginX                     | 80;          | A third-party software that implements the system web interfaces.                       | `NginX <https://www.nginx.com>`_           |
|                           | SSL: 8002,   |                                                                                         |                                            |
|                           | 8003, 443,   |                                                                                         |                                            |
|                           | 80           |                                                                                         |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+
| Redis                     | 6379         | A third-party in-memory database that is used by the ``findface-sf-api`` service        | `Redis <https://github.com/redis/redis>`_  |
|                           |              | as a temporary storage for extracted object feature vectors before they are written     |                                            |
|                           |              | to the feature vector storage.                                                          |                                            |
+---------------------------+--------------+-----------------------------------------------------------------------------------------+--------------------------------------------+


.. _application-module:


FindFace Multi Application Module
---------------------------------------------

The FindFace Multi application module includes the following components:

+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| Component                                   | Ports in use     | Description                                                                               | Vendor                                            |
+=============================================+==================+===========================================================================================+===================================================+
| findface-multi-legacy                       | Configurable     | A service that works as a gateway to the FindFace core. Provides interaction              | NtechLab own deployment                           |
|                                             |                  | between the FindFace core and the web interface, the system functioning as a whole.       |                                                   |
|                                             |                  | Read more about the :ref:`findface-multi-legacy <legacy-config>` service.                 |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
|                                             |                  | Internal services that assist ``findface-multi-legacy``. The ``findface-multi-audit``     |                                                   |
| findface-multi-audit                        | 8012, 8013, 8014 | service is created for the future. It will become full-fledged in the upcoming            |                                                   |
+---------------------------------------------+------------------+ versions. As for now, it partly duplicates the ``findface-multi-legacy``                  |                                                   |
| findface-multi-identity-provider            | 8022, 8023, 8024 | functionality. Use ``findface-multi-legacy`` to work with it. The                         |                                                   |
+---------------------------------------------+------------------+ ``findface-multi-identity-provider`` is the service for authentication, user              |                                                   |
| findface-onvif-discovery                    | n/a              | management, and management of role-based model of accesses.                               |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-legacy-singleton-services    | n/a              | Runs single-instance services, such as synchronization of cameras with the                |                                                   |
|                                             |                  | ``findface-video-manager``, clustering, and other single-instance services.               |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-identity-provider-migrate    | n/a              | Services that run database migration, create galleries, and make other configuration.     |                                                   |
+---------------------------------------------+------------------+                                                                                           |                                                   |
| findface-multi-legacy-migrate               | n/a              |                                                                                           |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-healthcheck                  | n/a              | Checks containers' health on startup. Expects the containers to be ``healthy``            |                                                   |
|                                             |                  | before starting a UI.                                                                     |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-cleaner                      | n/a              | A part of the ``findface-multi-legacy`` functionality, detached into a separate service.  |                                                   |
|                                             |                  | It serves for the FindFace Multi data clean up (except for cleaning the video archives    |                                                   |
|                                             |                  | of the :ref:`architecture-vms`).                                                          |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-cleaner-vms                  | n/a              | A part of the ``findface-multi-legacy`` functionality, detached into a separate service.  |                                                   |
|                                             |                  | It is responsible for cleaning the video archives of the :ref:`architecture-vms`.         |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-file-mover                   | n/a              | A file moving service that is used for :ref:`hot_cold_storage`. The service transfers     |                                                   |
|                                             |                  | images from the short-term storage to the long-term storage.                              |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-ui                           | Configurable     | The main web interface used to interact with FindFace Multi. Based on the                 |                                                   |
|                                             |                  | `Django framework <https://www.djangoproject.com>`_. Allows you to work with object       |                                                   |
|                                             |                  | recognition events, search for objects, manage cameras, users, records, watch lists,      |                                                   |
|                                             |                  | collect real-time statistics, and much more. The user interface configuration is located  |                                                   |
|                                             |                  | in a separate :ref:`ui-config.yaml <legacy-config>` configuration file.                   |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-alerts                       | n/a              | An internal service that is responsible for                                               |                                                   |
|                                             |                  | the :ref:`alert creation and functioning <configure_alert>`.                              |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-alerts-migrate               | n/a              | The service is responsible for evolution of the database schema of                        |                                                   |
|                                             |                  | the ``findface-multi-alerts`` service.                                                    |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-replication-receiver         | 8014             | An internal service that is responsible for receiving events when FindFace Multi is       |                                                   |
|                                             |                  | functioning as a :ref:`Processing System <replication>`.                                  |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-replication-sender           | 8015             | An internal service that is responsible for the event sending via HTTPS when              |                                                   |
|                                             |                  | FindFace Multi is functioning as an :ref:`Originating System <replication>`.              |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-replication-receiver-migrate | n/a              | The service is responsible for evolution of the database schema of the Processing System. |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-replication-sender-migrate   | n/a              | The service is responsible for evolution of the database schema of the Originating System.|                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+                                                   |
| findface-multi-line-crossing-analytics      | n/a              | An internal service that processes FindFace Multi data for further use in analytical      |                                                   |
|                                             |                  | purposes. The analytics works for line crossing only at the moment.                       |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+   
| superset                                    | 8088             | Services that belong to the third-party open-source software application —                | `Apache Superset <https://superset.apache.org/>`_ |                               
+---------------------------------------------+                  | `Apache Superset <https://superset.apache.org/>`_. Within FindFace Multi, it can be       |                                                   |
| superset-init                               |                  | used as a BI system for the :ref:`analytical service <bi_analytics>`.                     |                                                   |
+---------------------------------------------+                  |                                                                                           |                                                   |                                                                              
| superset-worker                             |                  |                                                                                           |                                                   |
+---------------------------------------------+                  |                                                                                           |                                                   |
| superset-worker-beat                        |                  |                                                                                           |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| TimescaleDB                                 | 5432, 8008, 8081 | A third-party software. It’s an open-source time series database that extends PostgreSQL. | `TimescaleDB <https://www.tigerdata.com/>`_       |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| NATS                                        | 4222             | A third-party software that implements a message broker in ``findface-multi-legacy``.     | `NATS <https://nats.io>`_                         |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| NATS JetStream                              | 4222, 6222, 8222 | NATS with JetStream is a third-party software. Both ``nats`` and ``nats-jetstream``       |                                                   |       
|                                             |                  | perform the same function with the difference in message delivery guarantee.              |                                                   |
|                                             |                  | ``nats-jetstream`` has a higher message delivery guarantee and, compared to ``nats``,     |                                                   | 
|                                             |                  | does not lose messages even if the system or user is unreachable. ``nats-jestream`` is    |                                                   |
|                                             |                  | used where message loss is unacceptable. ``nats`` and ``nats-jestream`` are not           |                                                   |
|                                             |                  | interchangeable, and both are essential for the system functioning.                       |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| etcd                                        | 2379             | A third-party software that implements locks in the ``findface-multi-legacy`` service,    | `etcd <https://github.com/etcd-io/etcd>`_         |
|                                             |                  | e.g., locks in license checker, reports, video processing, object clusterization, etc.    |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| Pgbouncer                                   | 5439             | A third-party software, a lightweight connection pooler for PostgreSQL. Optional,         | `PgBouncer <https://www.pgbouncer.org>`_          |
|                                             |                  | used to increase the database performance under high load.                                |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| PostgreSQL                                  | 5432             | A third-party software that implements the main system database. This database stores     | `PostgreSQL <https://www.postgresql.org>`_        |
|                                             |                  | records of persons and vehicles and data for internal use, including user accounts        |                                                   |
|                                             |                  | and camera settings.                                                                      |                                                   |
|                                             |                  | The object feature vectors and object identification events are stored in Tarantool       |                                                   |
|                                             |                  | (part of the FindFace core).                                                              |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+
| RabbitMQ                                    | 4369, 5671-5672, | A third-party open-source message-broker software.                                        | `RabbitMQ <https://www.rabbitmq.com/>`_           |
|                                             | 15671-15672,     |                                                                                           |                                                   |
|                                             | 15691-15692,     |                                                                                           |                                                   |
|                                             | 25672            |                                                                                           |                                                   |
+---------------------------------------------+------------------+-------------------------------------------------------------------------------------------+---------------------------------------------------+



.. _architecture-vms:


Video Recorder
-------------------------------------------------------

Video Recorder is an optional part of the FindFace core. It includes the following services:

+-----------------------------+--------------+---------------------------------------------------------------------------------------+-----------------------------------------------+
| Component                   | Ports in use | Description                                                                           | Vendor                                        |
+=============================+==============+=======================================================================================+===============================================+
| findface-video-storage      | 18611        | A service that implements video chunk management. It takes video chunks from the      | NtechLab own deployment                       |
|                             |              | ``findface-video-worker`` service, puts them into the storage                         |                                               |
|                             |              | (``findface-upload``), and writes their meta-information and whereabouts to the Video |                                               |
|                             |              | Recorder database (MongoDB). By request from ``findface-multi-legacy``, it issues     |                                               |
|                             |              | info about existing video chunks in the form of Websocket-links to the corresponding  |                                               |
|                             |              | streams. These links are further used by ``findface-video-streamer-cpu`` to deliver   |                                               |
|                             |              | the video to a user for viewing and downloading.                                      |                                               |
+-----------------------------+--------------+---------------------------------------------------------------------------------------+                                               |
| findface-video-streamer     | 9000         | After receiving a ``findface-multi-ui`` request, this service uses Websocket to       |                                               |
|                             |              | extract requested video chunks from ``findface-video-storage`` and                    |                                               |
|                             |              | ``findface-video-worker`` (only the last chunk if it's not yet recorded to the        |                                               |
|                             |              | storage). Then it merges the video chunks into a one-piece video and delivers it to a |                                               |
|                             |              | user for viewing and downloading.                                                     |                                               |
+-----------------------------+--------------+---------------------------------------------------------------------------------------+-----------------------------------------------+
| MongoDB                     | 27017        | A third-party software that implements the Video Recorder database. The database      | `MongoDB <https://github.com/mongodb/mongo>`_ |
|                             |              | stores meta-information of the video chunks, including their location. The video      |                                               |
|                             |              | chunks themselves are stored in the ``findface-upload`` service.                      |                                               |
+-----------------------------+--------------+---------------------------------------------------------------------------------------+-----------------------------------------------+

To work with the :ref:`alarm monitoring service <alarm_monitor>`, FindFace Multi uses Annex. The list of the involved services is described :ref:`here <architecture-annex>`.


.. seealso::
   * :ref:`storages`
   * :ref:`custom_components`
   
Single- and Multi-Host Deployment
==================================

You can deploy FindFace Multi on a single host or in a multi-host environment. If you opt for the latter, we offer you one of the following deployment schemes:

* Deploy FindFace Multi on a principal host and distribute additional ``findface-video-worker`` instances across remote hosts.

  See :ref:`video-worker-installer`.

* Distribute the FindFace Multi services across multiple hosts. If necessary, set up load balancing. Contact our experts at support@ntechlab.com for guidance.

  See :ref:`multi-host-installation`.



CPU- and GPU-acceleration
==========================================

The ``findface-extraction-api``, ``findface-video-worker``, and ``findface-liveness-api`` services can be either CPU- or GPU-based. You will have an opportunity to choose the acceleration type you need during the installation.

.. important::
   Refer to :ref:`requirements` when choosing hardware configuration.

.. important::
   If the resolution of a camera(s) in use is more than 1280x720px, it is strongly recommended to use the GPU-accelerated ``findface-video-worker-gpu`` variant.

.. note::
   The :ref:`liveness detector <liveness>` is much slower on CPU than on GPU.