.. _hot_cold_storage:

Tiered Data Storage
==============================

At peak loads, a HDD does not cope with timely saving of event files. As a result, new events are queued and, at a certain point, the saving process terminates.

This problem can be solved with the help of tiered data storage. It works as follows: first, FindFace Multi saves event images to the hot storage (a size-limited SSD), and then the file moving service transfers them to the cold storage (ordinary HDD). Only thumbnails and full frame images are saved to the hot storage. The remaining data, including videos, record images, reports, and so on, is saved immediately to the cold storage and does not need to be moved. To start using the file moving service, enable the hot storage.

Enable Hot Storage
----------------------------

By default, saving to the hot storage is disabled. The file moving service ``findface-multi-file-mover`` starts along with other services but stays non-functional.
To enable the hot storage service, do the following:

#. Allocate the SSD for the hot storage. Consider the SSD volume: it should be large enough to store the images at peak loads before they are moved to the cold storage.

#. In the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` file, configure the ``FAST_MEDIA_ROOT`` parameter. By default, it's set to ``None``.

   .. code::

      sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py

      FAST_MEDIA_ROOT = '/var/lib/fast-storage/uploads'

#. Open the ``/opt/findface-multi/docker-compose.yaml`` configuration file.

   #. In the ``/opt/findface-multi/docker-compose.yaml`` file, locate the ``findface-multi-file-mover`` service.

      .. code::

         findface-multi-file-mover:
           command: [python3, /app/service.py, --config=/etc/config.yml]
           depends_on: [findface-multi-legacy, nats-jetstream]
           image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
             './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads']

   #. Configure the ``findface-multi-file-mover`` service. In the ``volumes`` section specify the ``'ssd-path:/mnt/fast_storage/uploads'`` – the hot storage directory, from which the files will be subsequently transferred to the cold storage.

      .. note::
         The ``ssd-path`` in the example below is a path (directory) to the SSD that will be used for the hot storage. Here and on replace the ``ssd-path`` with an existing SSD directory.

      .. code::

         findface-multi-file-mover:
           ...
           volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
                 './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']

   #. In the ``findface-multi-legacy``, ``findface-multi-legacy-singleton-services`` and ``findface-multi-ui`` services, specify the hot storage directory ``'ssd-path:/var/lib/fast-storage/uploads'`` in the ``volumes`` section. This should be done for the following purposes: 
   
      * The ``findface-multi-legacy`` service saves event images to the specified hot storage directory.
      * The reports are generated inside the ``findface-multi-legacy-singleton-services`` service, so it is necessary to specify the hot storage directory ``'ssd-path:/var/lib/fast-storage/uploads'`` in the ``volumes`` of the ``findface-multi-legacy-singleton-services`` service.     
      * Before the file moving service moves a thumbnail and a full frame event image to the cold storage, the ``findface-multi-ui`` service fetches them from the specified hot storage directory to display them to a user.

      .. code::

         ...
           findface-multi-legacy:
             depends_on:
               findface-multi-identity-provider: {condition: service_healthy}
               findface-multi-legacy-migrate: {condition: service_completed_successfully}
             healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:8002/healthcheck
                 || exit 1', timeout: 3s}
             image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
             logging: {driver: journald}
             network_mode: service:pause
             restart: always
             volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
               './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads']
         ...
           findface-multi-legacy-singleton-services:
             command: [run-single-instance-services]
             depends_on:
               findface-multi-identity-provider: {condition: service_healthy}
               findface-multi-legacy-migrate: {condition: service_completed_successfully}
             healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:9901/healthcheck
                 || exit 1', timeout: 3s}
             image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
             logging: {driver: journald}
             network_mode: service:pause
             restart: always
             volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
               './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads']
         ...
           findface-multi-ui:
             depends_on: [findface-multi-legacy]
             image: docker.int.ntl/ntech/multi/multi/ui:ffmulti-2.2.1
             logging: {driver: journald}
             network_mode: service:pause
             restart: always
             volumes: ['./configs/findface-multi-ui/nginx-site.conf:/etc/nginx/conf.d/default.conf:ro',
               './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads']
         ...


   #. From the ``/opt/findface-multi`` directory rebuild all FindFace Multi containers.

      .. code::

         cd /opt/findface-multi

         sudo docker-compose down

         sudo docker-compose up -d


Start Multiple Instances of the ``findface-multi-file-mover`` Service
------------------------------------------------------------------------

To provide fault tolerance, you may want to start more than one instance of the ``findface-multi-file-mover`` service. Do the following:

#. In the ``/opt/findface-multi/docker-compose.yaml`` file, specify the ``findface-multi-file-mover`` configuration as many times as many service instances you plan to use, e.g.,

   .. code::

      findface-multi-file-mover:
        command: [python3, /app/service.py, --config=/etc/config.yml]
        depends_on: [findface-multi-legacy, nats-jetstream]
        image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.2.1
        logging: {driver: journald}
        network_mode: service:pause
        restart: always
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
          './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
      findface-multi-file-mover2:
        command: [python3, /app/service.py, --config=/etc/config.yml]
        depends_on: [findface-multi-legacy, nats-jetstream]
        image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.2.1
        logging: {driver: journald}
        network_mode: service:pause
        restart: always
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
          './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']

#. From the ``/opt/findface-multi`` directory rebuild all FindFace Multi containers.

   .. code::

      cd /opt/findface-multi

      sudo docker-compose down

      sudo docker-compose up -d

.. note::

   When saving to the hot storage is enabled, thumbnails and full frame images that were not transferred to the cold storage will not be saved by backup. If you need to save all images, copy them from the hot to the cold storage using a ``rsync`` utility. See :ref:`back-up-findface-multi`.