.. _s3_storage:

S3-Compatible Storage Configuration
======================================

S3 (Simple Storage Service) provides reliable and long-term storage of an unlimited number of files and data. It gives a way to avoid the limitations of the file system when it comes to large amounts of data.

Configure S3-Compatible Storage
-------------------------------------

In this section, we will describe how to configure S3-compatible `MinIO <https://min.io/>`_ storage for operation with FindFace Multi. If you prefer another S3-compatible storage, you are yourself responsible for its configuration.

#. :ref:`Prepare a server on Ubuntu <deploy-prepare-server>`.
   
   .. note::
      MinIO operation has been tested on Ubuntu only. Refer to the official `MinIO documentation <https://docs.min.io/>`_ to find out whether other OS are supported.

#. To start a container with MinIO, execute the following command, specify the ``MINIO_ROOT_USER`` and ``MINIO_ROOT_PASSWORD`` login credentials: 

   .. code::

      docker run -d \
      -p 9003:9000 \
      -p 9004:9001 \
      -e "MINIO_ROOT_USER=admin" \
      -e "MINIO_ROOT_PASSWORD=admin_admin" \
      -v /path/to/storage_directory:/data \
      quay.io/minio/minio server /data --console-address ":9001"

   where ``/path/to/storage_directory`` is a directory, allocated for the S3 storage.

#. MinIO web interface address will depend on the interface that you have selected during FindFace Multi :ref:`installation <installer>` in step #4.5. The port in use is ``9004``, e.g., ``192.168.112.254:9004``. Log in to the MinIO web interface with ``MINIO_ROOT_USER``, ``MINIO_ROOT_PASSWORD`` login credentials that you have specified in the previous step. 

#. In the MinIO UI, create a bucket (:guilabel:`Buckets` -> :guilabel:`Create Bucket`). Assign a default name ``ffsec.storage`` to the bucket.

   .. note::
      We advise keeping the bucket name ``ffsec.storage`` as suggested by default, because it corresponds to the ``'STORAGE_BUCKET_NAME'`` parameter in the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` and ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-identity-provider.py`` configuration files. If you use another name for a bucket, make sure to adjust it in the above-mentioned configuration files accordingly.


   |create_bucket_en|

    .. |create_bucket_en| image:: /_static/create_bucket_en.png
        :scale: 60%


#. Click on the bucket to open its settings.


   |bucket_settings_en|

    .. |bucket_settings_en| image:: /_static/bucket_settings_en.png
        :scale: 60%


#. Change access policy for the bucket to ``public`` for the correct processing of video archives.


   |access_policy_en|

    .. |access_policy_en| image:: /_static/access_policy_en.png
        :scale: 60%


#. In the :guilabel:`User` -> :guilabel:`Access Keys` section, create an access key. **Write down the** ``Access key`` **and** ``Secret key`` **values as you will need them later for FindFace Multi configuration.**


   |access_key_en|

    .. |access_key_en| image:: /_static/access_key_en.png
        :scale: 55%


   .. note::
      MinIO UI may differ. In the earlier versions, set up an access key in the :guilabel:`Identity` -> :guilabel:`Service Accounts` -> :guilabel:`Create service account` section.

.. _employ_s3_storage:

Configure FindFace Multi to Employ S3-Compatible Storage
------------------------------------------------------------

#. In the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file, locate the ``EXTERNAL_STORAGE_CONFIG`` section. 

   .. code::

      sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py

      EXTERNAL_STORAGE_CONFIG = {
          'STORAGE_TYPE': None,
          'STORAGE_BUCKET_NAME': 'ffsec.storage',
          'ACCESS_KEY_ID': 'access-key',
          'SECRET__KEY': 'secret-key',
          'REGION_NAME': 'eu-west-3',
          'LOCAL_S3_SETTINGS': {
              'CONNECTION_ADDRESS': 'localhost:9003',
          },
      }

   Configure the ``EXTERNAL_STORAGE_CONFIG`` parameters:

   #. The ``'STORAGE_TYPE'`` parameter may take values ``'LOCAL'``, ``'AWS'``, ``None``:

      * ``None`` – local file system (default)

      * ``'LOCAL'`` – any S3-compatible storage (e.g., MinIO)

      * ``'AWS'`` – `Amazon S3 <https://aws.amazon.com/s3/>`_ storage. 

      Change the ``'STORAGE_TYPE'`` parameter to ``'LOCAL'`` to enable S3-compatible storage.

   #. The values of the ``'STORAGE_BUCKET_NAME'``, ``'ACCESS_KEY_ID'``, and ``'SECRET__KEY'`` parameters should correspond to those that were specified during MinIO initialization.

   #. ``'REGION_NAME'`` – the default value is ``eu-west-3``. It can be changed in MinIO settings.

   #. In the ``'LOCAL_S3_SETTINGS'`` -> ``'CONNECTION_ADDRESS'``, replace the ``localhost`` with MinIO IP host address, e.g., ``192.168.112.254``.

#. Configure the ``/opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py`` file in the same way. Locate the ``EXTERNAL_STORAGE_CONFIG`` section and apply changes, analogous to those made in the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` file in the previous step.

   .. code::

      sudo vi /opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py

      EXTERNAL_STORAGE_CONFIG = {
          'STORAGE_TYPE': None,
          'STORAGE_BUCKET_NAME': 'ffsec.storage',
          'ACCESS_KEY_ID': 'access-key',
          'SECRET__KEY': 'secret-key',
          'REGION_NAME': 'eu-west-3',
          'LOCAL_S3_SETTINGS': {
              'CONNECTION_ADDRESS': 'localhost:9003',
          },
      }

#. In order for the UI to load content (thumbnails, full frame images, and so on), configure the ``/opt/findface-multi/configs/findface-multi-ui/nginx-site.conf`` file. In the ``server`` section, locate the ``add_header Content-Security-Policy`` and add ``{ip_host_s3}:9003/`` address, e.g.:

   .. code::

      sudo vi /opt/findface-multi/configs/findface-multi-ui/nginx-site.conf

      ...
      server {
              listen 80 default_server;
              listen [::]:80 default_server;
      ...
              location / {
      ...
              }
              add_header Content-Security-Policy "default-src 'self' 192.168.112.254:9003/ ; img-src 'self' 192.168.112.254:9003/ blob: data:; media-src 'self' blob:; style-src 'self' 'unsafe-inline';";
      ...
      }

#. Rebuild all FindFace Multi containers from the ``/opt/findface-multi`` directory .

   .. code::

      cd /opt/findface-multi

      sudo docker-compose down

      sudo docker-compose up -d

#. To make sure that the ``FindFace Local S3 Storage`` service is running, explore the ``findface-multi-findface-multi-legacy-1`` logs: 

   .. code::

      sudo docker logs findface-multi-findface-multi-legacy-1 

Data Transfer Command
---------------------------

To transfer data to either remote or local storage, use the following command:

.. code::

   sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py transfer_data --to_local / --to_remote


* The argument ``--to_remote`` transfers data to remote storage.
* The argument ``--to_local``  transfers data to the local file system.


Before executing the above command, make sure that you have :ref:`configured FindFace Multi <employ_s3_storage>` to use S3-compatible storage.