.. _update:

Update to FindFace Multi 2.2.1
=============================================

.. tip::

   If you use our product FindFace Security deployed on Ubuntu 18.04, `upgrade <https://docs.ntechlab.com/projects/ffmulti/en/1.2/update.html>`_ it to FindFace Multi 1.2 and then :ref:`update <update_ffmulti_1.2_2.2.1>` it to FindFace Multi 2.2.1.

   If you use FindFace Multi 2.2, update it to FindFace Multi 2.2.1 by :ref:`following the instruction <update_ffmulti_2.2_2.2.1>`.

If FindFace Multi is used as a puppet server for integrating with `FindFace CIBR <https://docs.ntechlab.com/projects/ntech-lab-sp-findface/en/2.1.2.1/>`_, re-enable and reconfigure the ``ffsecurity_vns`` plugin using the updated ``findface-multi-legacy.py`` configuration file.

Integration with Axxon Next is included into :ref:`external VMS <external-vms>` integration in FindFace Multi 2.1+ and configured through VMS integration plugin.

.. rubric:: In this section:

.. contents::
   :local:

.. _update_ffmulti_1.2_2.2.1:

Update FindFace Multi 1.2 to FindFace Multi 2.2.1
-----------------------------------------------------------------------------

.. important::

   Before you start the update, disable all cameras.

.. note::
   If you created person and car cards with custom fields in FindFace Multi 1.2 and want them to be displayed in the FindFace Multi 2.2.1 UI, then it is necessary to copy the ``CUSTOM_FIELDS`` section from the old configuration file in step #2 and paste it to the new configuration file in step #8.   

To update FindFace Multi 1.2 to FindFace Multi 2.2.1, do the following:

#. Create a backup copy of the old schema of the Tarantool-based feature vector database:

   .. code::

      sudo cp /etc/findface-security/tnt_schema.lua /etc/findface-security/old_tnt_schema.lua

   Starting with version 2.0, one of the most significant differences between FindFace Multi and earlier versions of the product is the structure of the Tarantool biometric database (so called meta-schema). The new structure is created as a set of spaces, while in previous versions of the product there was only one space by default in the structure of the Tarantool-based database.

#. Open the ``/etc/findface-security/config.py`` configuration file. Save the values of the following parameters for later use: ``EXTERNAL_ADDRESS``, ``SECRET_KEY``, ``VIDEO_DETECTOR_TOKEN``, ``ROUTER_URL``, ``CUSTOM_FIELDS``.

   .. code::

      sudo vi /etc/findface-security/config.py

      EXTERNAL_ADDRESS = "http://172.20.77.58"
      ...
      # use pwgen -sncy 50 1|tr "'" "." to generate your own unique key
      SECRET_KEY = 'c8b533847bbf7142102de1349d33a1f6'
      FFSECURITY = {
      'VIDEO_DETECTOR_TOKEN': '381b0f4a20495227d04185ab02f5085f',
      ...
      'ROUTER_URL': 'http://172.20.77.58',
      ...
      # -- Custom model fields --
      # Edit CUSTOM_FIELDS -> `human_card` section to customize human card fields.
      # Edit CUSTOM_FIELDS -> `face_object` section to customize face object fields.
      # Below is an example with every field type possible.
      # 'CUSTOM_FIELDS': {
      #     'human_card': {
      #         'items': [
      #             {
      #                 'name': 'personid',
      #                 'default': '',
      #                 'label': 'PersonID',
      #                 'display': ['list', 'form'],
      #                 'description': 'Sigur person ID',
      #                 'editable': False
      #             },
      #             {
      #                 'name': 'firstname',
      #                 'default': '',
      #                 'label': 'First Name',
      #                 'display': ['list', 'form'],
      #                 'description': 'Sigur first name',
      #                 'editable': False
      #             },
      #             {
      #                 'name': 'lastname',
      #                 'default': '',
      #                 'label': 'Last Name',
      #                 'display': ['list', 'form'],
      #                 'description': 'Sigur last name',
      #                 'editable': False
      #             },
      #             {
      #                 'name': 'version',
      #                 'default': '',
      #                 'label': 'Version',
      #                 'display': ['list', 'form'],
      #                 'description': 'Sigur photo version',
      #                 'editable': False
      #             }
      #         ],
      #         'filters': [
      #             {
      #                 'name': 'personid',
      #                 'label': 'Sigur person ID filter',
      #                 'field': 'personid'
      #             }
      #         ]
      #     },
      #     'face_object': {
      #         'items': [
      #             {
      #                 "field_name": "tag_name_1",
      #                 "type": "string",
      #                 "default": "change_me"
      #             },
      #             {
      #                 "field_name": "tag_name_2",
      #                 "type": "uint",
      #                 "default": 123
      #             },
      #             {
      #                 "field_name": "tag_name_3",
      #                 "type": "bool",
      #                 "default": True
      #             },
      #         ]
      #     }
      # },
      }

#. Stop the ``findface-security`` service.

   .. code::

      sudo systemctl stop findface-security.service

#. Create a backup of the Tarantool-based feature vector database in any directory of your choice, for example, ``/etc/findface_dump``.

   .. code::

      sudo mkdir -p /etc/findface_dump
      cd /etc/findface_dump
      sudo findface-storage-api-dump -config /etc/findface-sf-api.ini

   .. _mongodb_backup:

   .. note::
      If you have enabled a :ref:`Video Recorder <configure-vms>` and want to keep existing video recordings in FindFace Multi 2.2.1, take an additional step. Create a backup copy of MongoDB:

      .. code::

         sudo mongodump --out /etc/findface_dump/mongo_dump

#. To avoid port conflicts, stop and disable all services before installing a new version.

   .. note::
      There are eight shards in the example below. If it differs with the number of shards in your system, adjust the below command accordingly. E.g., for the system with sixteen shards, replace ``tarantool@shard-00{1..8}.service`` with ``tarantool@shard-0{01..16}.service``. Get the list of active shards with the ``ls /etc/tarantool/instances.enabled/`` command.

   You may stop and disable the services one by one:

   .. code::

      sudo systemctl stop postgresql.service
      sudo systemctl stop postgresql@10-main
      sudo systemctl stop findface-*.service
      sudo systemctl stop pgbouncer.service
      sudo systemctl stop tarantool@shard-00{1..8}.service
      sudo systemctl stop nats-server.service
      sudo systemctl stop etcd.service
      sudo systemctl stop mongod.service
      sudo systemctl stop mongodb.service
      sudo systemctl stop memcached.service
      sudo systemctl stop nginx.service
      sudo systemctl stop redis.service

      sudo systemctl disable postgresql.service
      sudo systemctl disable postgresql@10-main
      sudo systemctl disable pgbouncer.service
      sudo systemctl disable findface-extraction-api.service
      sudo systemctl disable findface-security.service
      sudo systemctl disable findface-security-onvif.service
      sudo systemctl disable findface-sf-api.service
      sudo systemctl disable findface-ntls.service
      sudo systemctl disable findface-video-manager.service
      sudo systemctl disable findface-video-worker-cpu.service
      sudo systemctl disable findface-video-worker-gpu.service
      sudo systemctl disable findface-counter.service
      sudo systemctl disable findface-liveness-api.service
      sudo systemctl disable findface-video-streamer-cpu.service
      sudo systemctl disable findface-video-streamer-gpu.service
      sudo systemctl disable findface-video-storage.service
      sudo systemctl disable tarantool@shard-00{1..8}.service
      sudo systemctl disable nats-server.service
      sudo systemctl disable etcd.service
      sudo systemctl disable mongod.service
      sudo systemctl disable mongodb.service
      sudo systemctl disable memcached.service
      sudo systemctl disable nginx.service
      sudo systemctl disable redis.service

   Or you may use the following compact commands instead:

   .. code::

      sudo systemctl stop postgresql.service postgresql@10-main findface-*.service pgbouncer.service tarantool@shard-00{1..8}.service nats-server.service etcd.service mongod.service mongodb.service memcached.service nginx.service redis.service

      sudo systemctl disable postgresql.service postgresql@10-main pgbouncer.service findface-extraction-api.service findface-security.service findface-security-onvif.service findface-sf-api.service findface-ntls.service findface-video-manager.service findface-video-worker-cpu.service findface-video-worker-gpu.service findface-counter.service findface-liveness-api.service findface-video-streamer-cpu.service findface-video-streamer-gpu.service findface-video-storage.service tarantool@shard-00{1..8}.service nats-server.service etcd.service mongod.service mongodb.service memcached.service nginx.service redis.service

#. :ref:`Install <installer>` the FindFace Multi 2.2.1 instance. Don’t forget to prepare a server first:

   See:

   * :ref:`deploy-prepare-server`
   * :ref:`deploy-prepare-debian`

#. After FindFace Multi installation, stop all the containers from the ``/opt/findface-multi/`` directory.

   .. code::

      cd /opt/findface-multi/

      sudo docker-compose stop

#. Open the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file and paste saved on step #2 values for the parameters ``EXTERNAL_ADDRESS``, ``SECRET_KEY``, ``VIDEO_DETECTOR_TOKEN``, ``ROUTER_URL``, and ``CUSTOM_FIELDS`` into it.

   .. code::

      sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
      ...
      # Use pwgen -sncy 50 1|tr "'" "." to generate your own unique key
      SECRET_KEY = '002231ccb690586f4d33e98322c591bb'
      ...
      SERVICE_EXTERNAL_ADDRESS = 'http://172.20.77.58'
      # EXTERNAL_ADDRESS is used to access objects created inside FFSecurity via external links.
      EXTERNAL_ADDRESS = 'http://172.20.77.58'
      ...
          # findface-video-worker authorization token
          'VIDEO_DETECTOR_TOKEN': '8977e1b0067d43f6c908d0bf60363255',
      ...
          # findface-video-worker face posting address,
          # it must be set to either FFSecurity EXTERNAL_ADDRESS (by default)
          # or findface-facerouter url (in some specific cases)
          'ROUTER_URL': 'http://127.0.0.1:80',

#. If you modified the configuration files of the ``findface-extraction-api``, ``findface-ntls`` and ``findface-sf-api`` services of FindFace Multi 1.2, update the corresponding parameters in the FindFace Multi 2.2.1 configuration files, such as network settings (ports, URLs, etc.).
   If the parameter in the previous configuration file of the FindFace Multi 1.2 version was not changed manually by the user, and in the new configuration file of the FindFace Multi 2.2.1 version has a different value after the update, then it is not recommended to change it manually.

   .. code::

      sudo vi /etc/findface-ntls.cfg
      sudo vi /opt/findface-multi/configs/findface-ntls/findface-ntls.yaml
      sudo vi /etc/findface-extraction-api.ini
      sudo vi /opt/findface-multi/configs/findface-extraction-api/findface-extraction-api.yaml
      sudo vi /etc/findface-sf-api.ini
      sudo vi /opt/findface-multi/configs/findface-sf-api/findface-sf-api.yaml


   .. important::
      FindFace Multi 2.2.1 is not backward compatible with FindFace Multi 1.2 (or earlier) neural network models. Therefore, neural network model names from the configuration file ``/etc/findface-extraction-api.ini`` cannot be transferred to the new configuration file ``/opt/findface-multi/configs/findface-extraction-api/findface-extraction-api.yaml``.


#. Modify the Tarantool database structure by applying the ``tnt_schema.lua`` schema from FindFace Multi 2.2.1.

   .. code::

      sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1 make-tnt-schema | sudo tee /etc/findface-security/tnt_schema.lua

#. Purge data from all the directories relevant to active shards.

   .. code::

      sudo rm /opt/ntech/var/lib/tarantool/shard-*/{index,snapshots,xlogs}/*

#. Copy the meta-schema of the ``default`` space from the ``old_tnt_schema.lua`` configuration file to the new ``tnt_schema.lua`` configuration file, so that the old meta-schema is still available. An easy way to do it is to follow these steps:

   12.1. In the ``/etc/findface-security/old_tnt_schema.lua`` file, rename the following fields:

      .. code::

         meta_scheme --> meta_scheme_default
         meta_indexes --> meta_indexes_default

   12.2. In the new configuration file ``/etc/findface-security/tnt_schema.lua``, replace the following lines at the beginning of the file:

      .. code::

         cfg_spaces = {
           default = {
             meta_scheme = {
               -- internal.normalized_id:
               {
                   default = '',
                   field_type = 'string',
                   id = 1,
                   name = 'normalized_id',
               },
               -- internal.feat:
               {
                   default = '',
                   field_type = 'string',
                   id = 2,
                   name = 'feat',
               },
             },
             meta_indexes = {}
           },


      with these ones:

      .. code::

         dofile("/etc/findface-security/old_tnt_schema.lua")
         spaces = {
          default = {
             meta_scheme=meta_scheme_default,
             meta_indexes=meta_indexes_default
           },

#. Navigate to the directory with Tarantool configuration file(s) ``/etc/tarantool/instances.available/``. Import new meta-schema ``tnt_schema.lua`` into each configuration file ``shard-00*.lua``, as in the example below.

   .. code::

      sudo vi /etc/tarantool/instances.available/shard-00*.lua

      dofile("/etc/findface-security/tnt_schema.lua")
      FindFace = require("FindFace")
      FindFace.start("127.0.0.1", 8104, {
          license_ntls_server="127.0.0.1:3133",
          replication = replication_master,
          spaces = spaces
       })

#. Remove the default configuration file ``FindFace.lua``, generated by the ``findface-tarantool-server`` package, as it will block the restart, required on the next step.

   .. code::

      sudo rm -rf /etc/tarantool/instances*/FindFace.lua

#. Restart the ``findface-tarantool-server`` shards.

   .. code::

      TNT=$(ls /etc/tarantool/instances.enabled/ | cut -c 7,8,9)
      for i in $TNT; do sudo systemctl restart tarantool@shard-$i.service ; done

   Upon completion of the above steps, the shards will still keep the old galleries created within the ``default`` space, but new spaces (e.g., ``ffsec_body_objects_space``, ``ffsec_face_clusters_space``, and so on) will also become available.


#. Restore old data from the backup. The data will be restored as it existed previously: all galleries will stay within the ``default`` space.

    .. code::

       sudo systemctl start findface-ntls.service
       cd /etc/findface_dump
       for x in *.json; do sudo findface-storage-api-restore -config /etc/findface-sf-api.ini < "$x"; done

#. Migrate galleries from the ``default`` space to new spaces:

    .. code::

       sudo systemctl start findface-sf-api.service
       sudo systemctl start nginx.service
       sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py migrate_tnt_space

#. Perform PostgreSQL database migrations for FindFace Multi 2.2.1 compatibility. Do the following:

   18.1. Navigate to the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file. In the ``DATABASES -> default`` section, temporarily replace ``PASSWORD`` with the old one, used in the ``/etc/findface-security/config.py`` configuration file.

      .. important::
         Make sure to write down the password from the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file. You will need it later.

      .. code::

         sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py

         DATABASES = {
             'default': {
                 'ENGINE': 'django.db.backends.postgresql',
                 'DISABLE_SERVER_SIDE_CURSORS': True,
                 'NAME': 'ffsecurity', 'HOST': '127.0.0.1', 'PORT': 5439, 'USER': 'ntech', 'PASSWORD': 'XXXXXXXXXXXXXXXX'
             }
         }

   18.2. In the ``/etc/pgbouncer/pgbouncer.ini`` file, add the following line to the ``databases`` section:

      .. code::

         ffsecurity_session = dbname=ffsecurity host=localhost port=5432 user=ntech pool_mode=session pool_size=10

   18.3. On the host system, perform the database migration:

      .. code::

         sudo systemctl start postgresql.service
         sudo systemctl start pgbouncer.service
         sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py migrate

   18.4. Back up an existing database with PostgreSQL, installed on the host system.

      .. code::

         cd /opt/findface-multi/
         sudo -u postgres pg_dump --verbose --disable-triggers ffsecurity | sudo tee dump_ffsecurity.sql
         sudo -u postgres pg_dump -t auth_group -t ffsecurity_adgroupguid -t ffsecurity_deviceblacklistrecord -t ffsecurity_ffsecauthsession -t ffsecurity_grouppermission -t ffsecurity_runtimesetting -t ffsecurity_user -t ffsecurity_user_groups -t ffsecurity_user_user_permissions -t ffsecurity_userkeyvalue -t knox_authtoken -t ffsecurity_watchlistpermission -t ffsecurity_cameragrouppermission --data-only --verbose --no-acl --no-owner --disable-triggers ffsecurity | sudo tee dump_identity_provider.sql

   18.5. Back up role permissions.

      .. code::

         sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py dump_permissions | sudo tee permissions.csv

   18.6. **Change back the password, replaced in step #18.1**

   18.7. Stop all the services.

      .. code::

         sudo systemctl stop findface-sf-api.service nginx.service tarantool@shard-00{1..8}.service postgresql.service pgbouncer.service

   18.8. Open the ``/opt/findface-multi/docker-compose.yaml`` file and copy a ``{NEW_POSTGRES_PASSWORD}`` value to use it in further commands.

      .. code::

         grep postgresql\: docker-compose.yaml -A5 | grep POSTGRES_PASSWORD
             environment: {POSTGRESQL_ALLOW_REMOTE_CONNECTIONS: 'no', POSTGRES_PASSWORD: {NEW_POSTGRES_PASSWORD}}

   18.9. Remove PostgreSQL data. Then, start PostgreSQL by starting the containers and recreate all existing databases.

      .. code::

         sudo rm -rf data/postgresql
         sudo docker-compose up -d --force-recreate postgresql

   18.10. Restore data into the recreated ``ffsecurity`` database. Paste ``{NEW_POSTGRES_PASSWORD}`` value that you previously copied in step #18.8 into the command below:

      .. code::

         sudo docker exec -i findface-multi-postgresql-1 /bin/bash -c "PGPASSWORD={NEW_POSTGRES_PASSWORD} psql --username postgres ffsecurity" < dump_ffsecurity.sql

   18.11. Run migration.

      .. code::

         sudo docker-compose up -d pgbouncer
         sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/identity-provider:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py migrate

   18.12. Restore data into the recreated ``ffsecurity_identity_provider`` database. Paste ``{NEW_POSTGRES_PASSWORD}`` value that you previously copied in step #18.8 into the command below:

      .. code::

         sudo docker exec -i findface-multi-postgresql-1 /bin/bash -c "PGPASSWORD={NEW_POSTGRES_PASSWORD} psql --username postgres ffsecurity_identity_provider" < dump_identity_provider.sql

   18.13. Start all the services.

      .. code::

         sudo docker-compose up -d

   18.14. Run the command to create records in the outbox table for watch lists and camera groups.

      .. code::

         sudo docker run --rm --network host --volume '/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro' docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py export_to_outbox

   18.15. Before you restore role permissions into the ``identity_provider`` service, examine the ``/opt/findface-multi/permissions.csv`` file. Make sure to replace ``*_ffsecauthtoken`` with ``*_authtoken`` if any. This is mostly applicable to those cases when FindFace Multi 1.2 installation was an upgrade from earlier versions of the product.

      After that, restore role permissions into the ``identity_provider`` service.

      .. code::

         sudo docker run --rm --network host -v /opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py:/etc/findface-security/config.py:ro -v $(pwd)/permissions.csv:/var/permissions.csv:ro docker.int.ntl/ntech/multi/multi/identity-provider:ffmulti-2.2.1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py load_permissions /var/permissions.csv

   18.16. Copy full frame photos, normalized images, and the license file. Copy files from the ``/opt/ntech/license/`` folder into the ``/opt/findface-multi/data/findface-ntls/`` folder, from the ``/var/lib/findface-security/uploads/`` folder into the ``/opt/findface-multi/data/findface-multi-legacy/uploads/`` folder, from the ``/var/lib/ffupload/uploads/`` folder into the ``/opt/findface-multi/data/findface-upload/uploads/`` folder.

      .. code::

         sudo cp -r /opt/ntech/license/* /opt/findface-multi/data/findface-ntls/
         sudo cp -r /var/lib/findface-security/uploads/* /opt/findface-multi/data/findface-multi-legacy/uploads/
         sudo cp -r /var/lib/ffupload/uploads/* /opt/findface-multi/data/findface-upload/uploads/
         sudo chmod 777 -R /opt/findface-multi/data/findface-upload/uploads/
         sudo chown www-data:www-data -R /opt/findface-multi/data/findface-upload/uploads/*

      Alternatively, the above folders can be directly mounted into the relevant docker containers via the ``docker-compose.yaml`` file, like in the example below:

      .. code::

         sudo vi /opt/findface-multi/docker-compose.yaml

         ...
         findface-upload:
           image: docker.int.ntl/ntech/universe/upload:ffserver-12.240830.2
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-upload/40-ffupload.sh:/docker-entrypoint.d/40-ffupload.sh:ro',
             '/var/lib/ffupload:/var/lib/ffupload']
         ...
         findface-multi-identity-provider:
           depends_on:
             findface-multi-identity-provider-migrate: {condition: service_completed_successfully}
           healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:8022/healthcheck
               || exit 1', timeout: 3s}
           image: docker.int.ntl/ntech/multi/multi/identity-provider:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-identity-provider/findface-multi-identity-provider.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-identity-provider-migrate:
           command: [migrate]
           depends_on: [pgbouncer, nats, nats-jetstream, findface-sf-api, findface-liveness-api,
             etcd]
           environment: {ADMIN_PASSWORD: admin}
           image: docker.int.ntl/ntech/multi/multi/identity-provider:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: on-failure
           volumes: ['./configs/findface-multi-identity-provider/findface-multi-identity-provider.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-legacy:
           depends_on:
             findface-multi-identity-provider: {condition: service_started}
             findface-multi-legacy-migrate: {condition: service_completed_successfully}
           healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:8002/healthcheck
               || exit 1', timeout: 3s}
           image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-legacy-migrate:
           command: [migrate]
           depends_on: [pgbouncer, nats, findface-sf-api, nats-jetstream]
           image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: on-failure
           volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-legacy-singleton-services:
           command: [run-single-instance-services]
           depends_on:
             findface-multi-identity-provider: {condition: service_healthy}
             findface-multi-legacy-migrate: {condition: service_completed_successfully}
           healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:9901/healthcheck
               || exit 1', timeout: 3s}
           image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         cleaner:
           command: [run-cleaner-service]
           depends_on: [pgbouncer, findface-sf-api, findface-multi-legacy]
           image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-ui:
           depends_on: [findface-multi-legacy]
           image: docker.int.ntl/ntech/multi/multi/ui:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-ui/nginx-site.conf:/etc/nginx/conf.d/default.conf:ro',
             '/var/lib/findface-security/uploads:/var/lib/findface-security/uploads']
         ...
         findface-multi-file-mover:
           command: [python3, /app/service.py, --config=/etc/config.yml]
           depends_on: [findface-multi-legacy, nats-jetstream]
           image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.2.1
           logging: {driver: journald}
           network_mode: service:pause
           restart: always
           volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
             '/var/lib/findface-security/uploads:/mnt/slow_storage/uploads']
         ...


   18.17. To move Tarantool data, do the following:

          Stop all FindFace Multi containers:

          .. code::

             sudo docker-compose down

          Start the old shards and the ``findface-sf-api`` service again:

          .. code::

             sudo systemctl start tarantool@shard-00{1..8}.service findface-sf-api.service

          Create a new backup of the feature vector database:

          .. code::

             sudo mkdir -p /etc/findface_dump_final
             sudo findface-storage-api-dump -output-dir=/etc/findface_dump_final -config /etc/findface-sf-api.ini

          Stop the rest of the services, clear the ``instances.enabled`` directory, start the containers again, and perform the storage-api-restore operation:

          .. code::

             sudo systemctl stop tarantool@shard-00{1..8}.service findface-sf-api.service findface-ntls.service
             sudo rm /etc/tarantool/instances.enabled/*
             sudo docker-compose up -d
             sudo findface-storage-api-restore -config /opt/findface-multi/configs/findface-sf-api/findface-sf-api.yaml /etc/findface_dump_final/*.json

.. note::
   If you have created a backup of MongoDB on :ref:`step 4 <mongodb_backup>` to keep video recordings of a Video Recorder, take this additional step. Move a MongoDB backup copy to the ``/opt/findface-multi/data/mongodb`` directory and restore MongoDB data. Do the following:

   .. code::

      sudo cp -r /etc/findface_dump/mongo_dump/ /opt/findface-multi/data/mongodb
      sudo chown mongodb:mongodb -R /opt/findface-multi/data/mongodb/mongo_dump/*
      sudo docker exec findface-multi-mongodb-1 mongorestore data/db/mongo_dump

The update has been completed, but the new version includes new neural network models, so it is also necessary to :ref:`migrate feature vectors to a different neural network model <migrate>`.

.. important::

   We highly recommend disabling the Ubuntu automatic update to preserve the FindFace Multi compatibility with the installation environment. In this case, you will be able to update your OS manually, fully controlling which packages to update.

   To disable the Ubuntu automatic update, execute the following commands:

   .. code::

      sudo apt-get remove unattended-upgrades
      sudo systemctl stop apt-daily.timer
      sudo systemctl disable apt-daily.timer
      sudo systemctl disable apt-daily.service
      sudo systemctl daemon-reload



.. _update_ffmulti_2.2_2.2.1:

Update FindFace Multi 2.2 to FindFace Multi 2.2.1
---------------------------------------------------------------

To update FindFace Multi 2.2 to FindFace Multi 2.2.1, do the following:

#. In a user home directory, create a folder for FindFace Multi 2.2 backup data.

   .. code::

      mkdir ~/backup_data/


#. Back up FindFace Multi configuration files and the ``docker-compose.yaml`` file and switch to the ``/opt/findface-multi/`` directory.

   .. code::

      sudo cp -r /opt/findface-multi/configs/ ~/backup_data/configs/
      sudo cp /opt/findface-multi/docker-compose.yaml ~/backup_data/

      cd /opt/findface-multi/

#. Back up PostgreSQL data.

   #. Open the ``/opt/findface-multi/docker-compose.yaml`` file and copy ``{POSTGRES PASSWORD}`` value to use it in further command.

      .. code::

         grep postgresql\: docker-compose.yaml -A5 | grep POSTGRES_PASSWORD
             environment: {POSTGRESQL_ALLOW_REMOTE_CONNECTIONS: 'no', POSTGRES_PASSWORD: {POSTGRES PASSWORD}}

   #. Create a folder for PostgreSQL backup data.

      .. code::

        mkdir ~/backup_data/pg_bkp

   #. Create backups and paste the ``{POSTGRES_PASSWORD}`` value that you previously copied.

      .. code::

        sudo docker exec -i findface-multi-postgresql-1 /bin/bash -c "PGPASSWORD={POSTGRES_PASSWORD} pg_dump --username postgres ffsecurity" > ~/backup_data/pg_bkp/legacy.sql
        sudo docker exec -i findface-multi-postgresql-1 /bin/bash -c "PGPASSWORD={POSTGRES_PASSWORD} pg_dump --username postgres ffsecurity_identity_provider" > ~/backup_data/pg_bkp/identity_provider.sql

#. Back up Tarantool data.

   .. code::

      sudo docker exec -it findface-multi-findface-sf-api-1 bash -c "mkdir ffmulti_dump; cd ffmulti_dump && /storage-api-dump -config /etc/findface-sf-api.ini"
      sudo docker cp findface-multi-findface-sf-api-1:/ffmulti_dump ~/backup_data/

#. List content of the ``backup_data/`` folder, located in a user home directory. It will look like this.

   .. code::

      ls ~/backup_data/
      configs  docker-compose.yaml  ffmulti_dump  pg_bkp

#. Stop all FindFace Multi containers.

   .. code::

      sudo docker-compose down

#. Remove the ``/opt/findface-multi/data/findface-tarantool-server/`` and ``/opt/findface-multi/data/postgresql/`` folders.

   .. code::

      sudo rm -r /opt/findface-multi/data/findface-tarantool-server/

      sudo rm -r /opt/findface-multi/data/postgresql/

#. :ref:`Install <installer>` the FindFace Multi 2.2.1 instance into the ``/opt/findface-multi`` directory.

#. Restore Tarantool data from the backup.

   #. Copy the ``~/backup_data/ffmulti_dump/`` folder to the ``findface-multi-findface-sf-api-1`` container.

      .. code::

         sudo docker cp  ~/backup_data/ffmulti_dump/ findface-multi-findface-sf-api-1:/

   #. Restore Tarantool data.

      .. note::
         If you are updating from FindFace Multi version with Guardant offline license implemented via USB dongle, first take an action to configure the ``findface-ntls`` section in the ``/opt/findface-multi/docker-compose.yaml`` file. Take steps 2, 3 from the :ref:`grd` instruction.      

     .. code::

        sudo docker exec -it findface-multi-findface-sf-api-1 bash -c 'cd ffmulti_dump && for x in *.json; do /storage-api-restore -config /etc/findface-sf-api.ini < "$x"; done;'


#. In the new configuration files, replace the ``NTECH USER`` password with the one, used in FindFace Multi 2.2. The old password can be obtained from the ``~/backup_data/configs/postgresql/40-init.sql`` backup file. The ``NTECH USER`` password should be replaced in the following configuration files:

   * ``findface-multi-legacy.py``
   * ``findface-multi-line-crossing-analytics.yaml``
   * ``findface-multi-identity-provider.py``
   * ``findface-multi-alerts.yaml``
   * ``findface-multi-audit.py``
   * ``pgbouncer -> userlist.txt``
   * ``postgresql -> 40-init.sql``

#. In the new configuration file ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py``, replace the ``SECRET_KEY`` value in the ``GENERAL SETTINGS`` section and ``'VIDEO_DETECTOR_TOKEN'`` value in the ``FINDFACE SECURITY SETTINGS`` section with the ones, used in FindFace Multi 2.2. The old parameter values can be obtained from the ``~/backup_data/configs/findface-multi-legacy/findface-multi-legacy.py`` backup file.

#. If you’ve changed the ``/opt/findface-multi/configs/findface-extraction-api/findface-extraction-api.yaml`` configuration file in FindFace Multi 2.2, then compare the parameters you’ve changed with the parameters in the new ``/opt/findface-multi/configs/findface-extraction-api/findface-extraction-api.yaml`` configuration file of FindFace Multi 2.2.1 version and modify custom parameters.

     .. code::

       sudo vi /opt/findface-multi/configs/findface-extraction-api/findface-extraction-api.yaml

   .. important::
      Follow these rules to transfer parameters from the old configuration file to the new one:

      * If there is a new neural network model in the new configuration file, replace it with the previous one (considering that the previous model is still included in FindFace Multi 2.2.1), and if the previous model is missing in FindFace Multi 2.2.1, then do not change anything. In this case, you will have to do :ref:`migration to a different neural network model <migrate>`.
      * If a parameter had an empty value in the old configuration file, but has a certain value in the new configuration file, delete its value in the new configuration file.
      * Keep as is those parameters that were not included in the old configuration file, but are present in the new configuration file.

   .. important::

      If you’ve changed configuration files ``/opt/findface-multi/configs/findface-ntls/findface-ntls.yaml`` and ``/opt/findface-multi/configs/findface-sf-api/findface-sf-api.yaml`` in FindFace Multi 2.2, then compare the parameters you’ve changed with the parameters in the new ``/opt/findface-multi/configs/findface-ntls/findface-ntls.yaml`` and ``/opt/findface-multi/configs/findface-sf-api/findface-sf-api.yaml`` configuration files of FindFace Multi 2.2.1 version and modify custom parameters if necessary.

       .. code::

         sudo vi /opt/findface-multi/configs/findface-ntls/findface-ntls.yaml
         sudo vi /opt/findface-multi/configs/findface-sf-api/findface-sf-api.yaml



      You may need to apply changes to the ``docker-compose.yaml`` file if you previously configured it to bring or exclude services or made any other changes.

       .. code::

         sudo vi /opt/findface-multi/docker-compose.yaml

#. Stop all FindFace Multi containers from the ``/opt/findface-multi`` directory.

   .. code::

      cd /opt/findface-multi
      sudo docker rm -f -v findface-multi-postgresql-1
      sudo rm -rf data/postgresql
      sudo docker-compose down

#. Restore data from backups.

   #. Recreate postgresql container.

      .. code::

          sudo docker-compose up -d --force-recreate postgresql

   #. Open the ``/opt/findface-multi/docker-compose.yaml`` file and copy a ``{NEW_POSTGRES_PASSWORD}`` value to use it in further step. Note that this password will be different from the previous one.

      .. code::

         grep postgresql\: docker-compose.yaml -A5 | grep POSTGRES_PASSWORD
             environment: {POSTGRESQL_ALLOW_REMOTE_CONNECTIONS: 'no', POSTGRES_PASSWORD: {NEW_POSTGRES_PASSWORD}}

   #. Restore data from backups. Paste the ``{NEW_POSTGRES_PASSWORD}`` value that you previously copied.

      .. code::

          sudo docker exec -i  findface-multi-postgresql-1  bash -c "PGPASSWORD={NEW_POSTGRES_PASSWORD} psql --username postgres ffsecurity" < <(cat ~/backup_data/pg_bkp/legacy.sql)
          sudo docker exec -i  findface-multi-postgresql-1  bash -c "PGPASSWORD={NEW_POSTGRES_PASSWORD} psql --username postgres ffsecurity_identity_provider" < <(cat ~/backup_data/pg_bkp/identity_provider.sql)

#. Start all FindFace Multi containers.

   .. code::

      sudo docker-compose up -d

.. note::
   You can additionally deploy Video Recorder. See :ref:`deploy-vms`.