.. _backup-restore: Back Up and Recover FindFace Multi and Its Data ============================================================= This section is all about the backup and recovery of the FindFace Multi data storages and your system as a whole. .. rubric:: In this section: .. contents:: :local: List of Data Storages -------------------------------- FindFace Multi uses the following data storages: * Tarantool-based feature vector database that stores object feature vectors and events. * PostgreSQL-based main system database ``ffsecurity`` that stores internal system data, cards, user accounts, and camera settings. * The ``/var/lib/findface-security/uploads`` directory that stores uploaded card photos, video files, full frames of events and counters, and object thumbnails. * The ``/var/lib/ffupload/`` directory that stores such event artifacts as normalized object images. .. note:: It also stores video chunks if you are using :ref:`Video Recorder `. * (Only with Video Recorder) MongoDB-based database that stores meta-information of the video chunks, including their exact location in the storage, ``/var/lib/ffupload/``. .. _tarantool-backup-recover: Feature Vector Database Backup and Recovery (Tarantool) ---------------------------------------------------------- There are the following galleries in the Tarantool-based feature vector database: * ``ffsec_body_events``: feature vectors extracted from bodies detected in the video. * ``ffsec_body_objects``: feature vectors extracted from body images uploaded to the card index. * ``ffsec_body_clusters``: centroids of body clusters. * ``ffsec_car_events``: feature vectors extracted from cars detected in the video. * ``ffsec_car_objects``: feature vectors extracted from car images uploaded to the card index. * ``ffsec_car_clusters``: centroids of car clusters. * ``ffsec_face_events``: feature vectors extracted from faces detected in the video. * ``ffsec_face_objects``: feature vectors extracted from face images uploaded to the card index. * ``ffsec_face_clusters``: centroids of face clusters. * ``ffsec_user_face``: feature vectors extracted from the FindFace Multi users' photos for face-based authentication. The database backup/recovery functionality allows you to fully restore all the galleries when needed. To avoid data loss, we recommend you back up a feature vector database at least once a week. Overall, the backups' frequency depends on the number of cards and object recognition events, and available disk space. Be sure to back up the database before :ref:`migrating ` your system to another neural network model. Utilities ^^^^^^^^^^^^^^^^^ To back up and recover the FindFace Multi feature vector database, the following utilities are needed: #. backup: ``findface-storage-api-dump``, #. recovery: ``findface-storage-api-restore``. These utilities are automatically installed along with ``findface-sf-api``. Back Up Database ^^^^^^^^^^^^^^^^^^^^^^^^ To back up the feature vector database, use the ``findface-storage-api-dump`` utility as follows: .. important:: The following services must be active: ``findface-tarantool-server``, ``findface-sf-api``. .. note:: The backup functionality can be applied to a distributed database. In this case, the ``findface-storage-api-dump`` utility will back up galleries on all the shards specified in ``/etc/findface-sf-api.ini``. #. On the server with ``findface-sf-api``, create a directory to store the backup files (``/etc/findface_dump`` in the example below). #. Launch the ``findface-storage-api-dump`` utility by executing: .. code:: sudo findface-storage-api-dump -output-dir=/etc/findface_dump -config /etc/findface-sf-api.ini The utility will back up at once all the galleries into the files with corresponding names (``ffsec_body_events.json``, ``ffsec_face_events``, etc.) and save them into the directory. These files contain all the data needed to restore the entire database. Recover Database ^^^^^^^^^^^^^^^^^^^^^^^^^^ To recover the feature vector database from the backup, launch the ``findface-storage-api-restore`` utility for all the files in the backup folder: .. code:: sudo findface-storage-api-restore -config /etc/findface-sf-api.ini /etc/findface_dump/*.json The recovery process can be interrupted and resumed whenever necessary. To resume the process after the interruption, launch the ``findface-storage-api-restore`` utility again. .. seealso:: * :ref:`backup-config` * :ref:`restore-config` .. _postgresql-backup-recover: Main Database Backup and Recovery (PostgreSQL) ---------------------------------------------------- To back up the main database ``ffsecurity`` based on PostgreSQL, execute: .. code:: sudo -u postgres pg_dump ffsecurity > ffsecurity_postgres_backup.sql To recover the main database, do the following: #. Stop the ``findface-security`` service. .. code:: sudo systemctl stop findface-security.service #. Stop the ``pgbouncer`` service to delete its active sessions with the ``ffsecurity`` database. .. code:: sudo systemctl stop pgbouncer.service #. Open the PostgreSQL interactive terminal. .. code:: sudo -u postgres psql #. Remove the old ``ffsecurity`` database. .. code:: DROP DATABASE ffsecurity; #. Create a new ``ffsecurity`` database. Leave the PostgreSQL interactive terminal. .. code:: CREATE DATABASE ffsecurity WITH OWNER ntech ENCODING 'UTF-8' LC_COLLATE='C.UTF-8' LC_CTYPE='C.UTF-8' TEMPLATE template0; #. Start the ``pgbouncer`` service. .. code:: sudo systemctl start pgbouncer.service #. Recover the database content from the backup. .. code:: sudo -u postgres psql -d ffsecurity -f ffsecurity_postgres_backup.sql #. Migrate the database architecture from FindFace Multi to :program:`PostgreSQL`, re-create user groups with :ref:`predefined ` rights and the first user with administrator rights. .. code:: sudo findface-security migrate sudo findface-security create_groups sudo findface-security create_default_user #. Start the ``findface-security`` service. .. code:: sudo systemctl start findface-security.service .. _upload-backup-recover: Artifacts Backup and Recovery (``findface-upload``) ------------------------------------------------------------- The FindFace Multi artifacts, such as photos uploaded to the card index, video files, and such event artifacts as full frames, object thumbnails, and normalized object images, are stored in the following directories: * ``/var/lib/findface-security/uploads`` * ``/var/lib/ffupload/`` .. note:: The ``/var/lib/ffupload/`` directory also stores video chunks if there is a running instance of Video Recorder in your system. .. note:: Both directories are operated by the ``findface-upload`` component. To back up the artifacts, execute: .. code:: sudo tar -cvzf /home/some_directory/var_lib_ffsecurity_uploads.tar.gz /var/lib/findface-security/uploads/ sudo tar -cvzf /home/some_directory/var_lib_ffupload.tar.gz /var/lib/ffupload/ To recover the artifacts, execute the following commands from the root directory: .. code:: cd / sudo tar -xvf /home/some_directory/var_lib_ffsecurity_uploads.tar.gz sudo tar -xvf /home/some_directory/var_lib_ffupload.tar.gz .. _mongodb-backup-recover: Video Recorder Database Backup and Recovery (MongoDB) ------------------------------------------------------------- MongoDB is a third-party software that implements the Video Recorder database. This database stores meta-information of the video chunks, including their exact location in the storage, ``/var/lib/ffupload/``. To back up MongoDB, execute: .. code:: sudo mongodump -o=/home/some_directory/mongodb_backup To recover MongoDB, execute: .. code:: sudo mongorestore /home/some_directory/mongodb_backup .. _settings-recover: Settings Backup and Recovery ----------------------------------- The entire set of FindFace Multi configuration files including the Tarantool structural schema is automatically backed up during the instance :ref:`removal `. It is saved to the ``~/ffmulti_bak_${datetime}/etc/`` directory. When re-installing FindFace Multi, recover the settings after completing the :ref:`console installation `, or right after installing services from the :ref:`APT repository `. The entire set of backed up files is the following: .. code:: ls -R -p .: findface-counter.ini.bak findface-liveness-api.ini.bak findface-security/ findface-video-manager.conf.bak findface-video-streamer-cpu.ini.bak nginx/ findface-extraction-api.ini.bak findface-ntls.cfg.bak findface-sf-api.ini.bak findface-video-storage.conf.bak findface-video-worker-cpu(gpu).ini.bak ./findface-security: config.py tnt_schema.lua ./nginx: sites-enabled/ ./nginx/sites-enabled: ffsecurity-nginx.conf To recover the FindFace Multi settings, do the following: #. Open the ``/etc/findface-security/config.py`` configuration file of the fresh FindFace Multi instance. Find the ``DATABASES`` -> ``default`` -> ``PASSWORD`` parameter that stores the ``ntech`` user password from the ``findface-security`` database of PostgreSQL. Copy/paste it to the ``~/ffmulti_bak_${datetime}/etc/findface-security/config.py`` backup. .. code:: sudo vi /etc/findface-security/config.py DATABASES = { ‘default’: ‘PASSWORD’: ‘some_pass’ #. In the ``~/ffmulti_bak_${datetime}/etc/`` directory, use any method to eliminate the ``.bak`` extension from the files. For example, you can execute the following command. .. code:: sudo rename 's/.ini.bak/.ini/' * && sudo rename 's/.conf.bak/.conf/' * && sudo rename 's/.cfg.bak/.cfg/' * .. tip:: If the ``rename`` command is absent in your system, you can install it as follows: .. code:: sudo apt install rename #. Recursively copy the backup files to the ``/etc`` directory. .. code:: sudo cp -r * /etc #. Modify the database structure by applying the initial ``tnt_schema.lua`` file. .. code:: sudo findface-security make_tnt_schema | sudo tee /etc/findface-security/tnt_schema.lua #. Restart the services. .. rubric:: On CPU: .. code:: sudo systemctl restart findface-counter findface-liveness-api findface-video-manager findface-extraction-api findface-ntls findface-sf-api findface-video-worker-cpu findface-security .. rubric:: On GPU: .. code:: sudo systemctl restart findface-counter findface-liveness-api findface-video-manager findface-extraction-api findface-ntls findface-sf-api findface-video-worker-gpu findface-security .. rubric:: Video Recorder services: .. code:: sudo systemctl restart findface-video-streamer-cpu findface-video-storage Back Up and Recover Entire System -------------------------------------------- If you intend to back up FindFace Multi before uninstalling it, it will be sufficient to follow the step-by-step instructions in the :ref:`remove-instance` section. The provided :download:`ffmulti_uninstall.sh <_scripts/ffmulti_uninstall.sh>` script can automatically back up the FindFace Multi configuration files and all data storages to the ``~/ffmulti_bak_${datetime}/`` directory. To recover FindFace Multi after uninstalling it, use the following algorithm: #. :ref:`Deploy FindFace Multi `. #. :ref:`Recover the settings from the backed up configuration files `. #. :ref:`Recover Tarantool `. #. :ref:`Recover PostgreSQL `. #. :ref:`Recover the artifacts `. #. :ref:`Recover MongoDB ` (if you are using Video Recorder).