Back Up and Recover FindFace and Data

This section is all about the backup and recovery of the FindFace data storages and your system as a whole.

In this section:

List of Data Storages 

FindFace uses the following data storages:

Tarantool-based feature vector database that stores face feature vectors and recognition events.
PostgreSQL-based main system database ffsecurity that stores internal system data, records, and user accounts.
The /var/lib/findface-security/uploads directory that stores photos uploaded to records, video files, full frames of events, and thumbnails.
The /var/lib/ffupload/ directory that stores such event artifacts as normalized face images.

Feature Vector Database Backup and Recovery (Tarantool)

There are the following galleries in the Tarantool-based feature vector database:

ffsec_face_events: feature vectors extracted from faces detected in the video.
ffsec_face_objects: feature vectors extracted from face images uploaded to the record index.
ffsec_face_clusters: centroids of face clusters.

The database backup/recovery functionality allows you to fully restore all the galleries when needed.

To avoid data loss, we recommend you back up a feature vector database at least once a week. Overall, the backups’ frequency depends on the number of records and face recognition events, and available disk space.

Be sure to back up the database before migrating your system to another neural network model.

Utilities 

To back up and recover the FindFace feature vector database, the following utilities are needed:

backup: findface-storage-api-dump,
recovery: findface-storage-api-restore.

These utilities are automatically installed along with findface-sf-api.

Back Up Database 

To back up the feature vector database, use the findface-storage-api-dump utility as follows:

Important

The following services must be active: findface-tarantool-server, findface-sf-api.

Note

The backup functionality can be applied to a distributed database. In this case, the findface-storage-api-dump utility will back up galleries on all the shards specified in /etc/findface-sf-api.ini.

On the server with findface-sf-api, create a directory to store the backup files (/etc/findface_dump in the example below).
Launch the findface-storage-api-dump utility by executing:
```
sudo findface-storage-api-dump -output-dir=/etc/findface_dump -config /etc/findface-sf-api.ini
```
The utility will back up at once all the galleries into the files with corresponding names (ffsec_body_events.json, ffsec_face_events, etc.) and save them into the directory. These files contain all the data needed to restore the entire database.

Recover Database 

To recover the feature vector database from the backup, launch the findface-storage-api-restore utility for all the files in the backup folder:

sudo findface-storage-api-restore -config /etc/findface-sf-api.ini /etc/findface_dump/*.json

The recovery process can be interrupted and resumed whenever necessary. To resume the process after the interruption, launch the findface-storage-api-restore utility again.

Main Database Backup and Recovery (PostgreSQL)

To back up the main database ffsecurity based on PostgreSQL, execute:

sudo -u postgres pg_dump ffsecurity > ffsecurity_postgres_backup.sql

To recover the main database, do the following:

Stop the findface-security service.

sudo systemctl stop findface-security.service

Stop the pgbouncer service to delete its active sessions with the ffsecurity database.
```
sudo systemctl stop pgbouncer.service
```
Open the PostgreSQL interactive terminal.
```
sudo -u postgres psql
```
Remove the old ffsecurity database.
```
DROP DATABASE ffsecurity;
```

Create a new ffsecurity database. Leave the PostgreSQL interactive terminal.

CREATE DATABASE ffsecurity WITH OWNER ntech ENCODING 'UTF-8' LC_COLLATE='C.UTF-8' LC_CTYPE='C.UTF-8' TEMPLATE template0;

Start the pgbouncer service.
```
sudo systemctl start pgbouncer.service
```

Recover the database content from the backup.

sudo -u postgres psql -d ffsecurity -f ffsecurity_postgres_backup.sql

Migrate the database architecture from FindFace to PostgreSQL, re-create user groups with predefined rights and the first user with administrator rights.
```
sudo findface-security migrate
sudo findface-security create_groups
sudo findface-security create_default_user
```

Start the findface-security service.

sudo systemctl start findface-security.service

Artifacts Backup and Recovery (`findface-upload`)

The FindFace artifacts, such as photos uploaded to the record index, video files, and such event artifacts as full frames, thumbnails, and normalized face images, are stored in the following directories:

/var/lib/findface-security/uploads
/var/lib/ffupload/

Note

Both directories are operated by the findface-upload component.

To back up the artifacts, execute:

sudo tar -cvzf /home/some_directory/var_lib_ffsecurity_uploads.tar.gz /var/lib/findface-security/uploads/
sudo tar -cvzf /home/some_directory/var_lib_ffupload.tar.gz /var/lib/ffupload/

To recover the artifacts, execute the following commands from the root directory:

cd /
sudo tar -xvf /home/some_directory/var_lib_ffsecurity_uploads.tar.gz
sudo tar -xvf /home/some_directory/var_lib_ffupload.tar.gz

Settings Backup and Recovery 

The entire set of FindFace configuration files including the Tarantool structural schema is automatically backed up during the instance removal. It is saved to the ~/ffmulti_bak_${datetime}/etc/ directory.

When re-installing FindFace, recover the settings after completing the console installation, or right after installing services from the APT repository.

The entire set of backed up files is the following:

ls -R -p
.:
findface-counter.ini.bak         findface-liveness-api.ini.bak  findface-security/       findface-video-manager.conf.bak  findface-video-streamer-cpu.ini.bak  nginx/
findface-extraction-api.ini.bak  findface-ntls.cfg.bak          findface-sf-api.ini.bak  findface-video-storage.conf.bak  findface-video-worker-cpu(gpu).ini.bak

./findface-security:
config.py  tnt_schema.lua

./nginx:
sites-enabled/

./nginx/sites-enabled:
ffsecurity-nginx.conf

To recover the FindFace settings, do the following:

Open the /etc/findface-security/config.py configuration file of the fresh FindFace instance. Find the DATABASES -> default -> PASSWORD parameter that stores the ntech user password from the findface-security database of PostgreSQL. Copy/paste it to the ~/ffmulti_bak_${datetime}/etc/findface-security/config.py backup.
```
sudo vi /etc/findface-security/config.py

DATABASES = {
   ‘default’:
     ‘PASSWORD’: ‘some_pass’
```
In the ~/ffmulti_bak_${datetime}/etc/ directory, use any method to eliminate the .bak extension from the files. For example, you can execute the following command.
```
sudo rename 's/.ini.bak/.ini/' * && sudo rename 's/.conf.bak/.conf/' * && sudo rename 's/.cfg.bak/.cfg/' *
```
Tip

If the rename command is absent in your system, you can install it as follows:
```
sudo apt install rename
```
Recursively copy the backup files to the /etc directory.
```
sudo cp -r * /etc
```

Modify the database structure by applying the initial tnt_schema.lua file.

sudo findface-security make_tnt_schema | sudo tee /etc/findface-security/tnt_schema.lua

Restart the services.

On CPU:

sudo systemctl restart findface-counter findface-liveness-api findface-video-manager findface-extraction-api findface-ntls findface-sf-api findface-video-worker-cpu findface-security

On GPU:

sudo systemctl restart findface-counter findface-liveness-api findface-video-manager findface-extraction-api findface-ntls findface-sf-api findface-video-worker-gpu findface-security

Back Up and Recover Entire System 

If you intend to back up FindFace before uninstalling it, it will be sufficient to follow the step-by-step instructions in the Remove FindFace Instance section. The provided findface_uninstall.sh script can automatically back up the FindFace configuration files and all data storages to the ~/ffmulti_bak_${datetime}/ directory.

To recover FindFace after uninstalling it, use the following algorithm:

Deploy FindFace.
Recover the settings from the backed up configuration files.
Recover Tarantool.
Recover PostgreSQL.
Recover the artifacts.