Tiered Data Storage
At peak loads, a HDD does not cope with timely saving of event files. As a result, new events are queued and, at a certain point, the saving process terminates.
This problem can be solved with the help of tiered data storage. It works as follows: first, FindFace Multi saves event images to the hot storage (a size-limited SSD), and then the file moving service transfers them to the cold storage (ordinary HDD). Only thumbnails and full frame images are saved to the hot storage. The remaining data, including videos, record images, reports, and so on, is saved immediately to the cold storage and does not need to be moved. To start using the file moving service, enable the hot storage.
Enable Hot Storage
By default, saving to the hot storage is disabled. The file moving service findface-multi-file-mover
starts along with other services but stays non-functional.
To enable the hot storage service, do the following:
Allocate the SSD for the hot storage. Consider the SSD volume: it should be large enough to store the images at peak loads before they are moved to the cold storage.
In the
/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
file, configure theFAST_MEDIA_ROOT
parameter. By default, it’s set toNone
.sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py FAST_MEDIA_ROOT = '/var/lib/fast-storage/uploads'
Open the
/opt/findface-multi/docker-compose.yaml
configuration file.In the
/opt/findface-multi/docker-compose.yaml
file, locate thefindface-multi-file-mover
service.findface-multi-file-mover: command: [python3, /app/service.py, --config=/etc/config.yml] depends_on: [findface-multi-legacy, nats-jetstream] image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.0 logging: {driver: journald} network_mode: service:pause restart: always volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro', './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads']
Configure the
findface-multi-file-mover
service. In thevolumes
section specify the'ssd-path:/mnt/fast_storage/uploads'
– the hot storage directory, from which the files will be subsequently transferred to the cold storage.Note
The
ssd-path
in the example below is a path (directory) to the SSD that will be used for the hot storage. Here and on replace thessd-path
with an existing SSD directory.findface-multi-file-mover: ... volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro', './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
In the
findface-multi-legacy
andfindface-multi-ui
services, specify the hot storage directory'ssd-path:/var/lib/fast-storage/uploads'
in thevolumes
section. This should be done for the following purpose: thefindface-multi-legacy
service saves event images to the specified hot storage directory. Also, before the file moving service moves a thumbnail and a full frame event image to the cold storage, thefindface-multi-ui
service fetches them from the specified hot storage directory to display them to a user.... findface-multi-legacy: depends_on: [pgbouncer, nats, findface-sf-api, findface-counter, findface-liveness-api, etcd, findface-multi-identity-provider] environment: {ADMIN_PASSWORD: admin} image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.1.0 logging: {driver: journald} network_mode: service:pause restart: always volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro', './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads'] ... findface-multi-ui: depends_on: [findface-multi-legacy] image: docker.int.ntl/ntech/multi/multi/ui:ffmulti-2.1.0 logging: {driver: journald} network_mode: service:pause restart: always volumes: ['./configs/findface-multi-ui/nginx-site.conf:/etc/nginx/conf.d/default.conf:ro', './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads'] ...
From the
/opt/findface-multi
directory rebuild all FindFace Multi containers.cd /opt/findface-multi sudo docker-compose down sudo docker-compose up -d
Start Multiple Instances of the findface-multi-file-mover
Service
To provide fault tolerance, you may want to start more than one instance of the findface-multi-file-mover
service. Do the following:
In the
/opt/findface-multi/docker-compose.yaml
file, specify thefindface-multi-file-mover
configuration as many times as many service instances you plan to use, e.g.,findface-multi-file-mover: command: [python3, /app/service.py, --config=/etc/config.yml] depends_on: [findface-multi-legacy, nats-jetstream] image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.0 logging: {driver: journald} network_mode: service:pause restart: always volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro', './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads'] findface-multi-file-mover2: command: [python3, /app/service.py, --config=/etc/config.yml] depends_on: [findface-multi-legacy, nats-jetstream] image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.0 logging: {driver: journald} network_mode: service:pause restart: always volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro', './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
From the
/opt/findface-multi
directory rebuild all FindFace Multi containers.cd /opt/findface-multi sudo docker-compose down sudo docker-compose up -d
Note
When saving to the hot storage is enabled, thumbnails and full frame images that were not transferred to the cold storage will not be saved by backup. If you need to save all images, copy them from the hot to the cold storage using a rsync
utility. See. Back up FindFace Multi.