Tiered Data Storage

At peak loads, a HDD does not cope with timely saving of event files. As a result, new events are queued and, at a certain point, the saving process terminates.

This problem can be solved with the help of tiered data storage. It works as follows: first, FindFace Multi saves event images to the hot storage (a size-limited SSD), and then the file moving service transfers them to the cold storage (ordinary HDD). Only thumbnails and full frame images are saved to the hot storage. The remaining data, including videos, record images, reports, and so on, is saved immediately to the cold storage and does not need to be moved. To start using the file moving service, enable the hot storage.

Enable Hot Storage

By default, saving to the hot storage is disabled. The file moving service findface-multi-file-mover starts along with other services but stays non-functional. To enable the hot storage service, do the following:

1. Allocate the SSD for the hot storage. Consider the SSD volume: it should be large enough to store the images at peak loads before they are moved to the cold storage.

2. In the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py file, configure the FAST_MEDIA_ROOT parameter. By default, it’s set to None.

   sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
   
   FAST_MEDIA_ROOT = '/var/lib/fast-storage/uploads'
   

3. Open the /opt/findface-multi/docker-compose.yaml configuration file.

   1. In the /opt/findface-multi/docker-compose.yaml file, locate the findface-multi-file-mover service.

      findface-multi-file-mover:
        command: [python3, /app/service.py, --config=/etc/config.yml]
        depends_on: [findface-multi-legacy, nats-jetstream]
        image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.3
        logging: {driver: journald}
        network_mode: service:pause
        restart: always
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
          './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads']
      

   2. Configure the findface-multi-file-mover service. In the volumes section specify the 'ssd-path:/mnt/fast_storage/uploads' – the hot storage directory, from which the files will be subsequently transferred to the cold storage.

      Note

      The ssd-path in the example below is a path (directory) to the SSD that will be used for the hot storage. Here and on replace the ssd-path with an existing SSD directory.

      findface-multi-file-mover:
        ...
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
              './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
      

   3. In the findface-multi-legacy and findface-multi-ui services, specify the hot storage directory 'ssd-path:/var/lib/fast-storage/uploads' in the volumes section. This should be done for the following purpose: the findface-multi-legacy service saves event images to the specified hot storage directory. Also, before the file moving service moves a thumbnail and a full frame event image to the cold storage, the findface-multi-ui service fetches them from the specified hot storage directory to display them to a user.

      ...
        findface-multi-legacy:
          depends_on:
            findface-multi-identity-provider: {condition: service_started}
            findface-multi-legacy-migrate: {condition: service_completed_successfully}
          healthcheck: {interval: 5s, retries: 3, start_period: 60s, test: 'curl -f http://localhost:8002/healthcheck
              || exit 1', timeout: 3s}
          image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.1.3
          logging: {driver: journald}
          network_mode: service:pause
          restart: always
          volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
            './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads']
      ...
        findface-multi-ui:
          depends_on: [findface-multi-legacy]
          image: docker.int.ntl/ntech/multi/multi/ui:ffmulti-2.1.3
          logging: {driver: journald}
          network_mode: service:pause
          restart: always
          volumes: ['./configs/findface-multi-ui/nginx-site.conf:/etc/nginx/conf.d/default.conf:ro',
            './data/findface-multi-legacy/uploads:/var/lib/findface-security/uploads', 'ssd-path:/var/lib/fast-storage/uploads']
      ...
      

   4. From the /opt/findface-multi directory rebuild all FindFace Multi containers.

      cd /opt/findface-multi
      
      sudo docker-compose down
      
      sudo docker-compose up -d
      

Start Multiple Instances of the findface-multi-file-mover Service

To provide fault tolerance, you may want to start more than one instance of the findface-multi-file-mover service. Do the following:

1. In the /opt/findface-multi/docker-compose.yaml file, specify the findface-multi-file-mover configuration as many times as many service instances you plan to use, e.g.,

   findface-multi-file-mover:
     command: [python3, /app/service.py, --config=/etc/config.yml]
     depends_on: [findface-multi-legacy, nats-jetstream]
     image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.3
     logging: {driver: journald}
     network_mode: service:pause
     restart: always
     volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
       './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
   findface-multi-file-mover2:
     command: [python3, /app/service.py, --config=/etc/config.yml]
     depends_on: [findface-multi-legacy, nats-jetstream]
     image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.1.3
     logging: {driver: journald}
     network_mode: service:pause
     restart: always
     volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
       './data/findface-multi-legacy/uploads:/mnt/slow_storage/uploads', 'ssd-path:/mnt/fast_storage/uploads']
   

2. From the /opt/findface-multi directory rebuild all FindFace Multi containers.

   cd /opt/findface-multi
   
   sudo docker-compose down
   
   sudo docker-compose up -d
   

Note

When saving to the hot storage is enabled, thumbnails and full frame images that were not transferred to the cold storage will not be saved by backup. If you need to save all images, copy them from the hot to the cold storage using a rsync utility. See Back up FindFace Multi.