Tiered Data Storage

At peak loads, a HDD does not cope with timely saving of event files. As a result, new events are queued and, at a certain point, the saving process terminates.

This problem can be solved with the help of tiered data storage. It works as follows: first, FindFace Multi saves event images to the hot storage (a size-limited SSD), and then the file moving service transfers them to the cold storage (ordinary HDD). Only thumbnails and full frame images are saved to the hot storage. The remaining data, including videos, record images, reports, and so on, is saved immediately to the cold storage and does not need to be moved. To start using the file moving service, enable the hot storage.

Enable Hot Storage

By default, saving to the hot storage is disabled. The file moving service findface-multi-file-mover starts along with other services but stays non-functional. To enable the hot storage service, do the following:

1. Allocate the SSD for the hot storage. Consider the SSD volume: it should be large enough to store the images at peak loads before they are moved to the cold storage. Create a directory for the hot storage on the host (for example, /home/username/fast).

   sudo mkdir ~/fast
   

2. In the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py file, configure the FAST_MEDIA_ROOT parameter. By default, it’s set to None.

   sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
   
   FAST_MEDIA_ROOT = '/var/lib/fast-storage/uploads'
   

3. Open the /opt/findface-multi/docker-compose.yaml configuration file.

   1. Define the fast-storage section in top-level volumes. Replace the /home/ubuntu/fast with an existing SSD directory.

      sudo vi /opt/findface-multi/docker-compose.yaml
      
      volumes:
        ...
        fast-storage:
          driver: local
          driver_opts: {device: /home/ubuntu/fast, o: bind, type: none}
      

   2. Locate the findface-multi-file-mover service.

      findface-multi-file-mover:
        command: [python3, /app/service.py, --config=/etc/config.yml]
        depends_on: [findface-multi-legacy, nats-jetstream]
        image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.3.0
        logging: {driver: journald}
        networks: [product-network]
        restart: always
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
          'findface-multi-uploads:/mnt/slow_storage/uploads']
      

   3. Configure the findface-multi-file-mover service. In the volumes section specify the 'fast-storage:/mnt/fast_storage/uploads' – the hot storage directory, from which the files will be subsequently transferred to the cold storage.

      Note

      The fast-storage volume is mounted to /mnt/fast_storage/uploads in the findface-multi-file-mover container. The fast-storage volume in the example below defines a path (directory) to the SSD that will be used for the hot storage.

      findface-multi-file-mover:
        ...
        volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
              'findface-multi-uploads:/mnt/slow_storage/uploads',
              'fast-storage:/mnt/fast_storage/uploads']
      

   4. In the findface-multi-legacy, findface-multi-legacy-singleton-services and findface-multi-ui services, mount the fast-storage volume at /var/lib/fast-storage/uploads as 'fast-storage:/var/lib/fast-storage/uploads' in the volumes section of the corresponding service. This should be done for the following purposes:

      • The findface-multi-legacy service saves event images to the specified hot storage directory.

      • The reports are generated inside the findface-multi-legacy-singleton-services service, so it is necessary to specify the hot storage directory 'fast-storage:/var/lib/fast-storage/uploads' in the volumes of the findface-multi-legacy-singleton-services service.

      • Before the file moving service moves a thumbnail and a full frame event image to the cold storage, the findface-multi-ui service fetches them from the specified hot storage directory to display them to a user.

        ...
        findface-multi-legacy:
          depends_on:
            findface-multi-identity-provider: {condition: service_healthy}
            findface-multi-legacy-migrate: {condition: service_completed_successfully}
          healthcheck: {interval: 5s, retries: 3, start_period: 300s, test: 'curl -f http://localhost:8002/healthcheck
              || exit 1', timeout: 3s}
          image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.3.0
          logging: {driver: journald}
          networks: [product-network]
          restart: always
          volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
            './configs/findface-multi-legacy/ui-config.yaml:/etc/findface-security/ui-config.yaml:ro',
            'findface-multi-uploads:/var/lib/findface-security/uploads',
            'fast-storage:/var/lib/fast-storage/uploads']
        ...
        findface-multi-legacy-singleton-services:
          command: [run-single-instance-services]
          depends_on:
            findface-multi-identity-provider: {condition: service_healthy}
            findface-multi-legacy-migrate: {condition: service_completed_successfully}
          healthcheck: {interval: 5s, retries: 3, start_period: 300s, test: 'curl -f http://localhost:9901/healthcheck
              || exit 1', timeout: 3s}
          image: docker.int.ntl/ntech/multi/multi/legacy:ffmulti-2.3.0
          logging: {driver: journald}
          networks: [product-network]
          restart: always
          volumes: ['./configs/findface-multi-legacy/findface-multi-legacy.py:/etc/findface-security/config.py:ro',
            './configs/findface-multi-legacy/ui-config.yaml:/etc/findface-security/ui-config.yaml:ro',
            'findface-multi-uploads:/var/lib/findface-security/uploads',
            'fast-storage:/var/lib/fast-storage/uploads']
        ...
        findface-multi-ui:
          depends_on: [findface-multi-legacy]
          image: docker.int.ntl/ntech/multi/multi/ui:ffmulti-2.3.0
          logging: {driver: journald}
          networks: [product-network]
          ports: ['80:80']
          restart: always
          volumes: ['./configs/findface-multi-ui/nginx-site.conf:/etc/nginx/conf.d/default.conf:ro',
            'findface-multi-uploads:/var/lib/findface-security/uploads',
            'fast-storage:/var/lib/fast-storage/uploads']
      ...
      

   5. From the /opt/findface-multi directory rebuild all FindFace Multi containers.

      cd /opt/findface-multi
      
      sudo docker-compose down
      
      sudo docker-compose up -d
      

Start Multiple Instances of the findface-multi-file-mover Service

To provide fault tolerance, you may want to start more than one instance of the findface-multi-file-mover service. Do the following:

1. In the /opt/findface-multi/docker-compose.yaml file, specify the findface-multi-file-mover configuration as many times as many service instances you plan to use, e.g.,

   findface-multi-file-mover:
     command: [python3, /app/service.py, --config=/etc/config.yml]
     depends_on: [findface-multi-legacy, nats-jetstream]
     image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.3.0
     logging: {driver: journald}
     networks: [product-network]
     restart: always
     volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
       'findface-multi-uploads:/mnt/slow_storage/uploads',
       'fast-storage:/mnt/fast_storage/uploads']
   findface-multi-file-mover-2:
     command: [python3, /app/service.py, --config=/etc/config.yml]
     depends_on: [findface-multi-legacy, nats-jetstream]
     image: docker.int.ntl/ntech/multi/multi/file-mover:ffmulti-2.3.0
     logging: {driver: journald}
     networks: [product-network]
     restart: always
     volumes: ['./configs/findface-multi-file-mover/findface-multi-file-mover.yaml:/etc/config.yml:ro',
       'findface-multi-uploads:/mnt/slow_storage/uploads',
       'fast-storage:/mnt/fast_storage/uploads']
   

2. From the /opt/findface-multi directory rebuild all FindFace Multi containers.

   cd /opt/findface-multi
   
   sudo docker-compose down
   
   sudo docker-compose up -d
   

Note

When saving to the hot storage is enabled, thumbnails and full frame images that were not transferred to the cold storage will not be saved by backup. If you need to save all images, copy them from the hot to the cold storage using a rsync utility. See Back up FindFace Multi.