S3-Compatible Storage Configuration

S3 (Simple Storage Service) provides reliable and long-term storage of an unlimited number of files and data. It gives a way to avoid the limitations of the file system when it comes to large amounts of data.

Configure S3-Compatible Storage

In this section, we will describe how to configure S3-compatible MinIO storage for operation with FindFace Multi. If you prefer another S3-compatible storage, you are yourself responsible for its configuration.

  1. Prepare a server on Ubuntu.

    Note

    MinIO operation has been tested on Ubuntu only. Refer to the official MinIO documentation to find out whether other OS are supported.

  2. To start a container with MinIO, execute the following command, specify the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD login credentials:

    docker run -d \
    -p 9003:9000 \
    -p 9004:9001 \
    -e "MINIO_ROOT_USER=admin" \
    -e "MINIO_ROOT_PASSWORD=admin_admin" \
    -v /path/to/storage_directory:/data \
    quay.io/minio/minio server /data --console-address ":9001"
    

    where /path/to/storage_directory is a directory, allocated for the S3 storage.

  3. MinIO web interface address will depend on the interface that you have selected during FindFace Multi installation in step #4.5. The port in use is 9004, e.g., 192.168.112.254:9004. Log in to the MinIO web interface with MINIO_ROOT_USER, MINIO_ROOT_PASSWORD login credentials that you have specified in the previous step.

  4. In the MinIO UI, create a bucket (Buckets -> Create Bucket). Assign a default name ffsec.storage to the bucket.

    Note

    We advise keeping the bucket name ffsec.storage as suggested by default, because it corresponds to the 'STORAGE_BUCKET_NAME' parameter in the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py and /opt/findface-multi/configs/findface-multi-legacy/findface-multi-identity-provider.py configuration files. If you use another name for a bucket, make sure to adjust it in the above-mentioned configuration files accordingly.

    create_bucket_en

  5. Click on the bucket to open its settings.

    bucket_settings_en

  6. Change access policy for the bucket to public for the correct processing of video archives.

    access_policy_en

  7. In the User -> Access Keys section, create an access key. Write down the Access key and Secret key values as you will need them later for FindFace Multi configuration.

    access_key_en

    Note

    MinIO UI may differ. In the earlier versions, set up an access key in the Identity -> Service Accounts -> Create service account section.

Configure FindFace Multi to Employ S3-Compatible Storage

  1. In the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py configuration file, locate the EXTERNAL_STORAGE_CONFIG section.

    sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
    
    EXTERNAL_STORAGE_CONFIG = {
        'STORAGE_TYPE': None,
        'STORAGE_BUCKET_NAME': 'ffsec.storage',
        'ACCESS_KEY_ID': 'access-key',
        'SECRET__KEY': 'secret-key',
        'REGION_NAME': 'eu-west-3',
        'LOCAL_S3_SETTINGS': {
            'CONNECTION_ADDRESS': 'localhost:9003',
        },
    }
    

    Configure the EXTERNAL_STORAGE_CONFIG parameters:

    1. The 'STORAGE_TYPE' parameter may take values 'LOCAL', 'AWS', None:

      • None – local file system (default)

      • 'LOCAL' – any S3-compatible storage (e.g., MinIO)

      • 'AWS'Amazon S3 storage.

      Change the 'STORAGE_TYPE' parameter to 'LOCAL' to enable S3-compatible storage.

    2. The values of the 'STORAGE_BUCKET_NAME', 'ACCESS_KEY_ID', and 'SECRET__KEY' parameters should correspond to those that were specified during MinIO initialization.

    3. 'REGION_NAME' – the default value is eu-west-3. It can be changed in MinIO settings.

    4. In the 'LOCAL_S3_SETTINGS' -> 'CONNECTION_ADDRESS', replace the localhost with MinIO IP host address, e.g., 192.168.112.254.

  2. Configure the /opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py file in the same way. Locate the EXTERNAL_STORAGE_CONFIG section and apply changes, analogous to those made in the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py file in the previous step.

    sudo vi /opt/findface-multi/configs/findface-multi-identity-provider/findface-multi-identity-provider.py
    
    EXTERNAL_STORAGE_CONFIG = {
        'STORAGE_TYPE': None,
        'STORAGE_BUCKET_NAME': 'ffsec.storage',
        'ACCESS_KEY_ID': 'access-key',
        'SECRET__KEY': 'secret-key',
        'REGION_NAME': 'eu-west-3',
        'LOCAL_S3_SETTINGS': {
            'CONNECTION_ADDRESS': 'localhost:9003',
        },
    }
    
  3. In order for the UI to load content (thumbnails, full frame images, and so on), configure the /opt/findface-multi/configs/findface-multi-ui/nginx-site.conf file. In the server section, locate the add_header Content-Security-Policy and add {ip_host_s3}:9003/ address, e.g.:

    sudo vi /opt/findface-multi/configs/findface-multi-ui/nginx-site.conf
    
    ...
    server {
            listen 80 default_server;
            listen [::]:80 default_server;
    ...
            location / {
    ...
            }
            add_header Content-Security-Policy "default-src 'self' 192.168.112.254:9003/ ; img-src 'self' 192.168.112.254:9003/ blob: data:; media-src 'self' blob:; style-src 'self' 'unsafe-inline';";
    ...
    }
    
  4. Rebuild all FindFace Multi containers from the /opt/findface-multi directory .

    cd /opt/findface-multi
    
    sudo docker-compose down
    
    sudo docker-compose up -d
    
  5. To make sure that the FindFace Local S3 Storage service is running, explore the findface-multi-findface-multi-legacy-1 logs:

    sudo docker logs findface-multi-findface-multi-legacy-1
    

Data Transfer Command

To transfer data to either remote or local storage, use the following command:

sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py transfer_data --to_local / --to_remote
  • The argument --to_remote transfers data to remote storage.

  • The argument --to_local transfers data to the local file system.

Before executing the above command, make sure that you have configured FindFace Multi to use S3-compatible storage.