Migrate to Different Detector or Model

Tip

Do not hesitate to contact our experts on migration by info@ntechlab.com.

Sometimes you have to migrate your FindFace Enterprise Server SDK instance to another face detector or neural network model. This usually happens when you decide to update to the latest version of the product.

Tip

You can find the models summary here.

If you need to re-detect faces, you must regenerate both normalized face images, thumbnails and facens. If you just want to apply a different model, it usually suffices to regenerate only facens. FindFace Enterprise Server SDK provides tools that can handle most migration use cases.

Warning

Different detectors have diverse sensitivity to certain facial features. Be aware that, after re-detecting your database, you may miss out on some previously found faces.

In this section:

Tools

To migrate your instance, you will need the following tools:

Tool Description
findface-regenerate Script that regenerates and overrides face data in MongoDB by applying different detector settings or another model to the images in the Uploads folder.
mongo2searchapi Script that copies newly generated facens from MongoDB to Tarantool.

Both tools are automatically installed with findface-facenapi.

Requirements

The /var/lib/ffupload/uploads/ folder (Uploads) has to be populated with at least the original images. Its content can be viewed at http://<findface_upload_IP:3333/uploads/ in your browser.

Overall, the findface-regenerate tool works with the Uploads folder in the following way:

Use case How it works
Different detector settings The findface-regenerate tool runs original images through the facenapi-nnapi pipeline with different detector [and model] settings, and returns regenerated normalized images, thumbnails and facens.
Different model The findface-regenerate tool runs normalized face images through nnapi with different model settings, and returns regenerated facens.

Regenerate Face Data

Important

Before conducting any operations on your MongoDB database, be sure to create its backup.

Apply findface-regenerate as follows:

  1. Navigate into /usr/bin/. Display and thoroughly examine the findface-regenerate help message:

    cd /usr/bin/
    findface-regenerate --help
    
    ## findface-regenerate --help
    
    Usage: /usr/bin/findface-regenerate [OPTIONS]
    
    Options:
    
     --help                           show this help information
    
    /usr/lib/python3/dist-packages/facenapi/core/decoders/decoder_threaded.py options:
    
     --max-size                       Maximum allowed photo width/height (default
                                     4000)
    
    /usr/lib/python3/dist-packages/facenapi/core/detectors/detector_dlib.py options:
    
     --dlib-adjust-threshold          Adjust face detector threshold (default 0.0)
     --dlib-max-size                  images with width or height larger than
                                      dlib_max_size will be scaled down before
                                      being fed into detector (default 1200)
     --dlib-normalizer                path to normalizer data (default
                                      /usr/share/findface-data/normalizer.dat)
    
    /usr/lib/python3/dist-packages/facenapi/core/detectors/detector_nnd.py options:
    
     --nnd-max-face-size              Maximum face size in pixels (no limit if 0)
                                      (default 0)
     --nnd-min-face-size              Minimum face size in pixels (default 30.0)
     --nnd-o-net-thresh                (default 0.9)
     --nnd-p-net-thresh                (default 0.5)
     --nnd-r-net-thresh                (default 0.5)
     --nnd-scale-factor                (default 0.79)
     --nnd-workers                    Number of detector workers threads. (0 - as
                                      much as there are cpus) (default 0)
    
    /usr/lib/python3/dist-packages/facenapi/core/main_utils.py options:
    
     --decoder                        Image decoder (threaded) (default threaded)
     --detector                       Face detector (dlib,nnd) (default nnd)
     --extractor                      Feature extractor (nnapi,extraction-api)
                                      (default nnapi)
     --facen-storage                  Feature vector storage
                                      (searchapi_replicated,tntapi,searchapi)
                                      (default tntapi)
     --id-generator                   Face id generator (tntime,mongo) (default
                                      tntime)
    
    /usr/lib/python3/dist-packages/facenapi/server/context.py options:
    
     --fetch-proxy                    Fetch images from urls via proxy, ex:
                                      http://1.2.3.4:3128
     --ffupload-url                   url (without path) to PUT images uploaded to
                                      /face, ex: http://127.0.0.1:1234
     --friend-count                    (default 5)
     --friend-interval                 (default 604800)
     --gae                            enable Gender, Age and Emotions support
                                      (default False)
     --mongo-host                     mongo database host (default localhost)
     --mongo-port                     mongo database port (default 27017)
     --person-identify                identify persons (default False)
     --person-identify-global         identify persons across all cameras (default
                                      False)
     --person-identify-threshold      threshold for persons identify (default
                                      0.75)
     --upload-path                    path of $ffupload_url (default uploads)
    
    /usr/lib/python3/dist-packages/facenapi/server/regenerate_facens.py options:
    
     --config                         path to config file
     --coroutines                     Number of parallel coroutines (default 30)
     --every-other                     (default 1)
     --every-other-offset              (default 0)
     --facen-size                     Facen size in number of floats. (facens of
                                      this sizes are not regenerated when smart
                                      regeneration is enabled) (default -1)
     --max-id                         Maximum id (inclusive)
     --min-id                         Minimum id (inclusive)
     --regenerate                     What to regenerate: facens, thumbs,
                                      normalized (comma-separated). (default
                                      facens)
    
    /usr/lib/python3/dist-packages/tornado/log.py options:
    
     --log-file-max-size              max size of log files before rollover
                                      (default 100000000)
     --log-file-num-backups           number of log files to keep (default 10)
     --log-file-prefix=PATH           Path prefix for log files. Note that if you
                                      are running multiple tornado processes,
                                      log_file_prefix must be different for each
                                      of them (e.g. include the port number)
     --log-rotate-interval            The interval value of timed rotating
                                      (default 1)
     --log-rotate-mode                The mode of rotating files(time or size)
                                      (default size)
     --log-rotate-when                specify the type of TimedRotatingFileHandler
                                      interval other options:('S', 'M', 'H', 'D',
                                      'W0'-'W6') (default midnight)
     --log-to-stderr                  Send log output to stderr (colorized if
                                      possible). By default use stderr if
                                      --log_file_prefix is not set and no other
                                      logging is configured.
    
     --logging=debug|info|warning|error|none
                                      Set the Python log level. If 'none', tornado
                                      won't touch the logging configuration.
                                      (default info)
    
  2. To change detector settings, uncomment and edit the detector-related parameters in the findface-facenapi configuration file.

    sudo vi /etc/findface-facenapi.ini
    
    detector                       = 'nnd'
    ...
    
  3. To switch the face biometric model, edit the model_facen parameter in the findface-nnapi configuration file:

    sudo vi /etc/findface-nnapi.ini
    
    model_facen = apricot_320
    
  4. Configure findface-regenerate by using command line arguments as described in the help message. For example, to switch the face detector, execute from /usr/bin:

    sudo findface-regenerate --regenerate=normalized,thumbs,facens --config=/etc/findface-facenapi.ini
    

    To switch the model, execute:

    sudo findface-regenerate --regenerate=facens --config=/etc/findface-facenapi.ini
    

Copy Facens from MongoDB to Tarantool

Apply mongo2searchapi as follows:

  1. Create a backup for Tarantool.

  2. Stop Tarantool.

    sudo systemctl stop tarantool@FindFace*
    
  3. Delete snapshot .snap, xlog .xlog and fast index .idx files for all tntapi shards.

    Tip

    By default, these files are stored in the following folders:

    • Standalone instance:
      • /opt/ntech/var/lib/tarantool/default/snapshots
      • /opt/ntech/var/lib/tarantool/default/xlogs
      • /opt/ntech/var/lib/tarantool/default/index
    • Cluster instance:
      • /opt/ntech/var/lib/tarantool/shard_N/snapshots
      • /opt/ntech/var/lib/tarantool/shard_N/xlogs
      • /opt/ntech/var/lib/tarantool/shard_N/index
  4. If facens differ in size for the old and new models, update the facen size in the FindFace.start section of the Tarantool configuration file /etc/tarantool/instances.enabled/FindFace_shard_N.lua. Do so for each shard.

    sudo vi /etc/tarantool/instances.enabled/FindFace_shard_N.lua
    
    FindFace.start("127.0.0.1", 8001, {license_ntls_server="127.0.0.1:3133", facen_size = 320})
    
  5. Run mongo2searchapi on the findface-facenapi host:

    sudo python3 -m facenapi.server.tools.mongo2searchapi --config=/etc/findface-facenapi.ini
    
  6. Start Tarantool

    sudo systemctl start tarantool@FindFace*