Migrate Face Data to Different Neural Network Model

Tip

Do not hesitate to contact our experts on migration by support@ntechlab.com.

Sometimes you have to migrate the face biometric data to another neural network model, such as when you decide to update to the latest version of the product that uses a different set of neural networks.

To migrate to a different neural network model, do the following:

Warning

In the current implementation, migration doesn’t support the persons functionality. So if you have a person gallery in your system, you have to remove it manually. Only after that can you proceed to the migration procedure.

To remove the person gallery, do the following:

1. Stop the findface-security service.

   sudo systemctl stop findface-security.service
   

2. Remove the person gallery, using the cleanup utility:

   sudo findface-security cleanup --person-events-age 0
   

1. Create a backup of the Tarantool-based feature vector database in any directory of your choice, for example, /etc/findface_dump.

   Tip

   See Back Up and Recover Data Storages for details.

   mkdir -p /etc/findface_dump
   cd /etc/findface_dump
   sudo findface-storage-api-dump -config /etc/findface-sf-api.ini
   

2. Stop the findface-sf-api service.

   sudo systemctl stop findface-sf-api.service
   

3. Create new shards that will host regenerated feature vectors.

   1. Open the /etc/tarantool/instances.available/ directory and find out the number of shards by counting the number of configuration files shard-*.lua.

      Note

      There are four shards in the example below.

      cd /etc/tarantool/instances.available/
      
      ls -l
      
      shard-001.lua
      shard-002.lua
      shard-003.lua
      shard-004.lua
      

   2. Create the same number of new shards by copying the configuration files shard-*.lua.

      Note

      For convenience, the second digit in the new names is 1: shard-01*.lua.

      sudo cp shard-001.lua shard-011.lua
      sudo cp shard-002.lua shard-012.lua
      sudo cp shard-003.lua shard-013.lua
      sudo cp shard-004.lua shard-014.lua
      

   3. Modify the following lines in each new shard’s configuration file, depending on its name (shard-011, shard-012, etc., in our example):

      Old value	New value
      listen = ‘127.0.0.1:32001’	Listen = ‘127.0.0.1:32011’
      vinyl_dir = ‘/opt/ntech/var/lib/tarantool/shard-001’	vinyl_dir = ‘/opt/ntech/var/lib/tarantool/shard-011’
      work_dir = ‘/opt/ntech/var/lib/tarantool/shard-001’	work_dir = ‘/opt/ntech/var/lib/tarantool/shard-011’
      memtx_dir = ‘/opt/ntech/var/lib/tarantool/shard-001/snapshots’	memtx_dir = ‘/opt/ntech/var/lib/tarantool/shard-011/snapshots’
      wal_dir = ‘/opt/ntech/var/lib/tarantool/shard-001/xlogs’	wal_dir = ‘/opt/ntech/var/lib/tarantool/shard-011/xlogs’
      FindFace.start(“127.0.0.1”, 8101, {	FindFace.start(“127.0.0.1”, 8111, {

   4. Create symbolic links to the new shards.

      cd /etc/tarantool/instances.enabled/
      
      sudo ln -s /etc/tarantool/instances.available/shard-01*.lua /etc/tarantool/instances.enabled/
      

   5. Create directories that will host files of the new shards. Assign permissions for the created directories.

      cd /opt/ntech/var/lib/tarantool/
      
      mkdir -p shard-01{1..4}/{index,snapshots,xlogs}
      
      chown tarantool:tarantool shard-01* shard-01*/*
      

4. Open the /etc/findface-extraction-api.ini configuration file and replace the old neural network model with the new one (kiwi_320.cpu.fnk in the example).

   sudo vi /etc/findface-extraction-api.ini
   
   face: face/jackfruit_480.cpu.fnk -> face: face/kiwi_320.cpu.fnk
   

   Restart the findface-extraction-api service.

   sudo systemctl restart findface-extraction-api.service
   

5. Start the new shards.

   for i in {11..14}; do sudo systemctl start tarantool@shard-0$i; done
   

6. Create a configuration file with migration settings <migration.ini> based on the example below.

   extraction-api:
     timeouts:
       connect: 5s
       response_header: 30s
       overall: 35s
       idle_connection: 0s
     extraction-api: http://127.0.0.1:18666
   storage-api-from: # current location of the gallery
     timeouts:
       connect: 5s
       response_header: 30s
       overall: 35s
       idle_connection: 10s
     max-idle-conns-per-host: 20
     shards:
       - master: http://127.0.0.1:8101/v2/
         slave: ""
       - master: http://127.0.0.1:8102/v2/
         slave: ""
       - master: http://127.0.0.1:8103/v2/
         slave: ""
       - master: http://127.0.0.1:8104/v2/
         slave: ""
   storage-api-to:
     timeouts:
       connect: 5s
       response_header: 30s
       overall: 35s
       idle_connection: 10s
     max-idle-conns-per-host: 20
     shards:
       - master: http://127.0.0.1:8111/v2/
         slave: ""
       - master: http://127.0.0.1:8112/v2/
         slave: ""
       - master: http://127.0.0.1:8113/v2/
         slave: ""
       - master: http://127.0.0.1:8114/v2/
         slave: ""
   workers_num: 3
   faces_limit: 100
   extraction_batch_size: 8
   normalized_storage:
     type: webdav
     enabled: True
     webdav:
       upload-url: http://127.0.0.1:3333/uploads/
     s3:
       endpoint: ""
       bucket-name: ""
       access-key: ""
       secret-access-key: ""
       secure: False
       region: ""
       public-url: ""
       operation-timeout: 30
   

   In the storage-api-from section, specify the old shards to migrate the data from.

   storage-api-from: # current location of the gallery
     ...
     shards:
       - master: http://127.0.0.1:8101/v2/
         slave: ""
       - master: http://127.0.0.1:8102/v2/
         slave: ""
       - master: http://127.0.0.1:8103/v2/
         slave: ""
        - master: http://127.0.0.1:8104/v2/
     ...
   

   In the storage-api-to section, specify the new shards that will host migrated data.

   storage-api-to:
     ...
     shards:
       - master: http://127.0.0.1:8111/v2/
         slave: ""
       - master: http://127.0.0.1:8112/v2/
         slave: ""
       - master: http://127.0.0.1:8113/v2/
         slave: ""
       - master: http://127.0.0.1:8114/v2/
         slave: ""
     ...
   

7. Launch the findface-sf-api-migrate utility with the -config option and provide the <migration.ini> configuration file.

   findface-sf-api-migrate -config migration.ini
   

   Note

   The migration process can take up a significant amount of time if there are many events and dossiers in the system.

8. After the migration is complete, stop the old shards and disable their autostart in OS (do not remove them).

   for i in {01..04}; do sudo systemctl stop tarantool@shard-0$i.service ; done
   
   for i in {01..04}; do sudo systemctl disable tarantool@shard-0$i.service ; done
   

9. Open the /etc/findface-sf-api.ini configuration file and adjust the shards ports, subject to the new shards settings. Restart the findface-sf-api service.

   sudo vi /etc/findface-sf-api.ini
   
   shards:
     - master: http://127.0.0.1:8111/v2/
       slave: ""
     - master: http://127.0.0.1:8112/v2/
       slave: ""
     - master: http://127.0.0.1:8113/v2/
       slave: ""
     - master: http://127.0.0.1:8114/v2/
       slave: ""
   
   sudo systemctl start findface-sf-api.service
   

10. Import the new database structure from the tnt_schema.lua file.

    sudo findface-security make_tnt_schema | sudo tee /etc/findface-security/tnt_schema.lua
    

    See also

    Modify Feature Vector Database Structure.

11. Migrate the main database architecture from FindFace Multi to PostgreSQL, re-create user groups with predefined rights, and the first user with administrator rights.

    sudo findface-security migrate
    sudo findface-security create_groups
    sudo findface-security create_default_user
    

12. Restart the services.

    sudo systemctl restart findface-security.service
    sudo systemctl restart findface-extraction-api findface-video-worker* findface-video-manager findface-sf-api