Migrate Feature Vectors to Different Neural Network Model

Tip

Do not hesitate to contact our experts on migration by support@ntechlab.com.

This section is about how to migrate face feature vectors to another neural network model.

Do the following:

Create a backup of the Tarantool-based feature vector database in any directory of your choice, for example, /etc/findface_dump.

Tip

See Back Up and Recover FindFace and Data for details.
```
sudo mkdir -p /etc/findface_dump
sudo cd /etc/findface_dump
sudo findface-storage-api-dump -config /etc/findface-sf-api.ini
```

Stop the findface-sf-api service.

sudo systemctl stop findface-sf-api.service

Create new shards that will host regenerated feature vectors.

Open the /etc/tarantool/instances.available/ directory and find out the number of shards by counting the number of configuration files shard-*.lua.

Note

There are four shards in the example below.
```
cd /etc/tarantool/instances.available/

ls -l

shard-001.lua
shard-002.lua
shard-003.lua
shard-004.lua
```

Create the same number of new shards by copying the configuration files shard-*.lua.

Note

For convenience, the second digit in the new names is 1: shard-01*.lua.

sudo cp shard-001.lua shard-011.lua
sudo cp shard-002.lua shard-012.lua
sudo cp shard-003.lua shard-013.lua
sudo cp shard-004.lua shard-014.lua

Modify the following lines in each new shard’s configuration file, subject to its name (shard-011, shard-012, etc., in our example):

Old value

New value

listen = ‘127.0.0.1:32001’

Listen = ‘127.0.0.1:32011’

vinyl_dir = ‘/opt/ntech/var/lib/tarantool/shard-001’

vinyl_dir = ‘/opt/ntech/var/lib/tarantool/shard-011’

work_dir = ‘/opt/ntech/var/lib/tarantool/shard-001’

work_dir = ‘/opt/ntech/var/lib/tarantool/shard-011’

memtx_dir = ‘/opt/ntech/var/lib/tarantool/shard-001/snapshots’

memtx_dir = ‘/opt/ntech/var/lib/tarantool/shard-011/snapshots’

wal_dir = ‘/opt/ntech/var/lib/tarantool/shard-001/xlogs’

wal_dir = ‘/opt/ntech/var/lib/tarantool/shard-011/xlogs’

FindFace.start(“127.0.0.1”, 8101, {

FindFace.start(“127.0.0.1”, 8111, {

Create symbolic links to the new shards.

cd /etc/tarantool/instances.enabled/

sudo ln -s /etc/tarantool/instances.available/shard-01*.lua /etc/tarantool/instances.enabled/

Create directories that will host files of the new shards. Assign permissions for the created directories.

cd /opt/ntech/var/lib/tarantool/

mkdir -p shard-01{1..4}/{index,snapshots,xlogs}

chown tarantool:tarantool shard-01* shard-01*/*

Open the /etc/findface-extraction-api configuration file and replace the extraction model with the new one in the face_emben parameter.

sudo vi /etc/findface-extraction-api.ini

extractors:
  ...
  models:
    ...
    face_emben: face/<new_model_face>.gpu.fnk

Restart the findface-extraction-api service.

sudo systemctl restart findface-extraction-api.service

Start the new shards.

for i in {11..14}; do sudo systemctl start tarantool@shard-0$i; done

Create a configuration file with migration settings <migration.ini> based on the example below.

extraction-api:
  timeouts:
    connect: 5s
    response_header: 30s
    overall: 35s
    idle_connection: 0s
  extraction-api: http://127.0.0.1:18666
storage-api-from: # current location of the gallery
  timeouts:
    connect: 5s
    response_header: 30s
    overall: 35s
    idle_connection: 10s
  max-idle-conns-per-host: 20
  shards:
    - master: http://127.0.0.1:8101/v2/
      slave: ""
    - master: http://127.0.0.1:8102/v2/
      slave: ""
    - master: http://127.0.0.1:8103/v2/
      slave: ""
    - master: http://127.0.0.1:8104/v2/
      slave: ""
storage-api-to:
  timeouts:
    connect: 5s
    response_header: 30s
    overall: 35s
    idle_connection: 10s
  max-idle-conns-per-host: 20
  shards:
    - master: http://127.0.0.1:8111/v2/
      slave: ""
    - master: http://127.0.0.1:8112/v2/
      slave: ""
    - master: http://127.0.0.1:8113/v2/
      slave: ""
    - master: http://127.0.0.1:8114/v2/
      slave: ""
workers_num: 3
faces_limit: 100
extraction_batch_size: 8
normalized_storage:
  type: webdav
  enabled: True
  webdav:
    upload-url: http://127.0.0.1:3333/uploads/
  s3:
    endpoint: ""
    bucket-name: ""
    access-key: ""
    secret-access-key: ""
    secure: False
    region: ""
    public-url: ""
    operation-timeout: 30

In the storage-api-from section, specify the old shards to migrate the data from.

storage-api-from: # current location of the gallery
  ...
  shards:
    - master: http://127.0.0.1:8101/v2/
      slave: ""
    - master: http://127.0.0.1:8102/v2/
      slave: ""
    - master: http://127.0.0.1:8103/v2/
      slave: ""
     - master: http://127.0.0.1:8104/v2/
  ...

In the storage-api-to section, specify the new shards that will host migrated data.

storage-api-to:
  ...
  shards:
    - master: http://127.0.0.1:8111/v2/
      slave: ""
    - master: http://127.0.0.1:8112/v2/
      slave: ""
    - master: http://127.0.0.1:8113/v2/
      slave: ""
    - master: http://127.0.0.1:8114/v2/
      slave: ""
  ...

Launch the findface-sf-api-migrate utility with the -config option and provide the <migration.ini> configuration file.
```
findface-sf-api-migrate -config migration.ini
```
Note

The migration process can take up a significant amount of time if there are many events and records in the system.

After the migration is complete, stop the old shards and disable their autostart in OS (do not remove them).

for i in {01..04}; do sudo systemctl stop tarantool@shard-0$i.service ; done

for i in {01..04}; do sudo systemctl disable tarantool@shard-0$i.service ; done

Open the /etc/findface-sf-api.ini configuration file and adjust the shards ports, subject to the new shards settings. Restart the findface-sf-api service.

sudo vi /etc/findface-sf-api.ini

shards:
  - master: http://127.0.0.1:8111/v2/
    slave: ""
  - master: http://127.0.0.1:8112/v2/
    slave: ""
  - master: http://127.0.0.1:8113/v2/
    slave: ""
  - master: http://127.0.0.1:8114/v2/
    slave: ""

sudo systemctl start findface-sf-api.service

Import the database structure from the tnt_schema.lua file.
```
sudo findface-security make_tnt_schema | sudo tee /etc/findface-security/tnt_schema.lua
```
See also

Modify Feature Vector Database Structure.
Migrate the main database architecture from FindFace to PostgreSQL, re-create predefined user roles and the first administrator.
```
sudo findface-security migrate
sudo findface-security create_groups
sudo findface-security create_default_user
```

Restart the services.

sudo systemctl restart findface-security.service
sudo systemctl restart findface-extraction-api findface-video-worker* findface-video-manager findface-sf-api