Modify Biometric Database Structure

Sometimes it may be necessary to apply a new structural schema to your Tarantool-based biometric database, for example, when updating to the latest version of the product, or when you want to enhance the default database structure with additional parameters, advanced face metadata, and so on.

In this section:

About Database Structure

In FindFace Security, the database structure is set via the /etc/findface-security/tnt_schema.lua file.

The structure is created as a set of fields. Each field is described with the following parameters:

• id: field id;
• name: field name, must be the same as the name of a relevant face parameter;
• field_type: data type;
• default: field default value. If a default value exceeds ‘1e14 – 1’, use a string data type to specify it, for example, "123123.." instead of 123123...

You can find the default tnt_schema.lua file below:

meta_scheme = {
    -- internal.normalized_id:
    {
        default = '',
        field_type = 'string',
        id = 1,
        name = 'normalized_id',
    },
    -- internal.feat:
    {
        default = '',
        field_type = 'string',
        id = 2,
        name = 'feat',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged:
    {
        default = 0,
        field_type = 'unsigned',
        id = 3,
        name = 'm:acknowledged',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_by:
    {
        default = 0,
        field_type = 'unsigned',
        id = 4,
        name = 'm:acknowledged_by',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 5,
        name = 'm:acknowledged_date',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_reaction:
    {
        default = '',
        field_type = 'string',
        id = 6,
        name = 'm:acknowledged_reaction',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:camera:
    {
        default = 0,
        field_type = 'unsigned',
        id = 7,
        name = 'm:camera',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:camera_group:
    {
        default = 0,
        field_type = 'unsigned',
        id = 8,
        name = 'm:camera_group',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:confidence:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 9,
        name = 'm:confidence',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:created_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 10,
        name = 'm:created_date',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:episode:
    {
        default = 0,
        field_type = 'unsigned',
        id = 11,
        name = 'm:episode',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:episode_open:
    {
        default = 0,
        field_type = 'unsigned',
        id = 12,
        name = 'm:episode_open',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_age:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 13,
        name = 'm:f_age',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_beard_class:
    {
        default = '',
        field_type = 'string',
        id = 14,
        name = 'm:f_beard_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_beard_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 15,
        name = 'm:f_beard_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_emotions_class:
    {
        default = '',
        field_type = 'string',
        id = 16,
        name = 'm:f_emotions_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_emotions_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 17,
        name = 'm:f_emotions_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_gender_class:
    {
        default = '',
        field_type = 'string',
        id = 18,
        name = 'm:f_gender_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_gender_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 19,
        name = 'm:f_gender_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_glasses_class:
    {
        default = '',
        field_type = 'string',
        id = 20,
        name = 'm:f_glasses_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_glasses_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 21,
        name = 'm:f_glasses_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_liveness_class:
    {
        default = '',
        field_type = 'string',
        id = 22,
        name = 'm:f_liveness_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_liveness_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 23,
        name = 'm:f_liveness_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_medmask_class:
    {
        default = '',
        field_type = 'string',
        id = 24,
        name = 'm:f_medmask_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_medmask_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 25,
        name = 'm:f_medmask_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_race_class:
    {
        default = '',
        field_type = 'string',
        id = 26,
        name = 'm:f_race_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_race_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 27,
        name = 'm:f_race_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:thumbnail:
    {
        default = '',
        field_type = 'string',
        id = 28,
        name = 'm:thumbnail',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame:
    {
        default = '',
        field_type = 'string',
        id = 29,
        name = 'm:frame',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_bottom:
    {
        default = 0,
        field_type = 'unsigned',
        id = 30,
        name = 'm:frame_coords_bottom',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_left:
    {
        default = 0,
        field_type = 'unsigned',
        id = 31,
        name = 'm:frame_coords_left',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_right:
    {
        default = 0,
        field_type = 'unsigned',
        id = 32,
        name = 'm:frame_coords_right',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_top:
    {
        default = 0,
        field_type = 'unsigned',
        id = 33,
        name = 'm:frame_coords_top',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:pk:
    {
        default = 0,
        field_type = 'unsigned',
        id = 34,
        name = 'm:pk',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched:
    {
        default = 0,
        field_type = 'unsigned',
        id = 35,
        name = 'm:matched',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_dossier:
    {
        default = 0,
        field_type = 'unsigned',
        id = 36,
        name = 'm:matched_dossier',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_face:
    {
        default = 0,
        field_type = 'unsigned',
        id = 37,
        name = 'm:matched_face',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_lists:
    {
        default = {},
        field_type = 'set[unsigned]',
        id = 38,
        name = 'm:matched_lists',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:normalized_photo:
    {
        default = '',
        field_type = 'string',
        id = 39,
        name = 'm:normalized_photo',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:quality:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 40,
        name = 'm:quality',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:scores:
    {
        default = '',
        field_type = 'string',
        id = 41,
        name = 'm:scores',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:temperature:
    {
        default = "0",
        field_type = 'unsigned',
        id = 42,
        name = 'm:temperature',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:video_source:
    {
        default = 0,
        field_type = 'unsigned',
        id = 43,
        name = 'm:video_source',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:dossier:
    {
        default = 0,
        field_type = 'unsigned',
        id = 44,
        name = 'm:dossier',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:modified_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 45,
        name = 'm:modified_date',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:source_photo:
    {
        default = '',
        field_type = 'string',
        id = 46,
        name = 'm:source_photo',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:source_photo_name:
    {
        default = '',
        field_type = 'string',
        id = 47,
        name = 'm:source_photo_name',
    },
}
-- Fields referenced by multiple models: m:thumbnail, m:frame_coords_left, m:frame_coords_right, m:pk, m:frame_coords_bottom, m:created_date, m:frame_coords_top
meta_indexes = {'m:episode', 'm:episode_open', 'm:video_source'}

Structure Modification

To modify the database structure, do the following:

1. Stop the findface-security service.

   sudo systemctl stop findface-security.service
   

2. Create a backup of the Tarantool-based biometric database in any directory of your choice, for example, /etc/findface_dump.

   Tip

   See Backup and Restore Data Storages for details.

   mkdir -p /etc/findface_dump
   cd /etc/findface_dump
   sudo findface-storage-api-dump -config /etc/findface-sf-api.ini
   

3. Prepare the tnt_schema.lua file containing the new database structure.

4. Modify the database structure by applying the new tnt_schema.lua file.

   sudo findface-security make_tnt_schema | sudo tee /etc/findface-security/tnt_schema.lua
   

5. Navigate to the directory with Tarantool configuration file(s) /etc/tarantool/instances.enabled/. For each shard, make sure that there is a line dofile("/etc/findface-security/tnt_schema.lua") before the FindFace.start section and meta_scheme and meta_indexes are defined in the FindFace.start parameters.

   sudo vi /etc/tarantool/instances.enabled/<shard_00N>.lua
   
   dofile("/etc/findface-security/tnt_schema.lua")
   
   FindFace.start("127.0.0.1", 8001, {
       license_ntls_server="127.0.0.1:3133",
       meta_indexes=meta_indexes,
       meta_scheme = meta_scheme
   })
   

6. Purge data from all the directories relevant to active shards.

   sudo rm /opt/ntech/var/lib/tarantool/shard-*/{index,snapshots,xlogs}/*
   

7. Restore the Tarantool database from the backup.

   Important

   If some fields were removed from the new database structure, you have to first manually delete the corresponding data from the backup copy.

   cd /tmp/dump
   for x in *.json; do curl -X POST "http://127.0.0.1:18411/v2/galleries/${x%%.json}"; done
   for x in *.json; do sudo findface-storage-api-restore -config /etc/findface-sf-api.ini < "$x"; done
   

8. Start the findface-security service.

   sudo systemctl start findface-security.service
   

See also

Dossier Face Custom Metadata in Tarantool