Modify Biometric Database Structure

Sometimes it may be necessary to apply a new structural schema to your Tarantool-based biometric database, for example, when updating to the latest version of the product, or when you want to enhance the default database structure with additional parameters, advanced face metadata, and so on.

In this section:

About Database Structure

In FindFace Security, the database structure is set via the /etc/ffsecurity/tnt_schema.lua file.

The structure is created as a set of fields. Each field is described with the following parameters:

• id: field id;
• name: field name, must be the same as the name of a relevant face parameter;
• field_type: data type;
• default: field default value. If a default value exceeds ‘1e14 – 1’, use a string data type to specify it, for example, "123123.." instead of 123123...

You can find the default tnt_schema.lua file below:

scheme = {
    -- internal.normalized_id:
    {
        default = '',
        field_type = 'string',
        id = 1,
        name = 'normalized_id',
    },
    -- internal.feat:
    {
        default = '',
        field_type = 'string',
        id = 2,
        name = 'feat',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged:
    {
        default = 0,
        field_type = 'unsigned',
        id = 3,
        name = 'm:acknowledged',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_by:
    {
        default = 0,
        field_type = 'unsigned',
        id = 4,
        name = 'm:acknowledged_by',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 5,
     name = 'm:acknowledged_date',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:acknowledged_reaction:
    {
        default = '',
        field_type = 'string',
        id = 6,
        name = 'm:acknowledged_reaction',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:camera:
    {
        default = 0,
        field_type = 'unsigned',
        id = 7,
        name = 'm:camera',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:camera_group:
    {
        default = 0,
        field_type = 'unsigned',
        id = 8,
        name = 'm:camera_group',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:confidence:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 9,
        name = 'm:confidence',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:created_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 10,
        name = 'm:created_date',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:episode:
    {
        default = 0,
        field_type = 'unsigned',
        id = 11,
        name = 'm:episode',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_age:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 12,
        name = 'm:f_age',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_beard_class:
    {
        default = '',
        field_type = 'string',
        id = 13,
        name = 'm:f_beard_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_beard_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 14,
        name = 'm:f_beard_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_emotions_class:
    {
        default = '',
        field_type = 'string',
        id = 15,
        name = 'm:f_emotions_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_emotions_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 16,
        name = 'm:f_emotions_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_gender_class:
    {
        default = '',
        field_type = 'string',
        id = 17,
        name = 'm:f_gender_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_gender_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 18,
        name = 'm:f_gender_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_glasses_class:
    {
        default = '',
        field_type = 'string',
        id = 19,
        name = 'm:f_glasses_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_glasses_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 20,
        name = 'm:f_glasses_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_liveness_class:
    {
        default = '',
        field_type = 'string',
        id = 21,
        name = 'm:f_liveness_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_liveness_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 22,
        name = 'm:f_liveness_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_race_class:
    {
        default = '',
        field_type = 'string',
        id = 23,
        name = 'm:f_race_class',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:f_race_score:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 24,
        name = 'm:f_race_score',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:thumbnail:
    {
        default = '',
        field_type = 'string',
        id = 25,
        name = 'm:thumbnail',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame:
    {
        default = '',
        field_type = 'string',
        id = 26,
        name = 'm:frame',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_bottom:
    {
        default = 0,
        field_type = 'unsigned',
        id = 27,
        name = 'm:frame_coords_bottom',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_left:
    {
        default = 0,
        field_type = 'unsigned',
        id = 28,
        name = 'm:frame_coords_left',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_right:
    {
        default = 0,
        field_type = 'unsigned',
        id = 29,
        name = 'm:frame_coords_right',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:frame_coords_top:
    {
        default = 0,
        field_type = 'unsigned',
        id = 30,
        name = 'm:frame_coords_top',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:pk:
    {
        default = 0,
        field_type = 'unsigned',
        id = 31,
        name = 'm:pk',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched:
    {
        default = 0,
        field_type = 'unsigned',
        id = 32,
        name = 'm:matched',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_dossier:
    {
        default = 0,
        field_type = 'unsigned',
        id = 33,
        name = 'm:matched_dossier',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_face:
    {
        default = 0,
        field_type = 'unsigned',
        id = 34,
        name = 'm:matched_face',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:matched_lists:
    {
        default = {},
        field_type = 'set[unsigned]',
        id = 35,
        name = 'm:matched_lists',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:normalized_photo:
    {
        default = '',
        field_type = 'string',
        id = 36,
        name = 'm:normalized_photo',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:quality:
    {
        default = "10000000000000000000",
        field_type = 'unsigned',
        id = 37,
        name = 'm:quality',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:scores:
    {
        default = '',
        field_type = 'string',
        id = 38,
        name = 'm:scores',
    },
    -- <class 'ffsecurity.entities_tnt.event.models.ListEvent'>.m:video_source:
    {
        default = 0,
        field_type = 'unsigned',
        id = 39,
        name = 'm:video_source',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:dossier:
    {
        default = 0,
        field_type = 'unsigned',
        id = 40,
        name = 'm:dossier',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:modified_date:
    {
        default = 0,
        field_type = 'unsigned',
        id = 41,
        name = 'm:modified_date',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:source_photo:
    {
        default = '',
        field_type = 'string',
        id = 42,
        name = 'm:source_photo',
    },
    -- <class 'ffsecurity.entities_tnt.dossier_face.models.DossierFace'>.m:source_photo_name:
    {
        default = '',
        field_type = 'string',
        id = 43,
        name = 'm:source_photo_name',
    },
}
-- Fields referenced by multiple models: m:frame_coords_left, m:pk, m:frame_coords_top, m:created_date, m:frame_coords_right, m:thumbnail, m:frame_coords_bottom

Structure Modification

To modify the database structure, do the following:

1. Stop the findface-security service.

   sudo systemctl stop findface-security.service
   

2. Create a backup of the Tarantool-based biometric database in any directory of your choice, for example, /tmp/dump.

   Tip

   See Backup and Restore Data Storages for details.

   mkdir -p /tmp/dump
   cd /tmp/dump
   sudo findface-storage-api-dump -config /etc/findface-sf-api.ini
   

3. Prepare the tnt_schema.lua file containing the new database structure.

4. Modify the database structure by applying the new tnt_schema.lua file.

   sudo findface-security make_tnt_schema | sudo tee /etc/ffsecurity/tnt_schema.lua
   

5. Open the Tarantool configuration file. Make sure that there is a line dofile("/etc/ffsecurity/tnt_schema.lua") before the FindFace.start section and meta_scheme=meta_scheme is defined in the FindFace.start parameters.

   sudo vi /etc/tarantool/instances.enabled/<shard_00N>.lua
   
   dofile("/etc/ffsecurity/tnt_schema.lua")
   
   FindFace.start("127.0.0.1", 8101, {
       license_ntls_server="127.0.0.1:3133",
       facen_size=480,
       meta_scheme = meta_scheme
    })
   

6. Remove the Tarantool database (default database or shards).

   sudo rm -f /opt/ntech/var/lib/tarantool/default/{index,snapshots,xlogs}/*
   
   sudo rm -f /opt/ntech/var/lib/tarantool/shard-001/{index,snapshots,xlogs}/*
   ...
   sudo rm -f /opt/ntech/var/lib/tarantool/shard-00N/{index,snapshots,xlogs}/*
   

7. Restore the Tarantool database from the backup.

   Important

   If some fields were removed from the new database structure, you have to first manually delete the corresponding data from the backup copy.

   cd /tmp/dump
   for x in *.json; do curl -X POST "http://127.0.0.1:18411/v2/galleries/${x%%.json}"; done
   for x in *.json; do sudo findface-storage-api-restore -config /etc/findface-sf-api.ini < "$x"; done
   

8. Restart the findface-security service.

   sudo systemctl stop findface-security.service