Face, Body, Vehicle Clusters

FindFace Multi supports automatic clustering of objects of the same origin:

Face images belonging to the same person form a face cluster.
Body images belonging to the same person form a body cluster.
Images of the same vehicle constitute a vehicle cluster.

Aggregated cluster galleries of faces, bodies, and vehicles are available on the Clusters tab.

Note

If a face cluster or a body cluster matches a person record, such a cluster will automatically appear in that record. Similarly, a vehicle cluster will be saved to the corresponding vehicle record.

Important

By default, the object clustering is disabled. Enable and configure it via the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py configuration file.

In this section:

Clustering Algorithms
Enable and Configure Clustering
Work with Cluster Galleries
- View and Filter Clusters
- Merge and Delete Clusters
Make Cluster Gallery Static
Manual Clustering

Clustering Algorithms 

FindFace Multi uses the following object clustering algorithms:

Real-time clustering. This clustering algorithm processes episodes to select suitable object images and clusters the selected ones. It works on the fly after an episode is closed. The real-time clustering results are dynamically displayed on the Clusters tab and in the relevant record.

Not all episodes are used for clustering. If an episode meets all requirements (see the details below), the system forms a cluster in the following way:
- Selects the best quality event.
- Creates a new entity cluster event in the main system database PostgreSQL. The entity contains the selected event metadata, a link to the parent episode, an object feature vector, and a thumbnail.
- Searches for a matching object centroid in the cluster_events gallery of the Tarantool feature vector database. An object centroid is a virtual feature vector averaged across all twin objects that have been detected so far (for example, a face centroid is a feature vector averaged across all face images of the same person). The system updates a matching centroid using the new event if such a centroid was found or creates a new centroid otherwise.
Scheduled clustering. This clustering algorithm works over and revises the cluster events created during the real-time clustering. This algorithm improves the cluster centroid quality as the centroid will be averaged across a more extensive array of accumulated feature vectors. The results of the scheduled clustering are displayed on the Clusters tab and in the relevant record after each scheduled iteration.

Use the RRULE format to define the schedule in the CLUSTERS_CLUSTERIZATION_SCHEDULE parameter at /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py. Late-night hours are preferred as the scheduled clustering takes up a lot of CPU resources and time.

Important

The scheduled clustering completely overwrites the cluster galleries, including ids, unless you pin specific clusters by enabling the CLUSTERS_AUTO_PIN_HEURISTICS and PIN_MATCHED_CLUSTERS parameters (see below).

Enable and Configure Clustering 

By default, the clustering is disabled. To enable and configure it, do the following:

Open the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py configuration file. Find the Clusters configuration section.

sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py

 # -- Clusters configuration --
 'ENABLE_NIGHT_CLUSTERIZATION': False,
 'ENABLE_REALTIME_CLUSTERIZATION': True,
 # rrule (recurrence rule) for scheduling "night" clusterization
 'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0',
 # clusterize only selected objects types (for realtime and nightly clusterization)
 # available are: face, body, car
 'CLUSTERIZE_OBJECT_TYPES': ['face'],
 # keep no more than N the best quality events in centroid (None to disable)
 'CENTROID_MAX_SIZE': None,
 # save cluster events without emben and images (only keep thumbnail for the best event)
 'LONG_LIVING_CLUSTER_EVENTS': False,
 # skip clusterization if unpinned cluster events count is greater than this value
 'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None,
 # create cluster only from cluster events in current case or only from cluster events without case
 'ISOLATE_CASE_CLUSTERS': True,
 # cluster event to cluster matching confidence threshold
 'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.714,  # model: [mango_320]
 'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [durga]
 # minimum required event quality for cluster creation
 'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.5,  # model: [quality_fast.v1]
 'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6,  # model: [pedattr.quality.v0]
 'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73,  # model: [carattr.quality.v0]
 # discard cluster event if `max_centroids` similar centroids found with confidence greater than `confidence`
 'FACE_CLUSTER_MAX_N_SIMILAR': {'enabled': False, 'max_centroids': 5, 'confidence': 0.714},
 'BODY_CLUSTER_MAX_N_SIMILAR': {'enabled': False, 'max_centroids': 5, 'confidence': 0.65},
 # minimum required object size in pixels for cluster creation
 'FACE_CLUSTER_EVENT_MIN_SIZE': 50,
 'BODY_CLUSTER_EVENT_MIN_SIZE': 50,
 'CAR_CLUSTER_EVENT_MIN_SIZE': 50,
 # minimum required number events in episode for cluster creation
 'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
 'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
 'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
 # age feature threshold for cluster creation
 'FACE_CLUSTER_EVENT_MIN_AGE_THRESHOLD': 16,
  ....
 # pinned clusters keep their id and events after reclusterization
 'CLUSTERS_AUTO_PIN_HEURISTICS': {
    'face': {
        # pin clusters with `value` minimum cluster events
        'min_events': {'enabled': True, 'value': 10},
        # cluster's centroid similarity confidence is less than
        'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.54},  # mango_320
        # minimum average event's quality
        'min_average_events_quality': {'enabled': True, 'value': 0.45},
    },
    'body': {},
    'car': {},
    },
 # always pin clusters with matched events (not affected by heuristics above)
 'PIN_MATCHED_CLUSTERS': False,
 ...

Enable the real-time clustering by setting ENABLE_REALTIME_CLUSTERIZATION: True.
If necessary, enable the scheduled clustering by setting ENABLE_NIGHT_CLUSTERIZATION: True.

Important

Enabling the scheduled clustering only makes sense if the real-time clustering is enabled. Otherwise, the system won’t form any new clusters, since only the real-time clustering produces unique cluster events.
```
...
# -- Clusters configuration --
    'ENABLE_NIGHT_CLUSTERIZATION': True,
    'ENABLE_REALTIME_CLUSTERIZATION': True,
...
```
If necessary, specify a recurrence rule (RRULE) for the scheduled clustering. If the RRULE is not specified, the clustering automatically starts at 00:00 GMT.

Tip

See the RRULE calculator here.
```
# rrule (recurrence rule) for scheduling clusters clusterization
'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0',
```
By default, the system forms only face clusters. To enable cluster formation of bodies and vehicles, add relevant object types to the following line:
```
# available are: face, body, car
'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'],
```

If necessary, modify the minimum number of events in the episodes used for clustering. It’s 1 by default. Do so separately for each object type.

# minimum required number events in episode for cluster creation
'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3,
'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3,
'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 2,

If necessary, modify the minimum quality of the object images used for clustering. Do so separately for each object type.

Note

As this setting requires a high level of expertise and knowledge, we highly recommend consulting with our technical experts prior.

# minimum required event quality for cluster creation
'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.5,  # model: [quality_fast.v1]
'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6,  # model: [pedattr.quality.v0]
'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73,  # model: [carattr.quality.v0]

If necessary, modify the confidence threshold for matching a cluster event and a cluster.

Warning

Be sure to consult with our experts by support@ntechlab.com before altering this parameter.

# cluster event to cluster matching confidence threshold
'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.714,  # model: [mango_320]
'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [durga]

The scheduled clustering completely overwrites all created clusters. You can “pin” specific clusters, i.e. keep them and associated cluster events, including their IDs, intact. To do so, use the following settings:

Note

These settings are independent. Apply both if necessary.

Note

These settings do not affect the real-time clustering. It will continue to create new cluster events for the pinned clusters.
- CLUSTERS_AUTO_PIN_HEURISTICS: set True or False for the following options and specify the corresponding values:
  
  Note
  
  Do so for each object type if applicable.
  - min_events: pin a cluster when the number of associated cluster events exceeds the given minimum value.
  - max_centroid_similarity_threshold: pin a cluster if the similarity between its centroid and centroids of other clusters is less than the given threshold. If a cluster looks similar to some other clusters, chances are these clusters belong to the exact person/vehicle. In this case, the system won’t pin such a cluster to have an opportunity to re-cluster it. On the contrary, dissimilar clusters will be pinned.
  - min_average_events_quality: pin a cluster if the average quality of associated cluster events is greater than the given minimum value.
```
 # pinned clusters keep their id and events after reclusterization
 'CLUSTERS_AUTO_PIN_HEURISTICS': {
     'face': {
         # pin clusters with `value` minimum cluster events
         'min_events': {'enabled': True, 'value': 10},
         # cluster's centroid similarity confidence is less then
         'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.54},
         # minimum average event's quality
         'min_average_events_quality': {'enabled': True, 'value': 0.45},
     },
     'body': {},
     'car': {},
},
```
- Enable the PIN_MATCHED_CLUSTERS parameter to pin matched clusters and the associated cluster events.
```
 # always pin clusters with matched events (not affected by heuristics above)
'PIN_MATCHED_CLUSTERS': True,
```
If necessary, specify the maximum number of cluster events in the clusters that remain non-pinned. After this number is reached, the scheduled clustering will be automatically disabled.
```
# skip clusterization if unpinned cluster events count is greater than this value
'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None,
```
Restart all FindFace Multi containers. You will see the Clusters tab appear in the FindFace Multi web interface.
```
cd /opt/findface-multi/

sudo docker-compose restart
```

Work with Cluster Galleries 

View and Filter Clusters 

To view the cluster galleries, navigate to the Clusters tab.

When working with the cluster galleries, the following filters may come in handy:

Note

Some filters from the list below may be hidden, depending on which recognition features are enabled.

Object: display clusters only for faces, bodies or vehicles.
Matches: display clusters only with/without matches or any.
Watch lists: display only clusters for a selected watch list.
Camera groups: display only clusters from a selected group of cameras.
Cameras: display only clusters from a selected camera.
Record name: display only clusters with a given name.
Date and time: display only clusters formed within a certain period.
First cluster event: display only the first cluster event within a certain period.
Cluster event: display only cluster event within a certain period.
ID: display a cluster with a given ID.

Specific filters for face clusters

Age: display clusters with people of a given age.
Beard: filter clusters by the fact of having a beard.
Emotions: display clusters with given emotions.
Gender: display clusters with people of a given gender.
Glasses: filter clusters by the fact of wearing glasses.
Liveness: filter clusters by face liveness.
Face mask: filter clusters by the fact of wearing a face mask.

Specific filters for body clusters

Gender by body: display only events with people of a given gender or all events.
Age by body: display only events with people of a given age.
Headwear: display only events with a person wearing headgear of a given type: hat/cap, hood/scarf, none.
Vest: display only events with a person wearing a vest of a given color.
Vest score: display only events with a person wearing a vest within a given score.
Helmet: display only events with a person wearing a helmet of a given color.
Helmet score: display only events with a person wearing a helmet within a given score.
Upper clothes color: display only events with a person wearing a top of a given color.
Lower clothes color: display only events with a person wearing a bottom of a given color.
Upper clothes type: display only events with a person wearing upper body wear of a given specific type: jacket, coat, sleeveless, sweatshirt, T-shirt, shirt, dress.
Lower body clothes: display only events with a person wearing lower body wear of a given type: pants, nondescript, skirt, shorts.
Upper body clothes: display only events with a person wearing upper body wear of a given generalized category: long sleeves, short sleeves, no sleeve.

Specific filters for vehicle clusters

Make: filter vehicle events by vehicle make.
Model: filter vehicle events by vehicle model.
Vehicle body type: display only events with vehicles of a given body type: minivan, limousine.
Vehicle body color: display only events with vehicles of a given color.
Country: display only events with vehicles registered in a given country.
License plate number: display an event with a given plate number.
Region: display only events with vehicles registered in a given region.
License plate color: display only events with a given license plate color.
Special vehicle: display only events with vehicles belonging to a given type: police, fire service and emergencies ministry vehicles, gas rescue and emergency services, military, municipal vehicles, other.
Vehicle category: display only events with vehicles belonging to a given category: unknown, motorcycle, scooter, car, car with a trailer, truck, truck with a trailer, bus, articulated bus, other.
Vehicle weight and body size: display only events with vehicles of a given weight and body size.

Note

License plate region and color are predicted for the United Arab Emirates (UAE) only. The values of these attributes will be marked as unknown for other countries.

Click a cluster to see the associated cluster events. You will be redirected to the Cluster events page.

Merge and Delete Clusters 

To manually merge several clusters, select them one by one and click Merge.

Note

By default, vehicle clusters are formed only by license plate. In this case, it is impossible to merge two vehicle clusters with different license plates.

To delete a cluster, select it and click Delete.

Make Cluster Gallery Static 

Sometimes it’s necessary to finalize the object clustering at a certain point in time and then operate with a static gallery of formed clusters.

To display the Clusters tab upon the disabled clustering, do the following:

Open the /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py configuration file.
```
sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py
```

Add the line "clusters": True to the SERVICES section, as shown in the example below:

...
SERVICES = {
    "ffsecurity": {
        ...
        "clusters": True,
    }
...

Disable the real-time and scheduled clustering processes.

...
'ENABLE_NIGHT_CLUSTERIZATION': False,
'ENABLE_REALTIME_CLUSTERIZATION': False,

Restart all FindFace Multi containers.

cd /opt/findface-multi/

sudo docker-compose restart

Manual Clustering 

To manually launch the clustering process, use the run_clusterization utility.

You can invoke the run_clusterization help message by executing:

 sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py run_clusterization --help

 usage: manage.py run_clusterization [-h]
                                  [--object-types OBJECT_TYPES [OBJECT_TYPES ...]]
                                  [--force] [--configuration CONFIGURATION]
                                  [--version] [-v {0,1,2,3}]
                                  [--settings SETTINGS]
                                  [--pythonpath PYTHONPATH] [--traceback]
                                  [--no-color] [--force-color]
                                  [--skip-checks]

 optional arguments:
-h, --help            show this help message and exit
--object-types OBJECT_TYPES [OBJECT_TYPES ...]
                      Clusterize selected object types. Uses
                      CLUSTERIZE_OBJECT_TYPES from config if not provided.
                      Allowed types: face, body, car
--force               Force clusterization even if
                      CLUSTERIZATION_MAX_CLUSTER_EVENTS condition is met
--configuration CONFIGURATION
                      The name of the configuration class to load, e.g.
                      "Development". If this isn't provided, the
                      DJANGO_CONFIGURATION environment variable will be
                      used.
--version             show program's version number and exit
-v {0,1,2,3}, --verbosity {0,1,2,3}
                      Verbosity level; 0=minimal output, 1=normal output,
                      2=verbose output, 3=very verbose output
--settings SETTINGS   The Python path to a settings module, e.g.
                      "myproject.settings.main". If this isn't provided, the
                      DJANGO_SETTINGS_MODULE environment variable will be
                      used.
--pythonpath PYTHONPATH
                      A directory to add to the Python path, e.g.
                      "/home/djangoprojects/myproject".
--traceback           Raise on CommandError exceptions
--no-color            Don't colorize the command output.
--force-color         Force colorization of the command output.
--skip-checks         Skip system checks.

With this utility, it is possible to separately launch clustering of faces, bodies, and vehicles and perform force clustering when the maximum number of cluster events exceeds the value of the CLUSTERIZATION_MAX_CLUSTER_EVENTS parameter (see Enable and Configure Clustering). For example, to force start face clustering, execute:

sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py run_clusterization --object-types face --force

Face, Body, Vehicle Clusters

Clustering Algorithms

Enable and Configure Clustering

Work with Cluster Galleries

View and Filter Clusters

Merge and Delete Clusters

Make Cluster Gallery Static

Manual Clustering

Clustering Algorithms 

Enable and Configure Clustering 

Work with Cluster Galleries 

View and Filter Clusters 

Merge and Delete Clusters 

Make Cluster Gallery Static 

Manual Clustering 