Face, Body, Car Clusters

FindFace Multi supports automatic clustering of objects of the same origin:

Face images belonging to the same person form a face cluster.
Body images belonging to the same person form a body cluster.
Images of the same car constitute a car cluster.

Aggregated cluster galleries of faces, bodies, and cars are available on the Clusters tab.

Note

If a face cluster or a body cluster matches with a person card, such a cluster will automatically appear in that card. Similarly, a car cluster will be saved to the corresponding car card.

Important

By default, the object clustering is disabled. Enable and configure it via the /etc/findface-security/config.py configuration file.

In this section:

Clustering Algorithms
Enable and Configure Clustering
Work with Cluster Galleries
Make Cluster Gallery Static
Manual Clustering

Clustering Algorithms 

FindFace Multi uses the following object clustering algorithms:

Real-time clustering. This clustering algorithm processes episodes to select suitable object images and clusters the selected ones. It works on the fly after an episode is closed. The real-time clustering results are dynamically displayed on the Clusters tab and in the relevant card.

Not all episodes are used for clustering. If an episode meets all requirements (see the details below), the system forms a cluster in the following way:
- Selects the best quality event.
- Creates a new entity cluster event in the main system database PostgreSQL. The entity contains the selected event metadata, a link to the parent episode, object feature vector, and thumbnail.
- Searches for a matching object centroid in the cluster_events gallery of the Tarantool feature vector database. An object centroid is a virtual feature vector averaged across all twin objects that have been detected so far (for example, a face centroid is a feature vector averaged across all face images of the same person). The system updates a matching centroid using the new event if such a centroid was found or creates a new centroid otherwise.
Scheduled clustering. This clustering algorithm works over and revises the cluster events created during the real-time clustering. This algorithm improves the cluster centroid quality as the centroid will be averaged across a more extensive array of accumulated feature vectors. The results of the scheduled clustering are displayed on the Clusters tab and in the relevant card after each scheduled iteration.

Use the RRULE format to define the schedule in the CLUSTERS_CLUSTERIZATION_SCHEDULE parameter at /etc/findface-security/config.py. Late-night hours are preferred as the scheduled clustering takes up a lot of CPU resources and time.

Important

The scheduled clustering completely overwrites the cluster galleries, including ids, unless you pin specific clusters by enabling the CLUSTERS_AUTO_PIN_HEURISTICS and PIN_MATCHED_CLUSTERS parameters (see below).

Enable and Configure Clustering 

By default, the clustering is disabled. To enable and configure it, do the following:

Open the /etc/findface-security/config.py configuration file. Find the Clusters configuration section.

sudo vi /etc/findface-security/config.py


# -- Clusters configuration --
    'ENABLE_NIGHT_CLUSTERIZATION': True,
    'ENABLE_REALTIME_CLUSTERIZATION': True,
    # rrule (recurrence rule) for scheduling clusters clusterization
    'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0',
    # clusterize only selected objects types (for realtime and nightly clusterization)
    # available are: face, body, car
    'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'],
    # skip clusterization if unpinned cluster events count is greater than this value
    'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None,
    # cluster event to cluster matching confidence threshold
    'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.723,  # model: [kiwi_320]
    'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [andariel]
    'CAR_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [alonso]
    # minimum required event quality for cluster creation
    'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.45,  # model: [quality.v1]
    'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6,  # model: [pedattr.quality.v0]
    'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73,  # model: [carattr.quality.v0]
    # minimum required number events in episode for cluster creation
    'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
    'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
    'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1,
    # coefficient of dependence of the clustering threshold on the person event's quality
    'FACE_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1,
    'BODY_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1,
    'CAR_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1,
    # match with clusters when no card objects matches found
    'MATCH_CLUSTERS': False,
    # create auto cards for unmatched clusters
    'CREATE_AUTO_CARDS': False,
    # pinned clusters keep their id and events after reclusterization
    'CLUSTERS_AUTO_PIN_HEURISTICS': {
        'face': {
            # pin clusters with `value` minimum cluster events
            'min_events': {'enabled': True, 'value': 10},
            # cluster's centroid similarity confidence is less then
            'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.55},
            # minimum average event's quality
            'min_average_events_quality': {'enabled': True, 'value': 0.45},
        },
        'body': {},
        'car': {},
    },
    # always pin clusters with matched events (not affected by heuristics above)
    'PIN_MATCHED_CLUSTERS': False,

Enable the real-time clustering by setting ENABLE_REALTIME_CLUSTERIZATION: True.
If necessary, enable the scheduled clustering by setting ENABLE_NIGHT_CLUSTERIZATION: True.

Important

Enabling the scheduled clustering only makes sense if the real-time clustering is enabled. Otherwise, the system won’t form any new clusters, since only the real-time clustering produces unique cluster events.
```
...
# -- Clusters configuration --
    'ENABLE_NIGHT_CLUSTERIZATION': True,
    'ENABLE_REALTIME_CLUSTERIZATION': True,
...
```
If necessary, specify a recurrence rule (RRULE) for the scheduled clustering. If the RRULE is not specified, the clustering automatically starts at 00:00 GMT.

Tip

See the RRULE calculator here.
```
# rrule (recurrence rule) for scheduling clusters clusterization
'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0',
```
By default, the system forms only face clusters. To enable cluster formation of bodies and cars, add relevant object types to the following line:
```
# available are: face, body, car
'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'],
```

If necessary, modify the minimum number of events in the episodes used for clustering. It’s 1 by default. Do so separately for each object type.

# minimum required number events in episode for cluster creation
'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3,
'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3,
'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 2,

If necessary, modify the minimum quality of the object images used for clustering. Do so separately for each object type.

Note

As this setting requires a high level of expertise and knowledge, we highly recommend consulting with our technical experts prior.

# minimum required event quality for cluster creation
'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.45,  # model: [quality.v1]
'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6,  # model: [pedattr.quality.v0]
'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73,  # model: [carattr.quality.v0]

If necessary, modify the confidence threshold for matching a cluster event and a cluster.

Warning

Be sure to consult with our experts by support@ntechlab.com before altering this parameter.

# cluster event to cluster matching confidence threshold
'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.723,  # model: [kiwi_320]
'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [andariel]
'CAR_CLUSTER_CONFIDENCE_THRESHOLD': 0.65,  # model: [alonso]

After an object is detected in the video, the system searches for its matches through the card index. Set 'MATCH_CLUSTERS': True to enable the system to search for matches through the cluster gallery as well if it didn’t find a matching card in the card index. In this case, the system will be notifying you whenever an object matches a previously created cluster.
```
# match with clusters when no card objects matches found
'MATCH_CLUSTERS': True,
```
Set 'CREATE_AUTO_CARDS': True to automatically create a new card for a cluster that has no matches in the card index. See Auto-Generated and Completed Cards for details.
```
# create auto cards for unmatched clusters
'CREATE_AUTO_CARDS': True,
```
The scheduled clustering completely overwrites all created clusters. You can “pin” specific clusters, i.e. keep them and associated cluster events, including their IDs, intact. To do so, use the following settings:

Note

These settings are independent. Apply both if necessary.

Note

These settings do not affect the real-time clustering. It will continue to create new cluster events for the pinned clusters.
- CLUSTERS_AUTO_PIN_HEURISTICS: set True or False for the following options and specify the corresponding values:
  
  Note
  
  Do so for each object type if applicable.
  - min_events: pin a cluster when the number of associated cluster events exceeds the given minimum value.
  - max_centroid_similarity_threshold: pin a cluster if the similarity between its centroid and centroids of other clusters is less than the given threshold. If a cluster looks similar to some other clusters, chances are these clusters belong to the exact person/car. In this case, the system won’t pin such a cluster to have an opportunity to re-cluster it. On the contrary, dissimilar clusters will be pinned.
  - min_average_events_quality: pin a cluster if the average quality of associated cluster events is greater than the given minimum value.
```
 # pinned clusters keep their id and events after reclusterization
 'CLUSTERS_AUTO_PIN_HEURISTICS': {
     'face': {
         # pin clusters with `value` minimum cluster events
         'min_events': {'enabled': True, 'value': 10},
         # cluster's centroid similarity confidence is less then
         'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.55},
         # minimum average event's quality
         'min_average_events_quality': {'enabled': True, 'value': 0.45},
     },
     'body': {},
     'car': {},
},
```
- Enable the PIN_MATCHED_CLUSTERS parameter to pin matched clusters and the associated cluster events.
```
 # always pin clusters with matched events (not affected by heuristics above)
'PIN_MATCHED_CLUSTERS': True,
```
If necessary, specify the maximum number of cluster events in the clusters that remain non-pinned. After this number is reached, the scheduled clustering will be automatically disabled.
```
# skip clusterization if unpinned cluster events count is greater than this value
'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None,
```
Restart the findface-security service. You will see the Clusters tab appear in the FindFace Multi web interface.
```
sudo systemctl restart findface-security.service
```

Work with Cluster Galleries 

View and Filter Clusters 

To view the cluster galleries, navigate to the Clusters tab.

When working with the cluster galleries, the following filters may come in handy:

Note

Some filters from the list below may be hidden, depending on which recognition features are enabled.

Card: display clusters only for a selected card.
Cameras: display only clusters from a selected camera.
Camera groups: display only clusters from a selected group of cameras.
Watch lists: display only clusters for a selected watch list.
Start, End: display only clusters formed within a certain period.
ID: display a cluster with a given ID.
Has card: display clusters subject to whether they have an associated card, auto-generated or completed one (see the Auto-Generated and Completed Cards section for a detailed explanation).

Specific filters for face clusters

Age: display clusters with people of a given age.
Gender: display clusters with people of a given gender.
Emotions: display clusters with given emotions.
Glasses: filter clusters by the fact of wearing glasses.
Beard: filter clusters by the fact of having a beard.
Liveness: filter clusters by face liveness.
Face mask: filter clusters by the fact of wearing a face mask.

Specific filters for body clusters

Headwear: display only clusters with a person wearing headgear of a given type: hat/cap, hood/headscarf, none.
Upper body clothes: display only clusters with a person wearing upper body wear of a given generalized category: long sleeves, short sleeves, no sleeve.
Upper clothes type: display only clusters with a person wearing upper body wear of a given specific type: jacket, coat, sleeveless vest, sweatshirt, T-shirt, shirt, dress.
Upper clothes color: display only clusters with a person wearing a top of a given color.
Lower body clothes: display only clusters with a person wearing lower body wear of a given type: pants, skirt, shorts, obscured.
Lower clothes color: display only clusters with a person wearing a bottom of a given color.

Specific filters for car clusters

Car body style: display only clusters with cars of a given body style.
Car color: display only clusters with cars of a given color.
Make: filter clusters by car make.
Model: filter clusters by car model.
Special vehicle: display only clusters with cars belonging to a given type: police, ambulance, rescue service, taxi, other.

Merge and Delete Clusters 

To manually merge several clusters, select them one by one and click Merge.

To delete a cluster, select it and click Delete.

View and Filter Associated Cluster Events 

Click a cluster to see the associated cluster events. You will be redirected to the Cluster events page with the corresponding cluster identifier set in the Cluster ID filter.

The set of filters to work with the cluster events are similar to that for the clusters.

Make Cluster Gallery Static 

Sometimes it’s necessary to finalize the object clustering at a certain point in time and then operate with a static gallery of formed clusters.

To display the Clusters tab upon the disabled clustering, do the following:

Open the /etc/findface-security/config.py configuration file.
```
sudo vi /etc/findface-security/config.py
```

Add the line "clusters": True to the SERVICES section, as shown in the example below:

...
SERVICES = {
    "ffsecurity": {
        ...
        "clusters": True,
    }
...

Disable the real-time and scheduled clustering processes.

...
'ENABLE_NIGHT_CLUSTERIZATION': False,
'ENABLE_REALTIME_CLUSTERIZATION': False,

Restart the findface-security service.

sudo systemctl restart findface-security.service

Manual Clustering 

To manually launch the clustering process, use the run_clusterization utility.

You can invoke the run_clusterization help message by executing:

sudo findface-security run_clusterization --help
usage: findface-security run_clusterization [-h]
                                            [--object-types OBJECT_TYPES [OBJECT_TYPES ...]]
                                            [--force]
                                            [--configuration CONFIGURATION]
                                            [--version] [-v {0,1,2,3}]
                                            [--settings SETTINGS]
                                            [--pythonpath PYTHONPATH]
                                            [--traceback] [--no-color]
                                            [--force-color] [--skip-checks]

optional arguments:
  -h, --help            show this help message and exit
  --object-types OBJECT_TYPES [OBJECT_TYPES ...]
                        Clusterize selected object types. Uses
                        CLUSTERIZE_OBJECT_TYPES from config if not provided.
                        Allowed types: face, body, car
  --force               Force clusterization even if
                        CLUSTERIZATION_MAX_CLUSTER_EVENTS condition is met
  --configuration CONFIGURATION
                        The name of the configuration class to load, e.g.
                        "Development". If this isn't provided, the
                        DJANGO_CONFIGURATION environment variable will be
                        used.
  --version             show program's version number and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Raise on CommandError exceptions
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.

With this utility, it is possible to separately launch clustering of faces, bodies, and cars and perform force clustering when the maximum number of cluster events exceeds the value of the CLUSTERIZATION_MAX_CLUSTER_EVENTS parameter (see Enable and Configure Clustering). For example, to force start face clustering, execute:

sudo findface-security run_clusterization --object-types face --force