.. _clusters: ******************************************************** Face, Body, Vehicle Clusters ******************************************************** FindFace Multi supports automatic clustering of objects of the same origin: * Face images belonging to the same person form a face cluster. * Body images belonging to the same person form a body cluster. * Images of the same vehicle constitute a vehicle cluster. Aggregated cluster galleries of faces, bodies, and vehicles are available on the :guilabel:`Clusters` tab. .. note:: If a face cluster or a body cluster matches a :ref:`person record `, such a cluster will automatically appear in that record. Similarly, a vehicle cluster will be saved to the corresponding vehicle record. .. important:: By default, the object clustering is disabled. :ref:`Enable and configure it ` via the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file. .. rubric:: In this section: .. contents:: :local: Clustering Algorithms ============================ FindFace Multi uses the following object clustering algorithms: * Real-time clustering. This clustering algorithm processes episodes to select suitable object images and clusters the selected ones. It works on the fly after an episode is closed. The real-time clustering results are dynamically displayed on the :guilabel:`Clusters` tab and in the relevant :ref:`record `. Not all episodes are used for clustering. If an episode meets all requirements (see the details :ref:`below `), the system forms a cluster in the following way: * Selects the best quality event. * Creates a new entity ``cluster event`` in the main system database :program:`PostgreSQL`. The entity contains the selected event metadata, a link to the parent episode, an object feature vector, and a thumbnail. * Searches for a matching object centroid in the ``cluster_events`` gallery of the :program:`Tarantool` feature vector database. An object centroid is a virtual feature vector averaged across all twin objects that have been detected so far (for example, a face centroid is a feature vector averaged across all face images of the same person). The system updates a matching centroid using the new event if such a centroid was found or creates a new centroid otherwise. * Scheduled clustering. This clustering algorithm works over and revises the cluster events created during the real-time clustering. This algorithm improves the cluster centroid quality as the centroid will be averaged across a more extensive array of accumulated feature vectors. The results of the scheduled clustering are displayed on the :guilabel:`Clusters` tab and in the relevant :ref:`record ` after each scheduled iteration. Use the RRULE format to define the schedule in the ``CLUSTERS_CLUSTERIZATION_SCHEDULE`` parameter at ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py``. Late-night hours are preferred as the scheduled clustering takes up a lot of CPU resources and time. .. important:: The scheduled clustering completely overwrites the cluster galleries, including ids, unless you pin specific clusters by enabling the ``CLUSTERS_AUTO_PIN_HEURISTICS`` and ``PIN_MATCHED_CLUSTERS`` parameters (see :ref:`below `). .. _enable-clustering: Enable and Configure Clustering ================================================== By default, the clustering is disabled. To enable and configure it, do the following: #. Open the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file. Find the ``Clusters configuration`` section. .. code:: sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py # -- Clusters configuration -- 'ENABLE_NIGHT_CLUSTERIZATION': False, 'ENABLE_REALTIME_CLUSTERIZATION': True, # rrule (recurrence rule) for scheduling "night" clusterization 'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0', # clusterize only selected objects types (for realtime and nightly clusterization) # available are: face, body, car 'CLUSTERIZE_OBJECT_TYPES': ['face'], # keep no more than N the best quality events in centroid (None to disable) 'CENTROID_MAX_SIZE': None, # save cluster events without emben and images (only keep thumbnail for the best event) 'LONG_LIVING_CLUSTER_EVENTS': False, # skip clusterization if unpinned cluster events count is greater than this value 'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None, # create cluster only from cluster events in current case or only from cluster events without case 'ISOLATE_CASE_CLUSTERS': True, # cluster event to cluster matching confidence threshold 'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.714, # model: [mango_320] 'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [clio] # minimum required event quality for cluster creation 'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.5, # model: [quality_fast.v1] 'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6, # model: [pedattr.quality.v0] 'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73, # model: [carattr.quality.v0] # discard cluster event if `max_centroids` similar centroids found with confidence greater than `confidence` 'FACE_CLUSTER_MAX_N_SIMILAR': {'enabled': False, 'max_centroids': 5, 'confidence': 0.714}, 'BODY_CLUSTER_MAX_N_SIMILAR': {'enabled': False, 'max_centroids': 5, 'confidence': 0.65}, # minimum required object size in pixels for cluster creation 'FACE_CLUSTER_EVENT_MIN_SIZE': 50, 'BODY_CLUSTER_EVENT_MIN_SIZE': 50, 'CAR_CLUSTER_EVENT_MIN_SIZE': 50, # minimum required number events in episode for cluster creation 'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, 'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, 'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, # age feature threshold for cluster creation 'FACE_CLUSTER_EVENT_MIN_AGE_THRESHOLD': 16, .... # pinned clusters keep their id and events after reclusterization 'CLUSTERS_AUTO_PIN_HEURISTICS': { 'face': { # pin clusters with `value` minimum cluster events 'min_events': {'enabled': True, 'value': 10}, # cluster's centroid similarity confidence is less than 'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.54}, # mango_320 # minimum average event's quality 'min_average_events_quality': {'enabled': True, 'value': 0.45}, }, 'body': {}, 'car': {}, }, # always pin clusters with matched events (not affected by heuristics above) 'PIN_MATCHED_CLUSTERS': False, ... #. Enable the real-time clustering by setting ``ENABLE_REALTIME_CLUSTERIZATION: True``. #. If necessary, enable the scheduled clustering by setting ``ENABLE_NIGHT_CLUSTERIZATION: True``. .. important:: Enabling the scheduled clustering only makes sense if the real-time clustering is enabled. Otherwise, the system won't form any new clusters, since only the real-time clustering produces **unique** cluster events. .. code:: ... # -- Clusters configuration -- 'ENABLE_NIGHT_CLUSTERIZATION': True, 'ENABLE_REALTIME_CLUSTERIZATION': True, ... #. If necessary, specify a recurrence rule (RRULE) for the scheduled clustering. If the RRULE is not specified, the clustering automatically starts at 00:00 GMT. .. tip:: See the RRULE calculator `here `_. .. code:: # rrule (recurrence rule) for scheduling clusters clusterization 'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0', #. By default, the system forms only face clusters. To enable cluster formation of bodies and vehicles, add relevant object types to the following line: .. code:: # available are: face, body, car 'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'], #. If necessary, modify the minimum number of events in the episodes used for clustering. It's 1 by default. Do so separately for each object type. .. code:: # minimum required number events in episode for cluster creation 'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3, 'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3, 'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 2, #. If necessary, modify the minimum quality of the object images used for clustering. Do so separately for each object type. .. note:: As this setting requires a high level of expertise and knowledge, we highly recommend consulting with our technical experts prior. .. code:: # minimum required event quality for cluster creation 'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.5, # model: [quality_fast.v1] 'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6, # model: [pedattr.quality.v0] 'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73, # model: [carattr.quality.v0] #. If necessary, modify the confidence threshold for matching a cluster event and a cluster. .. warning:: Be sure to consult with our experts by support@ntechlab.com before altering this parameter. .. code:: # cluster event to cluster matching confidence threshold 'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.714, # model: [mango_320] 'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [clio] #. The scheduled clustering completely overwrites all created clusters. You can "pin" specific clusters, i.e. keep them and associated cluster events, including their IDs, intact. To do so, use the following settings: .. note:: These settings are independent. Apply both if necessary. .. note:: These settings do not affect the real-time clustering. It will continue to create new cluster events for the pinned clusters. * ``CLUSTERS_AUTO_PIN_HEURISTICS``: set ``True`` or ``False`` for the following options and specify the corresponding values: .. note:: Do so for each object type if applicable. * ``min_events``: pin a cluster when the number of associated cluster events exceeds the given minimum value. * ``max_centroid_similarity_threshold``: pin a cluster if the similarity between its centroid and centroids of other clusters is less than the given threshold. If a cluster looks similar to some other clusters, chances are these clusters belong to the exact person/vehicle. In this case, the system won't pin such a cluster to have an opportunity to re-cluster it. On the contrary, dissimilar clusters will be pinned. * ``min_average_events_quality``: pin a cluster if the average quality of associated cluster events is greater than the given minimum value. .. code:: # pinned clusters keep their id and events after reclusterization 'CLUSTERS_AUTO_PIN_HEURISTICS': { 'face': { # pin clusters with `value` minimum cluster events 'min_events': {'enabled': True, 'value': 10}, # cluster's centroid similarity confidence is less then 'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.54}, # minimum average event's quality 'min_average_events_quality': {'enabled': True, 'value': 0.45}, }, 'body': {}, 'car': {}, }, * Enable the ``PIN_MATCHED_CLUSTERS`` parameter to pin matched clusters and the associated cluster events. .. code:: # always pin clusters with matched events (not affected by heuristics above) 'PIN_MATCHED_CLUSTERS': True, #. If necessary, specify the maximum number of cluster events in the clusters that remain non-pinned. After this number is reached, the scheduled clustering will be automatically disabled. .. code:: # skip clusterization if unpinned cluster events count is greater than this value 'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None, #. Restart the ``findface-multi-findface-multi-legacy-1`` container. You will see the :guilabel:`Clusters` tab appear in the FindFace Multi web interface. .. code:: sudo docker container restart findface-multi-findface-multi-legacy-1 Work with Cluster Galleries ================================= .. _view-clusters: View and Filter Clusters ---------------------------- To view the cluster galleries, navigate to the :guilabel:`Clusters` tab. |cluster_gallery_en| .. |cluster_gallery_en| image:: /_static/cluster_gallery_en.png :scale: 60% .. |cluster_gallery_ru| image:: /_static/cluster_gallery_ru.png :scale: 60% When working with the cluster galleries, the following filters may come in handy: .. note:: Some filters from the list below may be hidden, depending on which recognition features are enabled. * :guilabel:`Object`: display clusters only for faces, bodies or vehicles. * :guilabel:`Matches`: display clusters only with/without matches or any. * :guilabel:`Watch lists`: display only clusters for a selected watch list. * :guilabel:`Camera groups`: display only clusters from a selected group of cameras. * :guilabel:`Cameras`: display only clusters from a selected camera. * :guilabel:`Record name`: display only clusters with a given name. * :guilabel:`Date and time`: display only clusters formed within a certain period. * :guilabel:`First cluster event`: display only the first cluster event within a certain period. * :guilabel:`Cluster event`: display only cluster event within a certain period. * :guilabel:`ID`: display a cluster with a given ID. .. rubric:: Specific filters for face clusters * :guilabel:`Age`: display clusters with people of a given age. * :guilabel:`Beard`: filter clusters by the fact of having a beard. * :guilabel:`Emotions`: display clusters with given emotions. * :guilabel:`Gender`: display clusters with people of a given gender. * :guilabel:`Glasses`: filter clusters by the fact of wearing glasses. * :guilabel:`Liveness`: filter clusters by face liveness. * :guilabel:`Face mask`: filter clusters by the fact of wearing a face mask. .. rubric:: Specific filters for body clusters * :guilabel:`Gender by body`: display only events with people of a given gender or all events. * :guilabel:`Age by body`: display only events with people of a given age. * :guilabel:`Headwear`: display only events with a person wearing headgear of a given type: hat/cap, hood/scarf, none. * :guilabel:`Vest`: display only events with a person wearing a vest of a given color. * :guilabel:`Vest score`: display only events with a person wearing a vest within a given score. * :guilabel:`Helmet`: display only events with a person wearing a helmet of a given color. * :guilabel:`Helmet score`: display only events with a person wearing a helmet within a given score. * :guilabel:`Upper clothes color`: display only events with a person wearing a top of a given color. * :guilabel:`Lower clothes color`: display only events with a person wearing a bottom of a given color. * :guilabel:`Upper clothes type`: display only events with a person wearing upper body wear of a given specific type: jacket, coat, sleeveless, sweatshirt, T-shirt, shirt, dress. * :guilabel:`Lower body clothes`: display only events with a person wearing lower body wear of a given type: pants, nondescript, skirt, shorts. * :guilabel:`Upper body clothes`: display only events with a person wearing upper body wear of a given generalized category: long sleeves, short sleeves, no sleeve. .. rubric:: Specific filters for vehicle clusters * :guilabel:`Make`: filter vehicle events by vehicle make. * :guilabel:`Model`: filter vehicle events by vehicle model. * :guilabel:`Vehicle body type`: display only events with vehicles of a given body type: minivan, limousine. * :guilabel:`Vehicle body color`: display only events with vehicles of a given color. * :guilabel:`Country`: display only events with vehicles registered in a given country. * :guilabel:`License plate number`: display an event with a given plate number. * :guilabel:`Region`: display only events with vehicles registered in a given region. * :guilabel:`License plate color`: display only events with a given license plate color. * :guilabel:`Special vehicle`: display only events with vehicles belonging to a given type: police, fire service and emergencies ministry vehicles, gas rescue and emergency services, military, municipal vehicles, other. * :guilabel:`Vehicle category`: display only events with vehicles belonging to a given category: unknown, motorcycle, scooter, car, car with a trailer, truck, truck with a trailer, bus, articulated bus, other. * :guilabel:`Vehicle weight and body size`: display only events with vehicles of a given weight and body size. .. note:: License plate region and color are predicted for the United Arab Emirates (UAE) only. The values of these attributes will be marked as unknown for other countries. Click a cluster to see the associated cluster events. You will be redirected to the :guilabel:`Cluster events` page. Merge and Delete Clusters --------------------------------- To manually merge several clusters, select them one by one and click :guilabel:`Merge`. To delete a cluster, select it and click :guilabel:`Delete`. |cluster_merge_delete_en| .. |cluster_merge_delete_en| image:: /_static/cluster_merge_delete_en.png :scale: 60% .. |cluster_merge_delete_ru| image:: /_static/cluster_merge_delete_ru.png :scale: 60% Make Cluster Gallery Static =============================== Sometimes it's necessary to finalize the object clustering at a certain point in time and then operate with a static gallery of formed clusters. To display the :guilabel:`Clusters` tab upon the disabled clustering, do the following: #. Open the ``/opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py`` configuration file. .. code:: sudo vi /opt/findface-multi/configs/findface-multi-legacy/findface-multi-legacy.py #. Add the line ``"clusters": True`` to the ``SERVICES`` section, as shown in the example below: .. code:: ... SERVICES = { "ffsecurity": { ... "clusters": True, } ... #. Disable the real-time and scheduled clustering processes. .. code:: ... 'ENABLE_NIGHT_CLUSTERIZATION': False, 'ENABLE_REALTIME_CLUSTERIZATION': False, #. Restart the ``findface-multi-findface-multi-legacy-1`` container. .. code:: sudo docker container restart findface-multi-findface-multi-legacy-1 Manual Clustering ======================== To manually launch the clustering process, use the ``run_clusterization`` utility. You can invoke the ``run_clusterization`` help message by executing: .. code:: sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py run_clusterization --help usage: manage.py run_clusterization [-h] [--object-types OBJECT_TYPES [OBJECT_TYPES ...]] [--force] [--configuration CONFIGURATION] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] optional arguments: -h, --help show this help message and exit --object-types OBJECT_TYPES [OBJECT_TYPES ...] Clusterize selected object types. Uses CLUSTERIZE_OBJECT_TYPES from config if not provided. Allowed types: face, body, car --force Force clusterization even if CLUSTERIZATION_MAX_CLUSTER_EVENTS condition is met --configuration CONFIGURATION The name of the configuration class to load, e.g. "Development". If this isn't provided, the DJANGO_CONFIGURATION environment variable will be used. --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks. With this utility, it is possible to separately launch clustering of faces, bodies, and vehicles and perform force clustering when the maximum number of cluster events exceeds the value of the ``CLUSTERIZATION_MAX_CLUSTER_EVENTS`` parameter (see :ref:`enable-clustering`). For example, to force start face clustering, execute: .. code:: sudo docker exec -it findface-multi-findface-multi-legacy-1 /opt/findface-security/bin/python3 /tigre_prototype/manage.py run_clusterization --object-types face --force .. seealso:: * :ref:`record-index` * :ref:`interaction-analysis` * :ref:`analytics`