.. _clusters: ******************************************************** Face, Body, Car Clusters ******************************************************** FindFace Multi supports automatic clustering of objects of the same origin: * Face images belonging to the same person form a face cluster. * Body images belonging to the same person form a body cluster. * Images of the same car constitute a car cluster. Aggregated cluster galleries of faces, bodies, and cars are available on the :guilabel:`Clusters` tab. .. note:: If a face cluster or a body cluster matches with a :ref:`person card `, such a cluster will automatically appear in that card. Similarly, a car cluster will be saved to the corresponding car card. .. important:: By default, the object clustering is disabled. :ref:`Enable and configure it ` via the ``/etc/findface-security/config.py`` configuration file. .. rubric:: In this section: .. contents:: :local: Clustering Algorithms ============================ FindFace Multi uses the following object clustering algorithms: * Real-time clustering. This clustering algorithm processes episodes to select suitable object images and clusters the selected ones. It works on the fly after an episode is closed. The real-time clustering results are dynamically displayed on the :guilabel:`Clusters` tab and in the relevant :ref:`card `. Not all episodes are used for clustering. If an episode meets all requirements (see the details :ref:`below `), the system forms a cluster in the following way: * Selects the best quality event. * Creates a new entity ``cluster event`` in the main system database :program:`PostgreSQL`. The entity contains the selected event metadata, a link to the parent episode, object feature vector, and thumbnail. * Searches for a matching object centroid in the ``cluster_events`` gallery of the :program:`Tarantool` feature vector database. An object centroid is a virtual feature vector averaged across all twin objects that have been detected so far (for example, a face centroid is a feature vector averaged across all face images of the same person). The system updates a matching centroid using the new event if such a centroid was found or creates a new centroid otherwise. * Scheduled clustering. This clustering algorithm works over and revises the cluster events created during the real-time clustering. This algorithm improves the cluster centroid quality as the centroid will be averaged across a more extensive array of accumulated feature vectors. The results of the scheduled clustering are displayed on the :guilabel:`Clusters` tab and in the relevant :ref:`card ` after each scheduled iteration. Use the RRULE format to define the schedule in the ``CLUSTERS_CLUSTERIZATION_SCHEDULE`` parameter at ``/etc/findface-security/config.py``. Late-night hours are preferred as the scheduled clustering takes up a lot of CPU resources and time. .. important:: The scheduled clustering completely overwrites the cluster galleries, including ids, unless you pin specific clusters by enabling the ``CLUSTERS_AUTO_PIN_HEURISTICS`` and ``PIN_MATCHED_CLUSTERS`` parameters (see :ref:`below `). .. _enable-clustering: Enable and Configure Clustering ================================================== By default, the clustering is disabled. To enable and configure it, do the following: #. Open the ``/etc/findface-security/config.py`` configuration file. Find the ``Clusters configuration`` section. .. code:: sudo vi /etc/findface-security/config.py # -- Clusters configuration -- 'ENABLE_NIGHT_CLUSTERIZATION': True, 'ENABLE_REALTIME_CLUSTERIZATION': True, # rrule (recurrence rule) for scheduling clusters clusterization 'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0', # clusterize only selected objects types (for realtime and nightly clusterization) # available are: face, body, car 'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'], # skip clusterization if unpinned cluster events count is greater than this value 'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None, # cluster event to cluster matching confidence threshold 'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.723, # model: [kiwi_320] 'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [andariel] 'CAR_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [alonso] # minimum required event quality for cluster creation 'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.45, # model: [quality.v1] 'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6, # model: [pedattr.quality.v0] 'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73, # model: [carattr.quality.v0] # minimum required number events in episode for cluster creation 'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, 'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, 'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 1, # coefficient of dependence of the clustering threshold on the person event's quality 'FACE_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1, 'BODY_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1, 'CAR_CLUSTER_SOFT_CLUSTERIZATION_COEFFICIENT': 0.1, # match with clusters when no card objects matches found 'MATCH_CLUSTERS': False, # create auto cards for unmatched clusters 'CREATE_AUTO_CARDS': False, # pinned clusters keep their id and events after reclusterization 'CLUSTERS_AUTO_PIN_HEURISTICS': { 'face': { # pin clusters with `value` minimum cluster events 'min_events': {'enabled': True, 'value': 10}, # cluster's centroid similarity confidence is less then 'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.55}, # minimum average event's quality 'min_average_events_quality': {'enabled': True, 'value': 0.45}, }, 'body': {}, 'car': {}, }, # always pin clusters with matched events (not affected by heuristics above) 'PIN_MATCHED_CLUSTERS': False, #. Enable the real-time clustering by setting ``ENABLE_REALTIME_CLUSTERIZATION: True``. #. If necessary, enable the scheduled clustering by setting ``ENABLE_NIGHT_CLUSTERIZATION: True``. .. important:: Enabling the scheduled clustering only makes sense if the real-time clustering is enabled. Otherwise, the system won't form any new clusters, since only the real-time clustering produces **unique** cluster events. .. code:: ... # -- Clusters configuration -- 'ENABLE_NIGHT_CLUSTERIZATION': True, 'ENABLE_REALTIME_CLUSTERIZATION': True, ... #. If necessary, specify a recurrence rule (RRULE) for the scheduled clustering. If the RRULE is not specified, the clustering automatically starts at 00:00 GMT. .. tip:: See the RRULE calculator `here `_. .. code:: # rrule (recurrence rule) for scheduling clusters clusterization 'CLUSTERIZATION_SCHEDULE': 'RRULE:FREQ=DAILY;INTERVAL=1;WKST=MO;BYHOUR=0;BYMINUTE=0', #. By default, the system forms only face clusters. To enable cluster formation of bodies and cars, add relevant object types to the following line: .. code:: # available are: face, body, car 'CLUSTERIZE_OBJECT_TYPES': ['face','body','car'], #. If necessary, modify the minimum number of events in the episodes used for clustering. It's 1 by default. Do so separately for each object type. .. code:: # minimum required number events in episode for cluster creation 'FACE_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3, 'BODY_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 3, 'CAR_CLUSTER_EVENT_MIN_EPISODE_EVENTS': 2, #. If necessary, modify the minimum quality of the object images used for clustering. Do so separately for each object type. .. note:: As this setting requires a high level of expertise and knowledge, we highly recommend consulting with our technical experts prior. .. code:: # minimum required event quality for cluster creation 'FACE_CLUSTER_EVENT_MIN_QUALITY': 0.45, # model: [quality.v1] 'BODY_CLUSTER_EVENT_MIN_QUALITY': 0.6, # model: [pedattr.quality.v0] 'CAR_CLUSTER_EVENT_MIN_QUALITY': 0.73, # model: [carattr.quality.v0] #. If necessary, modify the confidence threshold for matching a cluster event and a cluster. .. warning:: Be sure to consult with our experts by support@ntechlab.com before altering this parameter. .. code:: # cluster event to cluster matching confidence threshold 'FACE_CLUSTER_CONFIDENCE_THRESHOLD': 0.723, # model: [kiwi_320] 'BODY_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [andariel] 'CAR_CLUSTER_CONFIDENCE_THRESHOLD': 0.65, # model: [alonso] #. After an object is detected in the video, the system searches for its matches through the card index. Set ``'MATCH_CLUSTERS': True`` to enable the system to search for matches through the cluster gallery as well if it didn't find a matching card in the card index. In this case, the system will be notifying you whenever an object matches a previously created cluster. .. code:: # match with clusters when no card objects matches found 'MATCH_CLUSTERS': True, #. Set ``'CREATE_AUTO_CARDS': True`` to automatically create a new card for a cluster that has no matches in the card index. See :ref:`auto-cards` for details. .. code:: # create auto cards for unmatched clusters 'CREATE_AUTO_CARDS': True, #. The scheduled clustering completely overwrites all created clusters. You can "pin" specific clusters, i.e. keep them and associated cluster events, including their IDs, intact. To do so, use the following settings: .. note:: These settings are independent. Apply both if necessary. .. note:: These settings do not affect the real-time clustering. It will continue to create new cluster events for the pinned clusters. * ``CLUSTERS_AUTO_PIN_HEURISTICS``: set ``True`` or ``False`` for the following options and specify the corresponding values: .. note:: Do so for each object type if applicable. * ``min_events``: pin a cluster when the number of associated cluster events exceeds the given minimum value. * ``max_centroid_similarity_threshold``: pin a cluster if the similarity between its centroid and centroids of other clusters is less than the given threshold. If a cluster looks similar to some other clusters, chances are these clusters belong to the exact person/car. In this case, the system won't pin such a cluster to have an opportunity to re-cluster it. On the contrary, dissimilar clusters will be pinned. * ``min_average_events_quality``: pin a cluster if the average quality of associated cluster events is greater than the given minimum value. .. code:: # pinned clusters keep their id and events after reclusterization 'CLUSTERS_AUTO_PIN_HEURISTICS': { 'face': { # pin clusters with `value` minimum cluster events 'min_events': {'enabled': True, 'value': 10}, # cluster's centroid similarity confidence is less then 'max_centroid_similarity_threshold': {'enabled': True, 'value': 0.55}, # minimum average event's quality 'min_average_events_quality': {'enabled': True, 'value': 0.45}, }, 'body': {}, 'car': {}, }, * Enable the ``PIN_MATCHED_CLUSTERS`` parameter to pin matched clusters and the associated cluster events. .. code:: # always pin clusters with matched events (not affected by heuristics above) 'PIN_MATCHED_CLUSTERS': True, #. If necessary, specify the maximum number of cluster events in the clusters that remain non-pinned. After this number is reached, the scheduled clustering will be automatically disabled. .. code:: # skip clusterization if unpinned cluster events count is greater than this value 'CLUSTERIZATION_MAX_CLUSTER_EVENTS': None, #. Restart the ``findface-security`` service. You will see the :guilabel:`Clusters` tab appear in the FindFace Multi web interface. .. code:: sudo systemctl restart findface-security.service Work with Cluster Galleries ================================= .. _view-clusters: View and Filter Clusters ---------------------------- To view the cluster galleries, navigate to the :guilabel:`Clusters` tab. |cluster_gallery_en| .. |cluster_gallery_en| image:: /_static/cluster_gallery_en.png :scale: 45% .. |cluster_gallery_ru| image:: /_static/cluster_gallery_ru.png :scale: 45% When working with the cluster galleries, the following filters may come in handy: .. note:: Some filters from the list below may be hidden, depending on which recognition features are enabled. * :guilabel:`Card`: display clusters only for a selected card. * :guilabel:`Cameras`: display only clusters from a selected camera. * :guilabel:`Camera groups`: display only clusters from a selected group of cameras. * :guilabel:`Watch lists`: display only clusters for a selected watch list. * :guilabel:`Start`, :guilabel:`End`: display only clusters formed within a certain period. * :guilabel:`ID`: display a cluster with a given ID. * :guilabel:`Has card`: display clusters subject to whether they have an associated card, auto-generated or completed one (see the :ref:`auto-cards` section for a detailed explanation). .. rubric:: Specific filters for face clusters * :guilabel:`Age`: display clusters with people of a given age. * :guilabel:`Gender`: display clusters with people of a given gender. * :guilabel:`Emotions`: display clusters with given emotions. * :guilabel:`Glasses`: filter clusters by the fact of wearing glasses. * :guilabel:`Beard`: filter clusters by the fact of having a beard. * :guilabel:`Liveness`: filter clusters by face liveness. * :guilabel:`Face mask`: filter clusters by the fact of wearing a face mask. .. rubric:: Specific filters for body clusters * :guilabel:`Headwear`: display only clusters with a person wearing headgear of a given type: hat/cap, hood/headscarf, none. * :guilabel:`Upper body clothes`: display only clusters with a person wearing upper body wear of a given generalized category: long sleeves, short sleeves, no sleeve. * :guilabel:`Upper clothes type`: display only clusters with a person wearing upper body wear of a given specific type: jacket, coat, sleeveless vest, sweatshirt, T-shirt, shirt, dress. * :guilabel:`Upper clothes color`: display only clusters with a person wearing a top of a given color. * :guilabel:`Lower body clothes`: display only clusters with a person wearing lower body wear of a given type: pants, skirt, shorts, obscured. * :guilabel:`Lower clothes color`: display only clusters with a person wearing a bottom of a given color. .. rubric:: Specific filters for car clusters * :guilabel:`Car body style`: display only clusters with cars of a given body style. * :guilabel:`Car color`: display only clusters with cars of a given color. * :guilabel:`Make`: filter clusters by car make. * :guilabel:`Model`: filter clusters by car model. * :guilabel:`Special vehicle`: display only clusters with cars belonging to a given type: police, ambulance, rescue service, taxi, other. Merge and Delete Clusters --------------------------------- To manually merge several clusters, select them one by one and click :guilabel:`Merge`. To delete a cluster, select it and click :guilabel:`Delete`. |cluster_merge_delete_en| .. |cluster_merge_delete_en| image:: /_static/cluster_merge_delete_en.png :scale: 45% .. |cluster_merge_delete_ru| image:: /_static/cluster_merge_delete_ru.png :scale: 45% View and Filter Associated Cluster Events ------------------------------------------- Click a cluster to see the associated cluster events. You will be redirected to the :guilabel:`Cluster events` page with the corresponding cluster identifier set in the :guilabel:`Cluster ID` filter. |cluster_associated_en| .. |cluster_associated_en| image:: /_static/cluster_associated_en.png :scale: 45% .. |cluster_associated_ru| image:: /_static/cluster_associated_ru.png :scale: 45% The set of filters to work with the cluster events are similar to :ref:`that ` for the clusters. Make Cluster Gallery Static =============================== Sometimes it's necessary to finalize the object clustering at a certain point in time and then operate with a static gallery of formed clusters. To display the :guilabel:`Clusters` tab upon the disabled clustering, do the following: #. Open the ``/etc/findface-security/config.py`` configuration file. .. code:: sudo vi /etc/findface-security/config.py #. Add the line ``"clusters": True`` to the ``SERVICES`` section, as shown in the example below: .. code:: ... SERVICES = { "ffsecurity": { ... "clusters": True, } ... #. Disable the real-time and scheduled clustering processes. .. code:: ... 'ENABLE_NIGHT_CLUSTERIZATION': False, 'ENABLE_REALTIME_CLUSTERIZATION': False, #. Restart the ``findface-security`` service. .. code:: sudo systemctl restart findface-security.service Manual Clustering ======================== To manually launch the clustering process, use the ``run_clusterization`` utility. You can invoke the ``run_clusterization`` help message by executing: .. code:: sudo findface-security run_clusterization --help usage: findface-security run_clusterization [-h] [--object-types OBJECT_TYPES [OBJECT_TYPES ...]] [--force] [--configuration CONFIGURATION] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color] [--skip-checks] optional arguments: -h, --help show this help message and exit --object-types OBJECT_TYPES [OBJECT_TYPES ...] Clusterize selected object types. Uses CLUSTERIZE_OBJECT_TYPES from config if not provided. Allowed types: face, body, car --force Force clusterization even if CLUSTERIZATION_MAX_CLUSTER_EVENTS condition is met --configuration CONFIGURATION The name of the configuration class to load, e.g. "Development". If this isn't provided, the DJANGO_CONFIGURATION environment variable will be used. --version show program's version number and exit -v {0,1,2,3}, --verbosity {0,1,2,3} Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output --settings SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Raise on CommandError exceptions --no-color Don't colorize the command output. --force-color Force colorization of the command output. --skip-checks Skip system checks. With this utility, it is possible to separately launch clustering of faces, bodies, and cars and perform force clustering when the maximum number of cluster events exceeds the value of the ``CLUSTERIZATION_MAX_CLUSTER_EVENTS`` parameter (see :ref:`enable-clustering`). For example, to force start face clustering, execute: .. code:: sudo findface-security run_clusterization --object-types face --force .. seealso:: * :ref:`Configuration file of findface-security ` * :ref:`cards` * :ref:`interaction-analysis` * :ref:`analytics`