Video object detection: findface-video-manager and findface-video-worker

Note

The findface-video-worker is delivered in a CPU-accelerated (findface-video-worker-cpu) and a GPU-accelerated (findface-video-worker-gpu) packages.

In this section:

Functions of findface-video-manager

The findface-video-manager service is the part of the video object detection module that is used for managing the video object detection functionality.

The findface-video-manager service interfaces with findface-video-worker as follows:

  • It supplies findface-video-worker with settings and the list of to-be-processed video streams. To do so, it issues a so-called job, a video processing task that contains configuration settings and stream data.

  • In a distributed system, it distributes video streams (jobs) across vacant findface-video-worker instances.

Note

The configuration settings passed via jobs have priority over the /etc/findface-video-manager.conf configuration file.

The findface-video-manager service functioning requires ETCD, third-party software that implements a distributed key-value store for findface-video-manager. In the FindFace core, ETCD is used as a coordination service, providing the video object detector with fault tolerance.

Functionality:

  • allows for configuring video object detection parameters

  • allows for managing the list of to-be-processed video streams

Functions of findface-video-worker

The findface-video-worker service (on CPU/GPU) is the part of the video object detection module, that recognizes objects in the video. It can work with both live streams and files, and supports most video formats and codecs that can be decoded by FFmpeg.

The findface-video-worker service interfaces with the findface-video-manager and findface-facerouter services as follows:

  • By request, findface-video-worker gets a job with settings and the list of to-be-processed video streams from findface-video-manager.

  • The findface-video-worker posts extracted normalized object images, along with the full frames and meta data (such as bbox, camera ID and detection time) to the findface-facerouter service for further processing.

Note

In FindFace Multi, the findface-facerouter functions are performed by findface-security.

Functionality:

  • detects objects in the video,

  • extracts normalized object images,

  • searches for the best object snapshot,

  • snapshot deduplication (only one snapshot per object detection event).

When processing a video, findface-video-worker consequently uses the following algorithms:

  • Motion detection. Used to reduce resource consumption. Only when the motion detector recognizes the motion of certain intensity that the object tracker can be triggered.

  • Object tracking. The object tracker traces, detects, and captures objects in the video. It can simultaneously be working with several objects. It also searches for the best object snapshot using the embedded neural network. After the best object snapshot is found, it is posted to findface-facerouter.

The best object snapshot can be found in one of the following modes:

  • Real-time

  • Offline

Real-Time Mode

In the real-time mode, findface-video-worker posts an object on-the-fly after it appears in the camera field. The following posting options are available:

  • If realtime_post_every_interval: true, the object tracker searches for the best object snapshot within each time period equal to realtime_post_interval and posts it to findface-facerouter.

  • If realtime_post_every_interval: false, the object tracker searches for the best face snapshot dynamically:

    1. First, the object tracker estimates whether the quality of an object snapshot exceeds a pre-defined internal threshold. If so, the snapshot is posted to findface-facerouter.

    2. The threshold value increases after each post. Each time the object tracker gets a higher quality snapshot of the same object, it is posted.

    3. When the object disappears from the camera field, the threshold value resets to default.

  • If realtime_post_first_immediately: true, the object tracker doesn’t wait for the first realtime_post_interval to complete and posts the first object from a track immediately after it passes through the quality, size, and ROI filters. The way the subsequent postings are sent depends on the realtime_post_every_interval value. If realtime_post_first_immediately: false, the object tracker posts the first object after the first realtime_post_interval completes.

Offline Mode

The offline mode is less storage intensive than the real-time one as in this mode findface-video-worker posts only one snapshot per track but of the highest quality. In this mode, the object tracker buffers a video stream with an object until the object disappears from the camera field. Then the object tracker picks up the best object snapshot from the buffered video and posts it to findface-facerouter.

Configure Video Object Detection

The video object detector configuration is done through the following configuration files:

  1. The findface-video-manager configuration file /etc/findface-video-manager.conf. You can find its default content here

When configuring findface-video-manager, refer to the following parameters:

Option

Description

etcd -> endpoints

IP address and port of the etcd service. Default value: 127.0.0.1:2379.

ntls -> enabled

If true, findface-video-manager will send a job to findface-video-worker only if the total number of processed cameras does not exceed the allowed number of cameras from the license. Default value: false.

ntls -> url

IP address and port of the findface-ntls host. Default value: http://127.0.0.1:3185/.

router_url

IP address and port of the findface-facerouter host to receive detected faces from findface-video-worker. In FindFace Multi, findface-facerouter functions are performed by findface-security. Default value: http://127.0.0.1:18820/v0/frame.

play_speed

If less than zero, the speed is not limited. In other cases, the stream is read with the given play_speed. Not applicable for live streams.

disable_drops

Enables posting all appropriate objects without drops. By default, if findface-video-worker does not have enough resources to process all frames with objects, it drops some of them. If this option is active, findface-video-worker puts odd frames on the waiting list to process them later. Default value: false.

imotion_threshold

Minimum motion intensity to be detected by the motion detector. The threshold value is to be fitted empirically. Empirical units: zero and positive rational numbers. Milestones: 0 = detector disabled, 0.002 = default value, 0.05 = minimum intensity is too high to detect motion.

router_timeout_ms

Timeout for a findface-facerouter (or findface-security in the standard FindFace Multi configuration) response to a findface-video-worker API request, in milliseconds. If the timeout has expired, the system will log an error. Default value: 15000.

router_verify_ssl

Enables a https certificate verification when findface-video-worker and findface-facerouter (or findface-security in the standard FindFace Multi configuration) interact over https. Default value: true. If false, a self-signed certificate can be accepted.

router_headers

Additional header fields in a request when posting an object: [“key = value”]. Default value: headers not specified.

router_body

Additional body fields in a request body when posting an object: [“key = value”]. Default value: body fields not specified.

ffmpeg_params

List of a video stream ffmpeg options with their values as a key=value array: [“rtsp_transpotr=tcp”, .., “ss=00:20:00”]. Check out the FFmpeg web site for the full list of options. Default value: options not specified.

ffmpeg_format

Pass FFMPEG format (mxg, flv, etc.) if it cannot be detected automatically.

use_stream_timestamp

If true, retrieve and post timestamps from a video stream. If false, post the actual date and time.

start_stream_timestamp

Add the specified number of seconds to timestamps from a stream.

rot

Enables detecting and tracking objects only inside a clipping rectangle WxH+X+Y. You can use this option to reduce findface-video-worker load. Default value: rectangle not specified.

video_transform

Change a video frame orientation right after decoding. Values (case insensitive, JPEG Exif Orientation Tag in brackets): None (1), FlipHorizontal (2), Rotate180 (3), FlipVertical (4), Transpose (5), Rotate90 (6), Transverse (7), Rotate270 (8). Default value: not specified.

The following parameters are available for configuration for each detector type (face, body, car):

Option

Description

filter_min_quality

Minimum threshold value for an object image quality. Default value: subject to the object type. Do not change the default value without consulting with our technical experts (support@ntechlab.com).

filter_min_size

Minimum size of an object in pixels. Undersized objects are not posted. Default value: 1.

filter_max_size

Maximum size of an object in pixels. Oversized objects are not posted. Default value: 8192.

roi

Enable posting objects detected only inside a region of interest WxH+X+Y. Default value: region not specified.

fullframe_crop_rot

Crop posted full frames by ROT. Default value: false.

fullframe_use_png

Send full frames in PNG and not in JPEG as set by default. Do not enable this parameter without supervision from our team as it can affect the entire system functioning. Default value: false (send in JPEG).

jpeg_quality

Quality of an original frame JPEG compression, in percents. Default value: 95%.

overall_only

Enables the offline mode for the best object search. Default value: true (CPU), false (GPU).

realtime_post_first_immediately

Enable posting an object image right after it appears in a camera field of view (real-time mode). Default value: false.

realtime_post_interval

Only for the real-time mode. Defines the time period in seconds within which the object tracker picks up the best snapshot and posts it to findface-facerouter. Default value: 1.

realtime_post_every_interval

Only for the realtime mode. Post best snapshots obtained within each realtime_post_interval time period. If false, search for the best snapshot dynamically and send snapshots in order of increasing quality. Default value: false.

track_interpolate_bboxes

Interpolate missed bboxes of objects in track. For example, if frames #1 and #4 have bboxes and #2 and #3 do not, the system will reconstruct the absent bboxes #2 and #3 based on the #1 and #4 data. Enabling this option allows you to increase the detection quality on account of performance. Default value: true.

track_miss_interval

The system closes a track if there has been no new object in the track within the specified time (seconds). Default value: 1.

track_max_duration_frames

The maximum approximate number of frames in a track after which the track is forcefully completed. Enable it to forcefully complete “eternal tracks,” for example, tracks with objects from advertisement media. The default value: 0 (option disabled).

track_send_history

Send track history. Default value: false.

post_best_track_frame

Send full frames of detected objects. Default value: true.

post_best_track_normalize

Send normalized images for detected objects. Default value: true.

post_first_track_frame

Post the first frame of a track. Default value: false.

post_last_track_frame

Post the last frame of a track. Default value: false.

  1. The findface-video-worker configuration file /etc/findface-video-worker-cpu.ini or /etc/findface-video-worker-gpu.ini, subject to the acceleration type in use.

    When configuring findface-video-worker (on CPU/GPU), refer to the following parameters:

    CPU

    GPU

    Description

    batch_size

    Post faces in batches of the given size.

    capacity

    Maximum number of video streams to be processed by findface-video-worker.

    N/a

    cpu

    If necessary, decode video on CPU.

    N/a

    device_number

    GPU device number to use.

    exit_on_first_finished

    (Only if input is specified) Exit on the first finished job.

    input

    Process streams from file, ignoring stream data from findface-video-manager.

    labels

    Labels used to allocate a video object detector instance to a certain group of cameras. See Allocate findface-video-worker to Camera Group.

    mgr-cmd

    (Optional, instead of the mgr-static parameter) A command to obtain the IP address of the findface-video-manager host.

    mgr-static

    IP address of the findface-video-manager host to provide findface-video-worker with settings and the list of to-be-processed streams.

    metrics_port

    HTTP server port to send metrics. If 0, the metrics are not sent.

    min_size

    Minimum object size to be detected.

    ntls-addr

    IP address and port of the findface-ntls host.

    resize_scale

    Rescale video frames with the given coefficient.

    resolutions

    Preinitialize findface-video-worker to work with specified resolutions. Example: “640x480;1920x1080”.

    save_dir

    (For debug) Save detected objects to the given directory.

    streamer -> port, url

    IP address and port to access the video wall.

    use_time_from_sei

    (For MPEG-2) Use SEI (supplemental enhancement information) timestamps.

    If necessary, you can also enable neural network models and normalizers to detect bodies, cars, and liveness. You can find the detailed step-by-step instructions in the following sections:

Jobs

The findface-video-manager service provides findface-video-worker with a so-called job, a video processing task that contains configuration settings and stream data.

You can find a job example here.

Each job has the following parameters:

  • id: job id.

  • enabled: active status.

  • stream_url: URL/address of video stream/file to process.

  • labels: tag(s) that will be used by the findface-facerouter component (findface-security in the standard FindFace Multi configuration) to find processing directives for faces detected in this stream.

  • single_pass: if true, disable restarting video processing upon error (by default, false).

  • router_url: IP address and port of the findface-facerouter component (findface-security in the standard FindFace Multi configuration) to receive detected faces from the findface-video-worker component for processing.

  • stream_settings: video stream settings that duplicate those in the /etc/findface-video-manager.conf configuration file (while having priority over them).

  • status: job status.

  • status_msg: additional job status info.

  • statistic: job progress statistics (progress duration, number of posted and not posted objects, processing fps, the number of processed and dropped frames, job start time, etc.).

  • worker_id: id of the findface-video-worker instance executing the job.