Video face detection: findface-video-manager and findface-video-worker

Note

The findface-video-worker is delivered in a CPU-accelerated (findface-video-worker-cpu) and a GPU-accelerated (findface-video-worker-gpu) packages.

In this section:

Functions of findface-video-manager

The findface-video-manager service is the part of the video face detection module that is used for managing the video face detection functionality.

The findface-video-manager service interfaces with findface-video-worker as follows:

  • It supplies findface-video-worker with settings and the list of to-be-processed video streams. To do so, it issues a so-called job, a video processing task that contains configuration settings and stream data.
  • In a distributed system, it distributes video streams (jobs) across vacant findface-video-worker instances.

Note

Configuration settings passed via jobs have priority over the findface-video-manager configuration file.

The findface-video-manager service functioning requires ETCD, third-party software that implements a distributed key-value store for findface-video-manager. In the FindFace core, ETCD is used as a coordination service, providing the video face detector with fault tolerance.

Functionality:

  • allows for configuring video face detection parameters,
  • allows for managing the list of to-be-processed video streams,
  • implements video face detection management.

Functions of findface-video-worker

The findface-video-worker service (on CPU/GPU) is the part of the video face detection module, that recognizes faces in the video. It can work with both live streams and files, and supports most video formats and codecs that can be decoded by FFmpeg.

The findface-video-worker service interfaces with the findface-video-manager and findface-facerouter services as follows:

  • By request, findface-video-worker gets a job with settings and the list of to-be-processed video streams from findface-video-manager.
  • The findface-video-worker posts extracted normalized face images, along with the full frames and meta data (such as bbox, camera ID and detection time) to the findface-facerouter service for further processing.

Note

In FindFace Security, the findface-facerouter functions are performs by findface-security.

Functionality:

  • detects faces in video,
  • extracts normalized face images,
  • searches for the best face snapshot,
  • snapshot deduplication (only one snapshot per face detection event).

When processing video, findface-video-worker consequently uses the following algorithms:

  • Motion detection. Used to reduce resource consumption. Only when the motion detector recognizes the motion of certain intensity that the face tracker can be triggered.
  • Face tracking. The face tracker tracks, detects and captures faces in the video. It can simultaneously be working with several faces. It also searches for the best face snapshot, using an embedded neural network. After the best face snapshot is found, it is posted to findface-facerouter.

The best face snapshot can be found in one of the following modes:

  • Real-time
  • Offline

Real-Time Mode

In the real-time mode, findface-video-worker posts a face immediately after it appears in the camera field of view.

  • If rt-perm=True, the face tracker searches for the best face snapshot within each time period equal to rt-delay and posts it to findface-facerouter.
  • If rt-perm=False, the face tracker searches for the best face snapshot dynamically:
    1. First, the face tracker estimates whether the quality of a face snapshot exceeds a pre-defined threshold value. If so, the snapshot is posted to findface-facerouter.
    2. The threshold value increases after each post. Each time the face tracker gets a higher quality snapshot of the same face, it is posted.
    3. When the face disappears from the camera field of view, the threshold value resets to default.

By default, the real-time mode is disabled (realtime=false in the /etc/findface-video-manager.conf file).

Offline Mode

The offline mode is less storage intensive than the real-time one as in this mode findface-video-worker posts only one snapshot per track, but of the highest quality. In this mode, the face tracker buffers a video stream with a face in it until the face disappears from the camera field of view. Then the face tracker picks up the best face snapshot from the buffered video and posts it to findface-facerouter.

By default, the offline mode is enabled (overall=true in the /etc/findface-video-manager.conf file).

Configure Video Face Detection

The video face detector configuration is done through the following configuration files:

  1. The findface-video-manager configuration file /etc/findface-video-manager.conf:

    listen: 127.0.0.1:18810
    etcd:
      endpoints: 127.0.0.1:2379
      dial_timeout: 3s
    kafka:
      enabled: false
      endpoints: 127.0.0.1:9092
    master:
      lease_ttl: 10
      self_url: 127.0.0.1:18811
      self_url_http: 127.0.0.1:18811
    rpc:
      listen: 127.0.0.1:18811
      heart_beat_timeout: 4s
    router_url: http://127.0.0.1:18820/v0/frame
    exp_backoff:
      enabled: false
      min_delay: 1s
      max_delay: 1m0s
      factor: 2
      flush_interval: 2m0s
    ntls:
      enabled: false
      url: http://127.0.0.1:3185/
      update_interval: 1m0s
    prometheus:
      jobs_processed_duration_buckets:
      - 1
      - 30
      - 60
      - 500
      - 1800
      - 3600
      - 21600
      - .inf
    job_scheduler_script: ''
    stream_settings:
      ffmpeg_params: []
      md_threshold: 0.002
      md_scale: 0.3
      fd_frame_height: -1
      uc_max_time_diff: 30
      uc_max_dup: 3
      uc_max_avg_shift: 10
      det_period: 8
      realtime: false
      npersons: 4
      disable_drops: false
      tracker_threads: 4
      parse_sei: false
      image_arg: photo
      additional_headers: []
      additional_body: []
      api_timeout: 15000
      api_ssl_verify: true
      post_uniq: true
      min_score: -2
      min_d_score: -1000
      realtime_dly: 500
      realtime_post_perm: false
      rot: ''
      roi: ''
      draw_track: false
      send_track: 0
      min_face_size: 0
      max_face_size: 0
      overall: true
      only_norm: false
      max_candidates: 0
      jpeg_quality: 95
      ffmpeg_format: ''
    stream_settings_gpu:
      play_speed: -1
      filter_min_quality: -2
      filter_min_face_size: 1
      filter_max_face_size: 8192
      normalized_only: false
      jpeg_quality: 95
      overall_only: true
      use_stream_timestamp: false
      ffmpeg_params: []
      router_timeout_ms: 15000
      router_verify_ssl: true
      router_headers: []
      router_body: []
      start_stream_timestamp: 0
      imotion_threshold: 0
      rot: ''
      roi: ''
      realtime_post_interval: 1
      realtime_post_every_interval: false
      ffmpeg_format: ''
      disable_drops: false
      router_full_frame_png: false
      router_disable_normalized: false
    

When configuring findface-video-manager, refer to the following parameters:

Option Description
router_url IP address and port of the findface-facerouter host to receive detected faces from findface-video-worker. In FindFace Security, findface-facerouter functions are performed by findface-security. Default value: http://127.0.0.1:18820/v0/frame.
etcd -> endpoints IP address and port of the etcd service. Default value: 127.0.0.1:2379.
ntls -> enabled If true, findface-video-manager will send a job to findface-video-worker only if the total number of processed cameras does not exceed the allowed number of cameras from the license. Default value: false.
ntls -> url IP address and port of the findface-ntls host. Default value: http://127.0.0.1:3185/.

You can also configure the following parameters:

Note

In the stream_settings-gpu section of the file, you will find general settings for all video streams. The settings in this section work for both CPU and GPU configuration. Settings of a particular stream, passed in a job, have priority over those in the configuration file (see Jobs).

Note

The stream_settings section of the file is deprecated and necessary only for backward compatibility.

Option Description
play_speed If less than zero, the speed is not limited. In other cases, the stream is read with the given play_speed. Not applicable for live streams.
filter_min_quality Minimum threshold value for a face image quality. A face is posted if it has better quality. The threshold value is to be fitted empirically. Empirical units: negative rational numbers to zero. Milestones: 0 = high quality faces, -1 = good quality, -2 = satisfactory quality, -5 = face recognition maybe inefficient. Default value: -2.
filter_min_face_size Minimum size of a face in pixels. Undersized faces are not posted. Default value: 0 (filter disabled).
filter_max_face_size Maximum size of a face in pixels. Oversized faces are not posted. Default value: 0 (filter disabled).
normalized_only Enable posting only normalized face images without full frames. Default value: false.
jpeg_quality Quality of an original frame JPEG compression, in percents. Default value: 95%.
overall_only Enables the offline mode for the best face search. Default value: true.
use_stream_timestamp If true, retrieve and post timestamps from a video stream. If false, post the actual date and time.
ffmpeg_params List of a video stream ffmpeg options with their values as a key=value array: [“rtsp_transpotr=tcp”, .., “ss=00:20:00”]. Check out the FFmpeg web site for the full list of options. Default value: options not specified.
router_timeout_ms Timeout for a findface-facerouter (or findface-security in the standard FindFace Security configuration) response to a findface-video-worker API request, in milliseconds. If the timeout has expired, the system will log an error. Default value: 15000.
router_verify_ssl Enables a https certificate verification when findface-video-worker and findface-facerouter (or findface-security in the standard FindFace Security configuration) interact over https. Default value: true. If false, a self-signed certificate can be accepted.
router_headers Additional header fields in a request when posting a face: [“key = value”]. Default value: headers not specified.
router_body Additional body fields in a request body when posting a face: [“key = value”]. Default value: body fields not specified.
start_stream_timestamp Add the specified number of seconds to timestamps from a stream.
imotion_threshold Minimum motion intensity to be detected by the motion detector. The threshold value is to be fitted empirically. Empirical units: zero and positive rational numbers. Milestones: 0 = detector disabled, 0.002 = default value, 0.05 = minimum intensity is too high to detect motion.
rot Enables detecting and tracking faces only inside a clipping rectangle WxH+X+Y. You can use this option to reduce findface-video-worker load. Default value: rectangle not specified.
roi Enable posting faces detected only inside a region of interest WxH+X+Y. Default value: region not specified.
realtime_post_interval Only for the real-time mode. If realtime_post_perm=True, defines the time period in milliseconds within which the face tracker picks up the best snapshot and posts it to findface-facerouter. If realtime_post_perm=False, defines the minimum time period between 2 posts of the same face with increased quality. Default value: 500.
realtime_post_every_interval Only for the realtime mode. Post best snapshots obtained within each realtime_dly time period. If false, search for the best snapshot dynamically and send snapshots in order of increasing quality. Default value: false.
ffmpeg_format Pass FFMPEG format (mxg, flv, etc.) if it cannot be detected automatically.
disable_drops Enables posting all appropriate faces without drops. By default, if findface-video-worker does not have enough resources to process all frames with faces, it drops some of them. If this option is active, findface-video-worker puts odd frames on the waiting list to process them later. Default value: false.
router_full_frame_png Send full frames in PNG and not in JPEG as set by default. Do not enable this parameter without supervision from our team as it can affect the entire system functioning. Default value: false (send in JPEG).
router_disable_normalized Send only full frames without normalized images. Do not enable this parameter without supervision from our team as it can affect the entire system functioning. Default value: false (send full frames and normalized images).
  1. If you opt for the CPU-accelerated package findface-video-worker-cpu, use the /etc/findface-video-worker-cpu.ini configuration file:

    ## read streams from file, do not use VideoManager
    input =
    
    ## exit on first finished job, only when --input specified
    exit_on_first_finished = false
    
    ## batch size
    batch_size = 4
    
    ## http server port for metrics, 0=do not start server
    metrics_port = 0
    
    ## resize scale, 1=do not resize
    resize_scale = 1.000000
    
    ## maximum number of streams
    capacity = 10
    
    ## command to obtain videomanager's grpc ip:port
    mgr_cmd =
    
    ## videomanager grpc ip:port
    mgr_static = 127.0.0.1:18811
    
    ## ntls server ip:port
    ntls_addr = 127.0.0.1:3133
    
    ## debug: save faces to dir
    save_dir =
    
    ## minimum face size
    min_face_size = 60
    
    ## preinit detector for specified resolutions: "640x480;1920x1080"
    resolutions =
    
    ## worker labels: "k=v;group=enter"
    labels =
    
    ## use timestamps from SEI packet
    use_time_from_sei = false
    
    #------------------------------
    [streamer]
    #------------------------------
    ## streamer server port, 0=disabled
    port = 18999
    
    ## streamer url - how to access this worker on streamer_port
    url = ws://127.0.0.1:18999/stream/
    
    #------------------------------
    [liveness]
    #------------------------------
    ## path to liveness fnk
    fnk =
    
    ## liveness threshold
    threshold = 0.945000
    
    ## liveness internal algo param
    interval = 1.000000
    
    ## liveness internal algo param
    stdev_cnt = 1
    
    #------------------------------
    [send]
    #------------------------------
    ## posting faces threads
    threads = 8
    
    ## posting faces maximum queue size
    queue_limit = 256
    
    #------------------------------
    [tracker]
    #------------------------------
    ## max face miss duration, sec
    miss_interval = 1.000000
    
    ## overlap threshold
    overlap_threshold = 0.250000
    
    #------------------------------
    [models]
    #------------------------------
    ## path to detector fnk
    detector = /usr/share/findface-data/models/facedet/mtcnn.cpu.fnk
    
    ## path to quality fnk
    quality = /usr/share/findface-data/models/faceattr/quality.v0.cpu.fnk
    
    ## path to norm for quality fnk
    norm_quality = /usr/share/findface-data/models/facenorm/ant.v2.cpu.fnk
    
    ## path to norm200 fnk, for face send
    norm_200 = /usr/share/findface-data/models/facenorm/ant.v2.cpu.fnk
    
    ## path to norm_crop2x fnk, for face send
    norm_crop2x = /usr/share/findface-data/models/facenorm/crop2x.v2_maxsize400.cpu.fnk
    

    If you opt for the GPU-accelerated package findface-video-worker-gpu, use the /etc/findface-video-worker-gpu.ini configuration file.

    ## cuda device number
    device_number = 0
    
    ## old gpu detector models directory
    models_dir = /usr/share/findface-gpudetector/models
    
    ## read streams from file, do not use VideoManager
    input =
    
    ## exit on first finished job, only when --input specified
    exit_on_first_finished = false
    
    ## batch size
    batch_size = 8
    
    ## http server port for metrics, 0=do not start server
    metrics_port =
    
    ## resize scale, 1=do not resize
    resize_scale = 1.000000
    
    ## maximum number of streams
    capacity = 30
    
    ## command to obtain videomanager's grpc ip:port
    mgr_cmd =
    
    ## videomanager grpc ip:port
    mgr_static = 127.0.0.1:18811
    
    ## ntls server ip:port
    ntls_addr = 127.0.0.1:3133
    
    ## debug: save faces to dir
    save_dir =
    
    ## minimum face size
    min_face_size = 60
    
    ## preinit detector for specified resolutions: "640x480;1920x1080"
    resolutions =
    
    ## worker labels: "k=v;group=enter"
    labels =
    
    ## use timestamps from SEI packet
    use_time_from_sei = false
    
    #------------------------------
    [streamer]
    #------------------------------
    ## streamer server port, 0=disabled
    port = 18999
    
    ## streamer url - how to access this worker on streamer_port
    url = ws://172.17.46.17:18999/stream/
    
    #------------------------------
    [liveness]
    #------------------------------
    ## path to liveness fnk
    fnk =
    
    ## liveness threshold
    threshold = 0.945000
    
    ## liveness internal algo param
    interval = 1.000000
    
    ## liveness internal algo param
    stdev_cnt = 1
    
    #------------------------------
    [send]
    #------------------------------
    ## posting faces threads
    threads = 8
    
    ## posting faces maximum queue size
    queue_limit = 256
    
    #------------------------------
    [tracker]
    #------------------------------
    ## max face miss duration, sec
    miss_interval = 1.000000
    
    ## overlap threshold
    overlap_threshold = 0.250000
    
    #------------------------------
    [models]
    #------------------------------
    ## path to detector fnk
    detector =
    
    ## path to quality fnk
    quality =
    
    ## path to norm for quality fnk
    norm_quality =
    
    ## path to norm200 fnk, for face send
    norm_200 = /usr/share/findface-data/models/facenorm/ant.v2.gpu.fnk
    
    ## path to norm_crop2x fnk, for face send
    norm_crop2x = /usr/share/findface-data/models/facenorm/crop2x.v2_maxsize400.gpu.fnk
    
    ## path to cache directory
    cache_dir =
    
    #------------------------------
    [video_decoder]
    #------------------------------
    ## decode video on cpu
    cpu = false
    

    When configuring findface-video-worker (on CPU/GPU), refer to the following parameters:

    CPU GPU Description
    ntls-addr IP address and port of the findface-ntls host.
    mgr-static IP address of the findface-video-manager host to provide findface-video-worker with settings and the list of to-be-processed streams.
    capacity Maximum number of video streams to be processed by findface-video-worker.
    mgr-exec (Optional, instead of the mgr-static parameter) A script to describe dynamic IP address of the findface-video-manager host.
    labels Labels used to allocate a video face detector instance to a certain group of cameras. See Allocate findface-video-worker to Camera Group.
    N/a fnk Path to the face liveness detector.
    input Process streams from file, ignoring stream data from findface-video-manager.
    exit_on_first_finished (Only if input is specified) Exit on the first finished job.
    resize_scale Rescale video frames with the given coefficient.
    save_dir (For debug) Save detected faces to the given directory.
    min_face_size Minimum face size to be detected.
    resolutions Preinitialize the findface-video-worker for specific resolutions to speed up its performance.
    N/a device_number GPU device number to use.
    N/a models_dir Old directory with GPU detector models. Otherwise, use the [models] section.
    N/a cpu If necessary, decode video on CPU.

Jobs

The findface-video-manager service provides findface-video-worker with a so-called job, a video processing task that contains configuration settings and stream data.

The content of a typical job is shown in the example below.

curl http://127.0.0.1:18810/job/1 | jq
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                              Dload  Upload   Total   Spent    Left  Speed
 100  1771  100  1771    0     0   447k      0 --:--:-- --:--:-- --:--:--  576k
{
  "id": "1",
  "enabled": true,
  "stream_url": "rtmp://restreamer.int.ntl/cams/openspace",
  "labels": {},
  "router_url": "http://172.17.46.13/video-detector/frame",
  "single_pass": false,
  "stream_settings": {
    "ffmpeg_params": [],
    "md_threshold": 0.002,
    "md_scale": 0.3,
    "fd_frame_height": -1,
    "uc_max_time_diff": 30,
    "uc_max_dup": 3,
    "uc_max_avg_shift": 10,
    "det_period": 8,
    "realtime": false,
    "npersons": 4,
    "disable_drops": false,
    "tracker_threads": 4,
    "parse_sei": false,
    "image_arg": "photo",
    "additional_headers": [
      "Authorization=Token b612396adc3a6dd71b82b5fe333a0a30"
    ],
    "additional_body": [],
    "api_timeout": 15000,
    "api_ssl_verify": true,
    "post_uniq": true,
    "min_score": -2,
    "min_d_score": -1000,
    "realtime_dly": 500,
    "realtime_post_perm": false,
    "rot": "",
    "roi": "",
    "draw_track": false,
    "send_track": 0,
    "min_face_size": 0,
    "max_face_size": 0,
    "overall": true,
    "only_norm": false,
    "max_candidates": 0,
    "jpeg_quality": 95,
    "ffmpeg_format": ""
  },
  "stream_settings_gpu": {
    "play_speed": -1,
    "filter_min_quality": -2,
    "filter_min_face_size": 1,
    "filter_max_face_size": 8192,
    "normalized_only": false,
    "jpeg_quality": 95,
    "overall_only": false,
    "use_stream_timestamp": false,
    "ffmpeg_params": [],
    "router_timeout_ms": 15000,
    "router_verify_ssl": true,
    "router_headers": [
      "Authorization=Token b612396adc3a6dd71b82b5fe333a0a30"
    ],
    "router_body": [],
    "start_stream_timestamp": 0,
    "imotion_threshold": 0,
    "rot": "",
    "roi": "",
    "realtime_post_interval": 1,
    "realtime_post_every_interval": false,
    "ffmpeg_format": "",
    "disable_drops": true
    "router_full_frame_png": false,
    "router_disable_normalized": false
  },
  "status": "INPROGRESS",
  "status_msg": "",
  "statistic": {
    "processed_duration": 14879,
    "faces_posted": 777,
    "faces_failed": 3,
    "faces_not_posted": 1206,
    "processing_fps": 18.816668,
    "frames_dropped": 0,
    "frames_processed": 0,
    "frames_imotion_skipped": 0,
    "decoding_soft_errors": 0,
    "job_starts": 56
  },
  "restream_url": "",
  "worker_id": "ffsec40_213ab8c0ed5d954e",
  "version": "bl068taaa7tcafrfsmq0"
}

Each job has the following parameters:

  • id: job id.
  • enabled: active status.
  • stream_url: URL/address of video stream/file to process.
  • labels: tag(s) that will be used by the findface-facerouter component (findface-security in the standard FindFace Security configuration) to find processing directives for faces detected in this stream.
  • single_pass: if true, disable restarting video processing upon error (by default, false).
  • router_url: IP address and port of the findface-facerouter component (findface-security in the standard FindFace Security configuration) to receive detected faces from the findface-video-worker component for processing.
  • stream_settings: used only for backward compatibility.
  • stream_settings_gpu: video stream settings that duplicate those in the findface-video-manager configuration file (while having priority over them).
  • status: job status.
  • status_msg: additional job status info.
  • statistic: job progress statistics (progress duration, number of posted and not posted faces, processing fps, the number of processed and dropped frames, job start time, etc.).
  • worker_id: id of the findface-video-worker instance executing the job.