Direct API requests to extraction-api

You can use HTTP API to extract data directly from the extraction-api component.

Note

extraction-api is the component that sf-api relies on for object detection and feature vector and attribute extraction. It is stateless and operates purely on request-response basis.

Tip

Normalized images received from extraction-api are qualified for posting to sf-api and vice versa.

In this section:

Structure

API Requests structure

The extraction-api component accepts requests to http://<extraction-api_ip>:18666/.

There are an API v1 and v2 (multiobject).

Important

Differences:

• The API v1 is selected automatically, if not route /v2 in the request URL.

• The API v2 defines the object type and return response in appropriate fields. Also, it has new unified attribute names.

There are 2 ways to format the request body:

• application/json: the request body contains only JSON.

• multipart/form-data: the request body contains a JSON part with the request itself, other body parts are used for image transfer.

The JSON part of the request body contains a set of requests:

{
    "requests": [request1, request2, .., requestN]
    "include_timings": true|false // include face processing timing in response, false by default
    "response_format": "msgpack" // return response in msgpack format
}

API Response Structure

A typical response from the extraction-api component contains a set of responses to the requests wrapped into the main API request:

{
    "response": [response1, response2, .., responseN]
}

API V2

API Request Format

Each request in the set applies to a specific image or region in the image and accepts the following parameters:

• "image": an uploaded image (use multipart:part to refer to a relevant request body part), or a publicly accessible image URL (http:, https:).

• "roi": a region of interest in the image. If the region is not specified, the entire image is processed.

• "detector": an object detector to apply to the image (face, body, car and etc or prenormalized or original). The prenormalized mode accepts normalized object images and omits detecting objects.

• object_type: it’s a required parameter, if the value of the detector is original. For example, to extract face_liveness from the original image the value must be face, to extract frameattrs the value must be none.

• bbox: object bbox. It is used for extraction from original, extraction will be from the image with that value of bbox.

• "need_normalized": returns a normalized object image encoded in base64. The normalized image can then be posted again to the extraction-api component as prenormalized.

• "auto_rotate": if true, auto-rotates an original image to 4 different orientations and returns objects detected in each orientation.

• "quality_estimator": if false, detection_score returns from the detector without face_quality attribute extract.

• "attributes": array of strings in the format ["face_gender", "face_age", "face_emotions"], enables recognition of the objects features passed in the array. Attribute name contains object type as prefix (face_, body_ and etc.)

{
    "image": "http://static.findface.pro/sample.jpg",
    "roi": {"left": 0, "right": 1000, "top": 0, "bottom": 1000},
    "detector": "face",
    "need_normalized": true,
    "auto_rotate": true,
    "attributes": ["face_emben", "face_gender", "face_age", "face_emotions", "face_beard", "face_glasses3"]
}

API Response Format

Each response in the set contains the following JSON data:

• "objects": a structure with sets of detected objects in the provided image or region of interest.

• "error": an error occurred during processing (if any). The error body includes the error code which can be interpreted automatically ("code") and a human-readable description ("desc").

• "timings": processing timings if "include_timings": true.

{
    "objects": {
        "face": [...] // detected face objects
        "car": [...], // detected car objects
        "head": [...] // detected head objects
        "body": [...] // detected body objects
    },
    "orientation": 1, // input image orientation
    "detector": "" // detector name
    "timings": ... // timings if requested
}

Each object in the set is provided with the following data:

• "group_id": detection group identifier. All bboxes from one detects will have same value. For example, uses for N-in-1 detectors for group head, body, face from 3-in-1 detector.

• "bbox": coordinates of a bounding box with the object.

• "detection_score": either the object detection accuracy, or the object quality score. Upright objects in frontal position are considered the best quality. They result in values around 0, mostly negative (such as -0.00067401276, for example). Inverted objects and large object angles are estimated with negative values some -5 and less.

• "rotation_angle": angle of input image, which them object was detected.

• "attributes": object with results of attributes extraction. As keys uses full attribute name (face_age, face_gender), as value objects with the next data:

  • "extractor": extractor name.

  • "model_name": name of the extractor model.

  • "value": extraction result, may be different types for different extractors.

• "normalized": a normalized face image encoded in base64, if requested.

• "timings": face processing timings, if requested.

{
    "group_id": "28c97d15",
    "bbox": { "left": 1, "right": 2, "top": 3, "bottom": 4},
    "detection_score": 0.99,
    "normalized": "...",
    "attributes": {
      "face_age": {
          "extractor": "face_age",
          "model": "age.v2",
          "result": 25
      },
      "face_beard": {
          "extractor": "face_beard",
          "model": "beard.v0",
          "result": [
              { "confidence": 0.015328666, "name": "beard" }
          ]
      },
      "face_emotions": {
          "extractor": "face_emotions",
          "model": "emotions.v1",
          "result": [
              { "confidence": 0.99959123, "name": "neutral" },
              { "confidence": 0.00039093022, "name": "sad" },
              { "confidence": 8.647058e-06, "name": "happy" },
              { "confidence": 7.994732e-06, "name": "surprise" },
              { "confidence": 6.495376e-07, "name": "disgust" },
              { "confidence": 6.063106e-07, "name": "angry" },
              { "confidence": 7.077886e-10, "name": "fear" }
          ]
      },
      ...
    }
    "timings": ...
}

Examples

Request #1

curl -X POST -F sample=@sample.jpg -F 'request={"requests":[{"image":"multipart:sample", "detector":"face", "attributes": ["face_age", "face_gender", "face_emben"]}]}' http://127.0.0.1:18666/v2 | jq .

Response

{
  "responses": [
    {
      "faces": null,
      "objects": {
        "face": [
          {
            "group_id": "b781670d",
            "bbox": {
              "left": 168,
              "top": 338,
              "right": 812,
              "bottom": 1234
            },
            "detection_score": 0.7689582,
            "rotation_angle": 0,
            "attributes": {
              "face_age": {
                "extractor": "face_age",
                "model": "age.v2",
                "result": 47
              },
              "face_emben": {
                "extractor": "face_emben",
                "model": "kiwi_320",
                "result": "..."
              },
              "face_gender": {
                "extractor": "face_gender",
                "model": "gender.v2",
                "result": [
                  {
                    "confidence": 1,
                    "name": "male"
                  },
                  {
                    "confidence": 5.503795e-08,
                    "name": "female"
                  }
                ]
              }
            }
          }
        ]
      },
      "orientation": 1,
      "detector": "face_jasmine"
    }
  ]
}

Important

If the requested attribute is not founded in the configuration file or isn’t loaded, or object attribute does not match detect object type, this attribute will be ignored in the returned response.

Important

If need_normalized: true is specified in the request, normalization from "objects:object:base_normalization" config field will be used. If there is no base normalization in config, the default normalization will be used.

Request #2 A simple request with a 3-in-1 headbodyface detector

curl -s -X POST -F sample=@sample_3in1.jpg -F 'request={"requests":[{"image":"multipart:sample", "detector":"headbodyface", "attributes": ["face_emben", "body_emben", "head_motohelmet"]}]}' http://127.0.0.1:18666/v2 | jq

Response

{
  "responses": [
    {
      "faces": null,
      "objects": {
        "face": [
          {
            "group_id": "43c199aa",
            "bbox": {
              "left": 616,
              "top": 232,
              "right": 645,
              "bottom": 266
            },
            "detection_score": 0.67829776,
            "rotation_angle": 0,
            "attributes": {
              "face_emben": {
                "extractor": "face_emben",
                "model": "kiwi_320",
                "result": "..."
              }
            }
          }
        ],
        "head": [
          {
            "group_id": "43c199aa",
            "bbox": {
              "left": 615,
              "top": 225,
              "right": 652,
              "bottom": 270
            },
            "detection_score": 0.94091797,
            "rotation_angle": 0,
            "attributes": {
              "head_motohelmet": {
                "extractor": "head_motohelmet",
                "model": "headattr.motohelmet.v1",
                "result": 0.109558105
              }
            }
          }
        ],
        "body": [
          {
            "group_id": "43c199aa",
            "bbox": {
              "left": 544,
              "top": 220,
              "right": 691,
              "bottom": 468
            },
            "detection_score": 0.7998271,
            "rotation_angle": 0,
            "attributes": {
              "body_emben": {
                "extractor": "body_emben",
                "model": "andariel",
                "result": "..."
              }
            }
          }
        ]
      },
      "orientation": 1,
      "detector": "headbodyface"
    }
  ]
}

Warning

headbodyface detector must be enabled in the extraction-api configuration file.

detectors:
    max_batch_size: 1
    instances: 1
    models:
    headbodyface:
      aliases:
      - headbodyface
      model: detector/headbodyface.gpu.fnk
      options:
        min_object_size: 32
        resolutions: [2048x2048]

Request #3 Request with "frameattr" extraction

curl -s -X POST -F sample=@/home/crowd.jpg -F 'request={"requests":[{"image":"multipart:sample", "detector":"original", "attributes":["crowd_count"], "object_type": "none"}]}' http://127.0.0.1:18666/v2/ | jq

Response

{
  "responses": [
    {
      "faces": null,
      "objects": {
        "none": [
          {
            "group_id": "",
            "bbox": {
              "left": 0,
              "top": 0,
              "right": 1276,
              "bottom": 608
            },
            "detection_score": 1,
            "rotation_angle": 0,
            "attributes": {
              "crowd_count": {
                "extractor": "crowd_count",
                "model": "frameattr.crowdcount.v0",
                "result": {
                  "count": 690.92346,
                  "heatmap_height": 76,
                  "heatmap_image": "...",
                  "heatmap_image_multiplier": 0.9641899,
                  "heatmap_width": 159
                }
              }
            }
          }
        ]
      },
      "orientation": 1,
      "detector": "original"
    }
  ]
}

Request #4 Request with liveness extraction from original

curl -s -X POST -F sample=@/home/sample2.jpg -F 'request={"requests":[{"image":"multipart:sample", "detector":"original", "attributes":["face_liveness", "face_emben"], "bbox": {"left": 10, "top": 10, "right": 444, "bottom": 444}, "object_type": "face"}]}' http://127.0.0.1:18666/v2/ | jq

Response

{
  "responses": [
    {
      "faces": null,
      "objects": {
        "face": [
          {
            "group_id": "",
            "bbox": {
              "left": 10,
              "top": 10,
              "right": 445,
              "bottom": 445
            },
            "detection_score": 1,
            "rotation_angle": 0,
            "attributes": {
              "face_emben": {
                "extractor": "face_emben",
                "model": "kiwi_160",
                "result": "..."
              },
              "face_liveness": {
                "extractor": "face_liveness",
                "model": "liveness.pvn.v0",
                "result": 0.77615935
              }
            }
          }
        ]
      },
      "orientation": 1,
      "detector": "original"
    }
  ]
}

Method GET /v2/models-info

This method returns the information about enabled detectors, normalizers, extractors and objects.

Request

curl -s  http://127.0.0.1:18666/v2/models-info | jq

Response

{
  "detectors": {
    "body": {
      "object_types": [
        "body"
      ]
    },
    "car": {
      "object_types": [
        "car"
      ]
    },
    "gustav_body": {
      "object_types": [
        "body"
      ]
    },
    "gustav_car": {
      "object_types": [
        "car"
      ]
    },
    "headbodyface": {
      "object_types": [
        "head",
        "body",
        "face"
      ]
    },
    "license_plate": {
      "object_types": [
        "license_plate"
      ]
    },
    "license_plate_gustav_accurate": {
      "object_types": [
        "license_plate"
      ]
    },
    "shiloette": {
      "object_types": [
        "body"
      ]
    }
  },
  "normalizers": {
    "carlicplate": {
      "normalization_type": "carlicplate"
    },
    "cropbbox": {
      "normalization_type": "cropbbox"
    },
    "norm200": {
      "normalization_type": "norm200"
    }
  },
  "extractors": {
    "car_color": {
      "normalization": "crop1x",
      "model_name": "carattr_color.v0"
    },
    "car_quality": {
      "normalization": "cropbbox",
      "model_name": "carattr.quality.v0"
    },
    "face_emben": {
      "normalization": "norm200",
      "model_name": "kiwi_160"
    },
    "face_quality": {
      "normalization": "crop1x",
      "model_name": "quality_fast.v1"
    },
    "license_plate_quality": {
      "normalization": "cropbbox",
      "model_name": "carlicplateattr.quality.v0"
    }
  },
  "objects": {
    "car": {
      "quality_attribute": "car_quality",
      "base_normalizer": "cropbbox"
    },
    "face": {
      "quality_attribute": "face_quality",
      "base_normalizer": "crop2x"
    },
    "license_plate": {
      "quality_attribute": "license_plate_quality",
      "base_normalizer": "carlicplate"
    }
  }
}

API V1

API Request Format

Each request in the set applies to a specific image or region in the image and accepts the following parameters:

Important

To enable recognition of face features, you can use either the new (preferred) or old API parameters. The old API allows you to recognize gender, age, and emotions, while the new API provides recognition of gender, age, emotions, country, beard, and glasses. Each face feature (gender, age, emotions, country, beard, or glasses) must be mentioned only once in a request, either in the new or old API format.

• "image": an uploaded image (use multipart:part to refer to a relevant request body part), or a publicly accessible image URL (http:, https:).

• "roi": a region of interest in the image. If the region is not specified, the entire image is processed.

• "detector": a face detector to apply to the image (legacy, nnd or prenormalized). The prenormalized mode accepts normalized face images and omits detecting faces. Use nnd if you need to estimate the face quality ("quality_estimator": true).

• "need_facen": if true, the request returns a facen in the response.

• "need_gender": returns gender (old API).

• "need_emotions": returns emotions (old API).

• "need_age": returns age (old API).

• "need_normalized": returns a normalized face image encoded in base64. The normalized image can then be posted again to the extraction-api component as “prenormalized”.

• "auto_rotate": if true, auto-rotates an original image to 4 different orientations and returns faces detected in each orientation. Works only if "detector": "nnd" and "quality_estimator": true.

• "attributes": array of strings in the format ["gender", "age", "emotions", "countries47", "beard", "glasses3"], enables recognition of the face features passed in the array (new API).

{
    "image": "http://static.findface.pro/sample.jpg",
    "roi": {"left": 0, "right": 1000, "top": 0, "bottom": 1000},
    "detector": "nnd",
    "need_facen": true,
    "need_gender": true,
    "need_emotions": true,
    "need_age": true,
    "need_normalized": true,
    "auto_rotate": true
}

API Response Format

Each response in the set contains the following JSON data:

• "faces": a set of faces detected in the provided image or region of interest.

• "error": an error occurred during processing (if any). The error body includes the error code which can be interpreted automatically ("code") and a human-readable description ("desc").

• "facen_model": face extraction model if "need_facen": true.

• "timings": processing timings if "include_timings": true.

{
    "faces": [face1, face2, .., faceN],
    "error": {
        "code": "IMAGE_DECODING_FAILED",
        "desc": "Failed to decode: reason"
    }
    "facen_model": "elderberry_576",
    "timings": ...

}

Each face in the set is provided with the following data:

• "bbox": coordinates of a bounding box with the face.

• "detection_score": either the face detection accuracy, or the face quality score (depending on whether quality_estimator is false or true at extraction-api.yaml). Upright faces in frontal position are considered the best quality. They result in values around 0, mostly negative (such as -0.00067401276, for example). Inverted faces and large face angles are estimated with negative values some -5 and less.

• "facen": face feature vector.

• "gender": gender information (MALE or FEMALE) with recognition accuracy if requested (old API).

• "age": age estimate if requested (old API).

• "emotions": all available emotions in descending order of probability if requested (old API).

• "countries47": probable countries of origin with algorithm confidence in the result if requested (old API).

• "attributes": gender (male or female), age (number of years), emotions (predominant emotion), probable countries of origin, beard (beard or none), glasses (sun, eye, or none), along with algorithm confidence in the result if requested (new API).

• "normalized": a normalized face image encoded in base64, if requested.

• "timings": face processing timings, if requested.

{
 "bbox": { "left": 1, "right": 2, "top": 3, "bottom": 4},
 "detection_score": 0.99,
 "facen": "...",
 "gender": {
     "gender": "MALE",
     "score": "1.123"
 },
 "age": 23.59,
 "emotions": [
     { "emotion": "neutral", "score": 0.95 },
     { "emotion": "angry", "score": 0.55 },
     ...
 ],
 "normalized": "...",
 "attributes": {
   "age": {
       "attribute": "age",
       "model": "age.v1",
       "result": 25
   },
   "beard": {
       "attribute": "beard",
       "model": "beard.v0",
       "result": [
           { "confidence": 0.015328666, "name": "beard" }
       ]
   },
   "countries47": {
       "attribute": "countries47",
       "model": "countries47.v1",
       "result": [
           { "confidence": 0.90330666, "name": "UKR" },
           { "confidence": 0.013165677, "name": "RUS" },
           { "confidence": 0.009136979, "name": "POL" },
           ...
       ]
   },
   "emotions": {
       "attribute": "emotions",
       "model": "emotions.v1",
       "result": [
           { "confidence": 0.99959123, "name": "neutral" },
           { "confidence": 0.00039093022, "name": "sad" },
           { "confidence": 8.647058e-06, "name": "happy" },
           { "confidence": 7.994732e-06, "name": "surprise" },
           { "confidence": 6.495376e-07, "name": "disgust" },
           { "confidence": 6.063106e-07, "name": "angry" },
           { "confidence": 7.077886e-10, "name": "fear"               }
       ]
   },
   "gender": {
       "attribute": "gender",
       "model": "gender.v2",
       "result": [
           { "confidence": 0.999894, "name": "female" },
           { "confidence": 0.00010597264, "name": "male" }
       ]
   },
   "glasses3": {
       "attribute": "glasses3",
       "model": "glasses3.v0",
       "result": [
           { "confidence": 0.9995815, "name": "none" },
           { "confidence": 0.0003348241, "name": "eye" },
           { "confidence": 8.363914e-05, "name": "sun" }
       ]
   }
 }
 "timings": ...
}

Examples

Request #1

curl -X POST -F sample=@sample.jpg -F 'request={"requests":[{"image":"multipart:sample","detector":"nnd", "need_gender":true, "need_normalized": true, "need_facen": true}]}' http://127.0.0.1:18666/| jq

Response

{
  "responses": [
    {
      "faces": [
        {
          "bbox": {
            "left": 595,
            "top": 127,
            "right": 812,
            "bottom": 344
          },
          "detection_score": -0.0012599,
          "facen": "qErDPTE...vd4oMr0=",
          "gender": {
            "gender": "FEMALE",
            "score": -2.6415858
          },
          "normalized": "iVBORw0KGgoAAAANSUhE...79CIbv"
        }
      ]
    }
  ]
}

Request #2

curl -X POST  -F 'request={"requests": [{"need_age": true, "need_gender": true, "detector": "nnd", "roi": {"left": -2975, "top": -635, "right": 4060, "bottom": 1720}, "image": "https://static.findface.pro/sample.jpg", "need_emotions": true}]}' http://127.0.0.1:18666/ |jq

Response

{
  "responses": [
    {
      "faces": [
        {
          "bbox": {
            "left": 595,
            "top": 127,
            "right": 812,
            "bottom": 344
          },
          "detection_score": 0.9999999,
          "gender": {
            "gender": "FEMALE",
            "score": -2.6415858
          },
          "age": 26.048346,
          "emotions": [
            {
              "emotion": "neutral",
              "score": 0.90854686
            },
            {
              "emotion": "sad",
              "score": 0.051211596
            },
            {
              "emotion": "happy",
              "score": 0.045291856
            },
            {
              "emotion": "surprise",
              "score": -0.024765536
            },
            {
              "emotion": "fear",
              "score": -0.11788454
            },
            {
              "emotion": "angry",
              "score": -0.1723868
            },
            {
              "emotion": "disgust",
              "score": -0.35445923
            }
          ]
        }
      ]
    }
  ]
}

Request #3. Auto-rotation

curl -s -F 'sample=@/path/to/your/photo.png' -F 'request={"requests":[{"image":"multipart:sample","detector":"nnd", "auto_rotate": true, "need_normalized": true }]}' http://192.168.113.79:18666/

Response

{
 "responses": [
   {
     "faces": [
       {
         "bbox": {
           "left": 96,
           "top": 99,
           "right": 196,
           "bottom": 198
         },
         "detection_score": -0.00019264,
         "normalized": "iVBORw0KGgoAAAANSUhE....quWKAAC"
        },
       {
         "bbox": {
           "left": 205,
           "top": 91,
           "right": 336,
           "bottom": 223
         },
         "detection_score": -0.00041600747,
         "normalized": "iVBORw0KGgoAAAANSUhEUgAA....AByquWKAACAAElEQVR4nKy96XYbybIdnF"
       }
     ]
   }
 ]
}

Request #4. New API usage (attributes: “beard”, “emotions”, “age”, “gender”, “glasses3”, “face”)

curl -s -F photo=@sample.jpg -Frequest='{"requests": [{"image":"multipart:photo", "detector": "nnd", "attributes": ["beard", "emotions", "age", "gender", "glasses3", "face"]}]}' http://127.0.0.1:18666 | jq

Response

{
  "responses": [
    {
      "faces": [
        {
          "bbox": {
            "left": 595,
            "top": 127,
            "right": 812,
            "bottom": 344
          },
          "detection_score": -0.00067401276,
          "rotation_angle": 0,
          "attributes": {
            "age": {
              "attribute": "age",
              "model": "age.v1",
              "result": 25
            },
            "beard": {
              "attribute": "beard",
              "model": "beard.v0",
              "result": [
                {
                  "confidence": 0.015324414,
                  "name": "beard"
                }
              ]
            },
            "emotions": {
              "attribute": "emotions",
              "model": "emotions.v1",
              "result": [
                {
                  "confidence": 0.99958,
                  "name": "neutral"
                },
                {
                  "confidence": 0.0004020365,
                  "name": "sad"
                },
                {
                  "confidence": 8.603454e-06,
                  "name": "happy"
                },
                {
                  "confidence": 8.076766e-06,
                  "name": "surprise"
                },
                {
                  "confidence": 6.6535216e-07,
                  "name": "disgust"
                },
                {
                  "confidence": 6.1434775e-07,
                  "name": "angry"
                },
                {
                  "confidence": 7.3372125e-10,
                  "name": "fear"
                }
              ]
            },
            "face": {
              "attribute": "face",
              "model": "elderberry_576",
              "result": "KjiHu6cWh70ppqa9l"
            },
            "gender": {
              "attribute": "gender",
              "model": "gender.v2",
              "result": [
                {
                  "confidence": 0.9998938,
                  "name": "female"
                },
                {
                  "confidence": 0.000106243206,
                  "name": "male"
                }
              ]
            },
            "glasses3": {
              "attribute": "glasses3",
              "model": "glasses3.v0",
              "result": [
                {
                  "confidence": 0.99958307,
                  "name": "none"
                },
                {
                  "confidence": 0.00033243417,
                  "name": "eye"
                },
                {
                  "confidence": 8.4465064e-05,
                  "name": "sun"
                }
              ]
            }
          }
        }
      ],
      "orientation": 1
    }
  ]
}