.. _bulk-upload:

Console Bulk Photo Upload
======================================================

In addition to the :ref:`web interface upload <cards>`, you can bulk-upload photos to the card index via the :program:`findface-security-uploader` console utility. We recommend preferring this utility over the web interface if the number of uploaded photos is more than 10,000.

.. warning::
   In the current version, the :program:`findface-security-uploader` utility does not support cars and bodies, only faces.  

.. tip::
   To view the :program:`findface-security-uploader` help, execute:

   .. code::

      findface-security-uploader --help

      Usage: findface-security-uploader [OPTIONS] COMMAND [ARGS]...

      Options:
        --job PATH        Job file (default: enroll-job.db)
        --log-level TEXT  Log level
        --fsync BOOLEAN   Call fsync() to prevent data loss on power failure
        --help            Show this message and exit.

      Commands:
        add    Add items from CSV or TSV file to job
        print  Print contents of job file as JSON
        run    Run upload job

   .. code::

      findface-security-uploader add --help

      Usage: findface-security-uploader add [OPTIONS] FILES...

      Options:
        --format [csv|tsv]  Input file format - CSV or TSV
        --delimiter TEXT    Field delimiter - by default it's "\t" for TSV and ","
                            for CSV
        --help              Show this message and exit.

   .. code::

      findface-security-uploader print --help

      Usage: findface-security-uploader print [OPTIONS]

      Print contents of job file as JSON

      Options:
       --failed  Show only failed images
       --noface  Show only images without detection
       --help    Show this message and exit.

   .. code::

      findface-security-uploader run --help

      Usage: uploader.py run [OPTIONS]

      Run upload job

      Options:
        --parallel INTEGER       Number of enroll threads (default: 10)
        --api TEXT               API url (default: http://127.0.0.1:80/)  [required]
        --user TEXT              API username  [required]
        --password TEXT          API password  [required]
        --watch-lists TEXT       Comma-separated list of card list ids  [required]
        --inactive               Mark new cards as inactive
        --failed                 Include failed images
        --noface                 Include images without detection
        --all-faces              Enroll all found faces on each image
        --logging-delta INTEGER  Logging period delta
        --help                   Show this message and exit.

Do the following:

#. Write the list of photos and metastrings to a CSV or TSV file. 

   .. important::
      The file used as a metadata source must have the following format: ``path to photo | metastring``. 

   To prepare a TSV file, use either a :download:`script <_scripts/tsv_builder.py>` or the ``find`` command.

   .. note::
      Both the script and the command in the examples below create the ``images.tsv`` file. Each image in the list will be associated with a metastring coinciding with the image file name in the format ``path to photo | metastring``.

   To build a TSV file listing photos from a specified directory (``/home/user/25_celeb/`` in the example below), run the following command:  

   .. code::

      python3 tsv_builder.py /home/user/25_celeb/

   The ``find`` usage example:

   .. code::

      find photos/ -type f -iname '*g' | while read x; do y="${x%.*}"; printf "%s\t%s\n" "$x" "${y##*/}"; done

#. Create a job file out of a CSV or TSV file by using ``add``. As a result, a file ``enroll-job.db`` will be created and saved in a current directory. 

   .. code::

      findface-security-uploader add images.tsv

   The ``add`` options:

   * ``--format``: input file format, ``tsv`` by default,
   * ``--delimiter``: field delimiter, by default ``"\t"`` for TSV, and ``","`` for CSV.

   .. note::
      A job file represents a sqlite database which can be opened on the :program:`sqlite3` console.
 
#. Process the job file by using ``run``.

   .. code::

      findface-security-uploader run --watch-lists 2 --api http://127.0.0.1:80 --user admin --password password

   The important ``run`` options:

   * ``--parallel``: the number of photo upload threads, 10 by default. The more threads you use, the faster the bulk upload is completed, however it requires more resources too.
   * ``--all-faces``: upload all faces from a photo if it features several faces.
   * ``--api``: ``findface-security`` API URL, ``http://127.0.0.1:80/`` by default. Mandatory option.
   * ``--user``: login. Mandatory option.
   * ``--password``: password. Mandatory option.
   * ``--watch-lists``: comma-separated list of the watch lists id’s. Mandatory option.
   * ``--failed``: should an error occur during the job file processing, correct the mistake and try again with this option.
   * ``--inactive``: mark new cards as inactive.
   * ``--noface``: by default, images classified as having no faces will be assigned the ``NOFACE`` status and automatically excluded from the upload. To attempt re-detecting faces in such images, re-run the job file with this option. If the re-detection gives a negative result again, an image will be skipped and a relevant record will appear in the upload log.

#. (Optional) Print the job processing results as JSON. If necessary, you can print only failed images/ images without detected faces.  
 
   .. code::

      findface-security-uploader print --failed
      findface-security-uploader print --noface