.. _storage-api-reshard:

Resharding
==============================================================

The ``storage-api-reshard`` utility explores moving data from N old shards to M new shards, where M, N is the full shards count on installation.
To get the information about command line parameters for ``storage-api-reshard`` utility, run the following command:

.. code:: bash

    docker run --rm -ti --entrypoint "/storage-api-reshard" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 --help

The most important command line flags are the following:

.. list-table::
     :widths: 14 8 45
     :header-rows: 1

     * - Command line flags
       - Type
       - Description
     * - ``-config``
       - string
       - Path to config file.
     * - ``-config-template``
       - –
       - Output config template and exit.
     * - ``-delete-old``
       - –
       - Delete migrated faces on old shard (default true).
     * - ``-dry-run``
       - –
       - Do not add/delete faces, perform only read requests (default true).
     * - ``-gallery``
       - string
       - Gallery to reshard (default "history").
     * - ``-help``
       - –
       - Print help information.
     * - ``-max-face-id``
       - uint
       - Max object id for execution (default 18446744073709551615).
     * - ``-min-face-id``
       - uint
       - Min object id for execution.
     * - ``-new-shards``
       - []string
       - List of new shards url.
     * - ``-old-shards``
       - []string
       - List of old shards url.
     * - ``-shard-idx``
       - int
       - Execute migration for old_shards[idx]. Required parameter. On one execution only 1 old shard can be migrated to new shards.


.. warning::
  It's recommended to create a :ref:`dump <storage-api-utilities>` before resharding.

To run the ``storage-api-reshard`` utility, do the following:

#. Create a default ``storage-api-reshard`` configuration file.

   .. code:: bash

      docker run --rm -ti --entrypoint "/storage-api-reshard" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \
          --config-template > /opt/ffserver/configs/storage-api-reshard.yaml

   * ``/opt/ffserver/configs/``: the directory on the host to store the configuration file.

#. Modify configuration file:

   .. code:: bash

      ...
      gallery: history
      min_face_id: 0
      max_face_id: 18446744073709551615
      shard_idx: 0
      dry_run: true
      delete_old: true
      old_shards:
      - http://tnt-1-1:8001/v2/
      - http://tnt-2-1:8001/v2/
      new_shards: []
      - http://tnt-1-1:8001/v2/
      - http://tnt-2-1:8001/v2/
      - http://tnt-3-1:8001/v2/
      ...

   .. important::

      ``storage-api-resharder`` makes decisions based solely on shard number, so the ``new_shards`` list should be either
      shortened version of ``old_shards`` when scaling down, or the same as ``old_shards`` plus a few new lines when scaling up.

#. Run ``storage-api-reshard`` utility with the configuration file in ``dry_run`` to check your configuration.

   .. code:: bash

        docker run --rm -ti --network server --entrypoint "/storage-api-reshard" \
            --volume /opt/ffserver/configs/storage-api-reshard.yaml:/storage-api-reshard.yaml \
            docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \
            --config /storage-api-reshard.yaml


   Console output:

   .. code:: bash

       Config:
       ---
       gallery: face:test
       min_face_id: 0
       max_face_id: 18446744073709551615
       shard_idx: 0
       dry_run: true
       delete_old: true
       old_shards:
       - http://tnt-1-1:8001/v2
       - http://tnt-2-1:8001/v2
       new_shards:
       - http://tnt-1-1:8001/v2
       - http://tnt-2-1:8001/v2
       - http://tnt-3-1:8001/v2
       ...
       2024/06/06 06:58:47 SAR-BpLnfgDs | got faces(4 ... 6), will migrate 3/3
       2024/06/06 06:58:47 SAR-BpLnfgDs | total:3 migrated:3 migrateRatio:1.00 duration:0.00s speed:2138.16/s

       2024/06/06 06:58:47 Done

#. Run the ``storage-api-reshard`` utility with the configuration file without ``dry_run`` mode for all shard idx for migration.