.. _storage-api-utilities: Feature Vector Database Dump And Restore =========================================== Utilities for dump and restore tarantool databases are included to the ``sf-api`` component docker image. .. rubric:: In this section: .. contents:: :local: Database Dump ------------------------------ To get the information about command line parameters for ``storage-api-dump`` utility, run the following command: .. code:: bash docker run --rm -ti --entrypoint "/storage-api-dump" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 --help The most important command line flags are the following: .. list-table:: :widths: 14 8 45 :header-rows: 1 * - Command line flags - Type - Description * - ``-config`` - string - Path to config file. * - ``-config-template`` - – - Output config template and exit. * - ``-continue-on-errors`` - – - Continue on errors instead of exiting. * - ``-debug`` - – - Enable debug logging * - ``-gallery`` - string - Dump only galleries whose name match this regular expression (dump all if empty). * - ``-help`` - – - Print help information. * - ``-output-dir`` - string - Output directory (default ``"."``). * - ``-recent-id`` - uint - Last ID from previous run for incremental dump. To run the ``storage-api-dump`` utility, do the following: #. Create a default ``storage-api-dump`` configuration file. .. code:: bash docker run --rm -ti --entrypoint "/storage-api-dump" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \ --config-template > /opt/ffserver/configs/storage-api-dump.yaml * ``/opt/ffserver/configs/``: the directory on the host to store the configuration file. #. Modify section ``storage-api -> shards`` and setup actual values of shards: .. code:: bash ... storage-api: timeouts: connect: 5s response_header: 30s overall: 35s idle_connection: 10s max-idle-conns-per-host: 20 keepalive: 24h0m0s trace: false shards: - master: http://tnt-1-1:8001/v2/ slaves: [] - master: http://tnt-1-2:8002/v2/ slaves: [] ... #. Run ``storage-api-dump`` utility with the configuration file. .. code:: bash docker run --rm -ti --network server --entrypoint "/storage-api-dump" \ --volume /opt/ffserver/configs/storage-api-dump.yaml:/storage-api-dump.yaml \ --volume /opt/ffserver/dumps:/dumps \ docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \ --config /storage-api-dump.yaml -output-dir /dumps Console output: .. code:: bash 2024/05/28 15:23:02 6 Galleries in DB, going to dump 6: 2024/05/28 15:23:02 - default_0(default) (100 faces) 2024/05/28 15:23:02 - test_0(test) (100 faces) 2024/05/28 15:23:02 - default_1(default) (100 faces) 2024/05/28 15:23:02 - test_1(test) (100 faces) 2024/05/28 15:23:02 - default_2(default) (100 faces) 2024/05/28 15:23:02 - test_2(test) (100 faces) 2024/05/28 15:23:02 Going to dump all of these 2024/05/28 15:23:02 Dumping default_0(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_0(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_0(default) last ID 99 2024/05/28 15:23:02 Dumping default_1(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_1(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_1(default) last ID 99 2024/05/28 15:23:02 Dumping default_2(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_2(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_2(default) last ID 99 2024/05/28 15:23:02 Dumping test_0(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_0(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_0(test) last ID 99 2024/05/28 15:23:02 Dumping test_1(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_1(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_1(test) last ID 99 2024/05/28 15:23:02 Dumping test_2(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_2(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_2(test) last ID 99 2024/05/28 15:23:02 Done Database Restore -------------------------------- To get the information about command line parameters for ``storage-api-restore`` utility, run the following command: .. code:: bash docker run --rm -ti --entrypoint "/storage-api-restore" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 --help The most important command line flags are the following: .. list-table:: :widths: 14 8 45 :header-rows: 1 * - Command line flags - Type - Description * - ``-config`` - string - Path to config file. * - ``-config-template`` - – - Output config template and exit. * - ``-debug`` - – - Enable debug logging. * - ``-dont-create-gallery`` - – - Don't create gallery, fail if doesn't exist. * - ``-help`` - – - Print help information * - ``-rename`` - string - Ignore dump header and use this string as gallery name. To run the ``storage-api-restore`` utility, do the following: #. Create a default ``storage-api-restore`` configuration file. .. code:: bash docker run --rm -ti --entrypoint "/storage-api-restore" docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \ --config-template > /opt/ffserver/configs/storage-api-restore.yaml * ``/opt/ffserver/configs/``: the directory on the host to store the configuration file. #. Modify section ``storage-api -> shards`` and setup actual values of shards: .. code:: bash ... storage-api: timeouts: connect: 5s response_header: 30s overall: 35s idle_connection: 10s max-idle-conns-per-host: 20 keepalive: 24h0m0s trace: false shards: - master: http://tnt-1-1:8001/v2/ slaves: [] - master: http://tnt-1-2:8002/v2/ slaves: [] ... #. Run ``storage-api-restore`` utility with the configuration file. .. code:: bash cat /opt/ffserver/dumps/test_0 | docker run --rm -i --network server --entrypoint "/storage-api-restore" \ --volume /opt/ffserver/configs/storage-api-restore.yaml:/storage-api-restore.yaml \ docker.int.ntl/ntech/universe/sf-api:ffserver-12.240830.2 \ --config /storage-api-restore.yaml Console output: .. code:: bash 2024/05/28 16:10:48 No positional arguments specified, restoring from stdin 2024/05/28 16:10:48 Dump contains faces from "test_0"("test") 2024/05/28 16:10:48 Restoring into "test_0"("test") 2024/05/28 16:10:48 Gallery created 2024/05/28 16:10:48 100 2024/05/28 16:10:48 Waiting for workers (25 < 100)