Feature Vector Database Dump And Restore
Utilities for dump and restore tarantool databases are included to the sf-api component docker image.
In this section:
Database Dump
To get the information about command line parameters for storage-api-dump utility, run the following command:
docker run --rm -ti --entrypoint "/storage-api-dump" docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 --help
The most important command line flags are the following:
Command line flags |
Type |
Description |
|---|---|---|
|
string |
Path to config file. |
|
– |
Output config template and exit. |
|
– |
Continue on errors instead of exiting. |
|
– |
Enable debug logging |
|
string |
Dump only galleries whose name match this regular expression (dump all if empty). |
|
– |
Print help information. |
|
string |
Output directory (default |
|
uint |
Last ID from previous run for incremental dump. |
To run the storage-api-dump utility, do the following:
Create a default
storage-api-dumpconfiguration file.docker run --rm -ti --entrypoint "/storage-api-dump" docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 \ --config-template > /opt/ffserver/configs/storage-api-dump.yaml
/opt/ffserver/configs/: the directory on the host to store the configuration file.
Modify section
storage-api -> shardsand setup actual values of shards:... storage-api: timeouts: connect: 5s response_header: 30s overall: 35s idle_connection: 10s max-idle-conns-per-host: 20 keepalive: 24h0m0s trace: false shards: - master: http://tnt-1-1:8001/v2/ slaves: [] - master: http://tnt-1-2:8002/v2/ slaves: [] ...
Run
storage-api-dumputility with the configuration file.docker run --rm -ti --network server --entrypoint "/storage-api-dump" \ --volume /opt/ffserver/configs/storage-api-dump.yaml:/storage-api-dump.yaml \ --volume /opt/ffserver/dumps:/dumps \ docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 \ --config /storage-api-dump.yaml -output-dir /dumps
Console output:
2024/05/28 15:23:02 6 Galleries in DB, going to dump 6: 2024/05/28 15:23:02 - default_0(default) (100 faces) 2024/05/28 15:23:02 - test_0(test) (100 faces) 2024/05/28 15:23:02 - default_1(default) (100 faces) 2024/05/28 15:23:02 - test_1(test) (100 faces) 2024/05/28 15:23:02 - default_2(default) (100 faces) 2024/05/28 15:23:02 - test_2(test) (100 faces) 2024/05/28 15:23:02 Going to dump all of these 2024/05/28 15:23:02 Dumping default_0(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_0(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_0(default) last ID 99 2024/05/28 15:23:02 Dumping default_1(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_1(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_1(default) last ID 99 2024/05/28 15:23:02 Dumping default_2(default) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping default_2(default) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery default_2(default) last ID 99 2024/05/28 15:23:02 Dumping test_0(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_0(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_0(test) last ID 99 2024/05/28 15:23:02 Dumping test_1(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_1(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_1(test) last ID 99 2024/05/28 15:23:02 Dumping test_2(test) from shard 0 with masterURL http://tnt-1-1:8001/v2/ 2024/05/28 15:23:02 52 2024/05/28 15:23:02 Dumping test_2(test) from shard 1 with masterURL http://tnt-1-2:8002/v2/ 2024/05/28 15:23:02 100 2024/05/28 15:23:02 Gallery test_2(test) last ID 99 2024/05/28 15:23:02 Done
Database Restore
To get the information about command line parameters for storage-api-restore utility, run the following command:
docker run --rm -ti --entrypoint "/storage-api-restore" docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 --help
The most important command line flags are the following:
Command line flags |
Type |
Description |
|---|---|---|
|
string |
Path to config file. |
|
– |
Output config template and exit. |
|
– |
Enable debug logging. |
|
– |
Don’t create gallery, fail if doesn’t exist. |
|
– |
Print help information |
|
string |
Ignore dump header and use this string as gallery name. |
To run the storage-api-restore utility, do the following:
Create a default
storage-api-restoreconfiguration file.docker run --rm -ti --entrypoint "/storage-api-restore" docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 \ --config-template > /opt/ffserver/configs/storage-api-restore.yaml
/opt/ffserver/configs/: the directory on the host to store the configuration file.
Modify section
storage-api -> shardsand setup actual values of shards:... storage-api: timeouts: connect: 5s response_header: 30s overall: 35s idle_connection: 10s max-idle-conns-per-host: 20 keepalive: 24h0m0s trace: false shards: - master: http://tnt-1-1:8001/v2/ slaves: [] - master: http://tnt-1-2:8002/v2/ slaves: [] ...
Run
storage-api-restoreutility with the configuration file.cat /opt/ffserver/dumps/test_0 | docker run --rm -i --network server --entrypoint "/storage-api-restore" \ --volume /opt/ffserver/configs/storage-api-restore.yaml:/storage-api-restore.yaml \ docker.int.ntl/ntech/universe/sf-api:ffserver-11.240325 \ --config /storage-api-restore.yaml
Console output:
2024/05/28 16:10:48 No positional arguments specified, restoring from stdin 2024/05/28 16:10:48 Dump contains faces from "test_0"("test") 2024/05/28 16:10:48 Restoring into "test_0"("test") 2024/05/28 16:10:48 Gallery created 2024/05/28 16:10:48 100 2024/05/28 16:10:48 Waiting for workers (25 < 100)