Added initial version of dump-restore script. #380

ivanmrsulja · 2023-03-17T15:59:25Z

What's new?

This functionality was only available through UI in admin panel, now, it is available through .sh and .bat script so that it can be setup as cron job with minimal configuration.

How should this be tested?

If you are willing to use the script the server has to be running. The CLI arguments are as follows:
bash dumpRestore.sh <action>

Action can be: dump or restore

For script to work, you have to configure parameters in setenv script. Because these parameters rarely change, I figured that someone can just edit few lines of the setenv script so that the command doesn't get cluttered with a lot of arguments:

models="CONTENT"                          # Models to be dumped/restored (CONTENT, CONFIGURATION etc...)
email="[email protected]"                   # Admin account email
password="password"                       # Admin account password

host="http://example:port"                # URL where the VIVO instance is hosted
purge="true"                              # Will the restoration process purge the models before restoring
restoration_files_path="."                # Directory containing the files used for restoration (restore files must be named as their corresponding models >> CONTENT.nq, CONFIGURATION.nq etc...)
dumped_files_path=$restoration_files_path # Directory containing the backed-up files
app_name="vivo"                           # app-name parameter

Interested parties

@chenejac @milospp

…ments.

litvinovg · 2023-03-20T17:02:47Z

I tried to dump data in linux environment, but it didn't work for me.
It prints that dump has been successfully created, but the file CONTENT.nq is empty
I suggest to add error checks and messages in case cookie wasn't returned and check return code of the dump request as well as the file size (not empty).
Also it might be useful to use date in output file name.

curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0

litvinovg · 2023-03-20T17:31:31Z

home/src/main/resources/scripts/dumpRestore.sh

+loginForm="Log in"
+
+# Log in and get session cookies
+curl --cookie-jar "$session/cookies.txt" -d "loginName=$loginName" -d "loginPassword=$loginPassword" -d "loginForm=$loginForm" $url


I think it should be --data-urlencode for login and password instead of -d as with -d "+" sign is interpreted as a space character

I have improved URL encoding as you advised, thank you for debugging the problem for me.

litvinovg · 2023-03-20T17:35:34Z

home/src/main/resources/scripts/dumpRestore.bat

+set loginForm=Log in
+
+:: Log in and get session cookies
+curl --cookie-jar "%session%\cookies.txt" --create-dirs -d "loginName=%loginName%" -d "loginPassword=%loginPassword%" -d "loginForm=%loginForm%" %url%


I think if you try to use login with + sign it wouldn't work here too. See comment below.

It is still -d here. I haven't tested it on Windows, but maybe we should also use --data-urlencode here?

You are right, I forgot to add it here for username and password. I will change it right now.

litvinovg · 2023-03-20T17:37:10Z

home/src/main/resources/scripts/dumpRestore.bat

+if "%action%" == "dump" (
+    echo Starting dump...
+
+    curl --cookie "%session%\cookies.txt" "%host%/%app_name%/dumpRestore/dump/%models%.nq?which=%models%" -o "%dumped_files_path%\%models%.nq"


Check return code (should be 200, not 302)
If return code is 302 then most likely authorization failed.
I would also check size of returned file. It shouldn't be empty.

I have added the necessary check, as you advised.

litvinovg · 2023-03-20T17:37:53Z

home/src/main/resources/scripts/dumpRestore.sh

+
+if [[ "$action" == "dump" ]]; then
+    echo "Starting dump..."
+    curl --cookie "$session/cookies.txt" "$host/$app_name/dumpRestore/dump/$models.nq?which=$models" -o "$dumped_files_path/$models.nq"


Check return code (should be 200, not 302)
If return code is 302 then most likely authorization failed.
I would also check size of returned file. It shouldn't be empty.

I have added the necessary check, as you advised.

litvinovg

I think login/password escaping should be fixed.
We should also check if dump failed and print some messages to inform user.

ivanmrsulja · 2023-03-21T09:21:48Z

I have improved the URL encoding and added necessary checks, as @litvinovg advised.

chenejac

@ivanmrsulja can we apply similar approach as Tomcat is using with setenv.sh (setenv.bat) file, meaning dumpRestore.bat and dumpRestore.sh might be renamed to vivo-admin.bat and vivo-admin.sh, and there might be additional files example.setenv.sh and example.setenv.bat? All scripts parameters might be specified in those setenv.sh/bat files except action. The files might stay at the script directory in the VIVO code base, but can be copied to VIVO_HOME/bin by using this file - https://github.com/vivo-project/Vitro/blob/main/home/src/main/assembly/home.xml

…figuring functionality on linux.

…tory.

chenejac

@ivanmrsulja it works for me (Windows 10). I have tried to create dump of content graph and to restore. Thanks for the contribution.

ivanmrsulja added 3 commits March 17, 2023 16:38

Added initial dump-restore script.

0f8df35

Changed email and password to get their values from command line argu…

d6c514d

…ments.

Added .bat script for dump-restore functionality.

a9ede4c

chenejac requested review from a user, litvinovg and brianjlowe March 20, 2023 12:32

ivanmrsulja added 2 commits March 20, 2023 15:30

Slight refactor of variable names and initialization.

399bd18

Removed testing value for host variable.

06578b4

litvinovg reviewed Mar 20, 2023

View reviewed changes

litvinovg requested changes Mar 20, 2023

View reviewed changes

Added necessary checks and fixed url encoding.

daee759

ivanmrsulja requested review from litvinovg and removed request for brianjlowe and a user March 21, 2023 09:21

chenejac requested review from a user and brianjlowe March 21, 2023 09:26

ivanmrsulja added 2 commits March 21, 2023 12:43

Added --data-urlencode option when logging in via .bat script.

9b498fb

Renamed variables to match snake-case convention.

6772b47

chenejac linked an issue Apr 6, 2023 that may be closed by this pull request

Making scripts for live backup of TDB vivo-project/VIVO#3819

Open

3 tasks

chenejac self-requested a review April 10, 2023 10:15

chenejac requested changes Apr 10, 2023

View reviewed changes

ivanmrsulja added 4 commits April 11, 2023 14:56

Renamed scripts to vivo-admin. Added example.setenv.sh script for con…

6748039

…figuring functionality on linux.

Added example.setenv.bat configuration script.

cfe9d48

Updated home.xml to copy scripts/ directory into vivo-home/bin/ direc…

5581fd3

…tory.

Fixed bug when comments are parsed as CLI arguments.

d3d8126

ivanmrsulja added 2 commits April 17, 2023 12:56

Fixed bug when performing restoration.

3fbcb57

Removed unneeded quotation marks.

f095b44

chenejac approved these changes Apr 17, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added initial version of dump-restore script. #380

Added initial version of dump-restore script. #380

ivanmrsulja commented Mar 17, 2023 •

edited

Loading

litvinovg commented Mar 20, 2023

litvinovg Mar 20, 2023 •

edited

Loading

ivanmrsulja Mar 21, 2023

litvinovg Mar 20, 2023 •

edited

Loading

litvinovg Mar 21, 2023

ivanmrsulja Mar 21, 2023

litvinovg Mar 20, 2023

ivanmrsulja Mar 21, 2023

litvinovg Mar 20, 2023

ivanmrsulja Mar 21, 2023

litvinovg left a comment

ivanmrsulja commented Mar 21, 2023

chenejac left a comment

chenejac left a comment

Added initial version of dump-restore script. #380

Are you sure you want to change the base?

Added initial version of dump-restore script. #380

Conversation

ivanmrsulja commented Mar 17, 2023 • edited Loading

What's new?

How should this be tested?

Interested parties

litvinovg commented Mar 20, 2023

litvinovg Mar 20, 2023 • edited Loading

Choose a reason for hiding this comment

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

litvinovg Mar 20, 2023 • edited Loading

Choose a reason for hiding this comment

litvinovg Mar 21, 2023

Choose a reason for hiding this comment

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

litvinovg Mar 20, 2023

Choose a reason for hiding this comment

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

litvinovg Mar 20, 2023

Choose a reason for hiding this comment

ivanmrsulja Mar 21, 2023

Choose a reason for hiding this comment

litvinovg left a comment

Choose a reason for hiding this comment

ivanmrsulja commented Mar 21, 2023

chenejac left a comment

Choose a reason for hiding this comment

chenejac left a comment

Choose a reason for hiding this comment

ivanmrsulja commented Mar 17, 2023 •

edited

Loading

litvinovg Mar 20, 2023 •

edited

Loading

litvinovg Mar 20, 2023 •

edited

Loading