Skip to content

1. OptiNiSt Environment Setup Procedure

Jump to bottom
milesAraya edited this page Mar 26, 2025 · 1 revision

OptiNiSt-for-Server Environment Setup Procedure (Docker)

Update History

  • v1.0 (2024-03) … New Creation

Prerequisites

  • Target Environment Information
  • OS … Docker on (Ubuntu 22.x / Mac / Windows 11, WSL2)

OptiNiSt_for_Server Setup

Perform the following steps to set up the environment.

Preparation

Get latest version of optinist-for-server.

git clone https://github.com/arayabrain/optinist-for-server.git -b release/v2.0.0

Rename .env file name.

cd {OPTINIST_SETUP_DIR}
mv studio/config/.env.example stdio/config/.env
mv frontend/.env.example frontend/.env

Edit stdio/config/.env contents.

EXPDB_DIR="/app/experiments_datasets"
PUBLIC_EXPDB_DIR="/app/experiments_public"

1. Docker for Developer

The most up-to-date multi-user firebase setup is specified here. Please check this as well. However, the outline of the procedure is written here.

docker compose -f docker-compose.dev.yml up

Note: At this point, the multiuser mode setup is still in progress, so when you access http://localhost:3000, you will see a blank page. (You will be able to access it after the setup is complete.)

If the host is Linux, perform the following additional tasks:

  1. Set the permissions of allowing access from within the container (non-root user)
  2. Set optinist(uid:500):www-data(gid:33)

These can be performed using this single command:

cd {OPTINIST_SETUP_DIR}
sudo chown 500:53 ../optinist-docker-volumes/* ../optinist-docker-volumes/.snakemake/

2. Host OptiNiSt yourself for multiuser users

The most up-to-date multi-user firebase setup is specified here. Please check this as well. However, the outline of the procedure is written here.

2.1 Firebase Setup

The following procedure can be carried out by individual developers. However, it is also possible to use the shared development firebase configs which can be found here. In that case, copy firebase_config.json and firebase_private.json to studio/config/auth.

  • If you want to use the shared development firebase configs, ask the administrator of the shared development Firebase Project to create your worker account.

Otherwise, follow these steps below to make your own firebase config credentials.

Create Firebase Project
  1. Go to https://console.firebase.google.com/
  2. Click "Add project"
  3. Enter your project name and click "Continue"
  4. Choose whether to enable Google Analytics (optional)
  5. Click "Continue" when project is ready
Setup Firebase Authentication
  1. Select "Build > Authentication" from the left menu
  2. Click "Get started"
  3. Select "Sign-in method" tab
  4. Click "Add new provider" in "Sign-in providers" section
  5. Enable "Email/Password"
  6. Click "Save"
Create Admin User

  1. Select "Authentication" from the left menu

  2. Select "Users" tab

  3. Click "Add user"

  4. Fill in the form:

    • Email address: your email address
    • Password: your password

    Important: Save the "User UID" - you'll need this later for database setup

Get Firebase Configuration Files
  1. Click settings icon (next to Project Overview)
  2. Select "Project settings"
  3. Select "General" tab
  4. Select "web app" in "Your apps" section
  5. Enter your app name and click "Register app"

    "Firebase Hosting" setup not required

  6. Copy the following contents to studio/config/auth/firebase_config.json
// For Firebase JS SDK v7.20.0 and later, measurementId is optional
const firebaseConfig = {
  apiKey: "xxxxxxxxxxx",
  authDomain: "xxxxxxxxxx.firebaseapp.com",
  projectId: "xxxxxxxxxxxx",
  storageBucket: "xxxxxxxxxxx.firebasestorage.app",
  messagingSenderId: "xxxxxxxxxxxxxxxx",
  appId: "1:xxxxxxxxxx:web:xxxxxxxxxxxxxxxxxxx",
  measurementId: "G-xxxxxxxxxx"
};
  1. Select "Service accounts" tab
  2. Click "Generate new private key"
  3. Save the downloaded private key file as studio/config/auth/firebase_private.json

2.2 Save initial data and set permissions

Obtain the test data from the following: https://drive.google.com/drive/folders/1-oY2yYl2QuteA13o2SvAAxKljhwKsVJY

Check the Readme file for information on the data organisation. Save in a directory on your computer within the same parent directory as your optimist repo e.g. optinist-for-server. Unzip the data. Then set permissions:

The file structure should be:

/app/experiments_datasets/
└── M000024/
    └── M000024_ori017/
        ├── M000024_ori017.nd2
        ├── M000024_ori017_metadata.json
        └── M000024_ori017_trialstructure.mat 

cd optinist-docker-volumes ||
sudo chmod 777 ./ logs/ .snakemake/ ../experiments_datasets/ ../experiments_public/

Maybe try:

sudo chmod -R a+rw ./experiments_datasets

If the host is Linux, perform the following additional tasks:

  1. Set the permissions of allowing access from within the container (non-root user)
  2. Set optinist(uid:500):www-data(gid:33)

These can be performed using this single command:

cd {OPTINIST_SETUP_DIR} # most probably optinist-for-server
sudo chown -R 500:53 ../experiments_datasets/ ../experiments_public/

2.3 Set all of the config files

2.3.1 Backend Setup

studio/config.env should have these variables. Set the bold variables with your own details. Examples are given for docker implementation studio/config.env.docker and local implementation studio/config.env.local

SECRET_KEY='XXXXXXX'
USE_FIREBASE_TOKEN=True # for log in with multiple users
REFRESH_TOKEN_EXPIRE_MINUTES=1440 # 24 hours

ECHO_SQL=True
MYSQL_SERVER=db  # for docker
MYSQL_ROOT_PASSWORD=MYSQL_ROOT_PASSWORD
MYSQL_DATABASE=OPTINIST_FOR_SERVER
MYSQL_USER=MY_NAME
MYSQL_PASSWORD=MYSQL_PASSWORD

PMA_ARBITRARY=1
PMA_HOST=db  # for docker
PMA_USER=root
PMA_PASSWORD=docker

# EXPDB_DIR is a directory path of the data unpacked.
# PUBLIC_EXPDB_DIR = Optional directory path (path to generate external publication data)
# Paths to these variables are different depending on if you are running locally or via docker.
EXPDB_DIR="/app/experiments_datasets" # for when docker
PUBLIC_EXPDB_DIR="/app/experiments_public" # for when docker
EXPDB_DIR="/MY_DATA_PATH/experiments_datasets" # for local development
PUBLIC_EXPDB_DIR="MY_DATA_PATH/experiments_public" # for local development

GRAPH_HOST="http://localhost:8000/datasets"  # URL for publishing data externally
SELFHOST_GRAPH=True
2.3.2 Frontend Setup

To setup frontend/.env.development, make a copy of frontend/.env.example frontend/.env.development should have these variables:

# (default: location.hostname)
REACT_APP_SERVER_HOST=localhost
# (default: location.port)
REACT_APP_SERVER_PORT=8000
# (default: location.protocol)
REACT_APP_SERVER_PROTO=http
# configs
REACT_APP_EXPDB_METADATA_EDITABLE=true

2.4 MySQL Setup (or mariadb)

  • You can set up your own mysql server, or you can use the DB container defined in docker-compose(.dev).yml. To set up your own mysql server follow these instructions:
3.1 Start Database (Development)
# Start the database container
docker compose -f docker-compose.dev.yml up -d

Wait a few seconds for the database to initialize. Verify database is running:

docker ps # Should see the database container running, You'll use the ID in the next step.

# Verify database connection
docker exec -it [DB_CONTAINER_NAME] mysql -u studio_db_user -p studio
docker exec -it optinist-for-server-db-1 mysql -u docker -pdocker docker
# Find DB_CONTAINER_NAME using 'docker ps'
3.1.2 Create Initial User
INSERT INTO organization (name) VALUES ('Your Organization');
INSERT INTO roles (id, role) VALUES (1, 'admin'), (20, 'operator');
INSERT INTO roles (id, role) VALUES  (10, 'data manager'),  (30, 'guest operator'); -- Additions in optinist-for-server
INSERT INTO users (uid, organization_id, name, email, active) VALUES ('YOUR_FIREBASE_UID', 1, 'YOUR_NAME', '[email protected]', true);
INSERT INTO user_roles (user_id, role_id) VALUES (1, 1);

2.5 Confirm Setup

After performing the above setup procedures, perform the following operation checks:

docker compose -f docker-compose.dev.yml up
# or
# docker compose -f docker-compose.yml up

Access http://localhost:3000 and confirm that the TOP page is displayed. Confirm that you can log in with the registered Firebase account from SignIn. At this point, the number of data items on the TOP page is 0 because no Database records have been created.

3. Execute the analysis batch process

  • First, place the command file that specifies the data to be processed by the batch analysis.
  • Create a file which ends in the extension .proc
    1. Example: /{YOUR_DATA_DEPLOYMENT_PATH}/M000024/M000024_ori001.proc
  • Add the text to the file: command: regist
  • Be careful to include the space between command: and regist

Then run the docker image. The -d runs the image in the background and allows you to use commands from the same window:

docker compose -f docker-compose.dev.yml up

In another terminal window login to the docker container:

docker exec -it optinist-for-server-studio-dev-be-1 bash # Enter container
conda activate expdb_batch # Activate conda environment
alembic upgrade head # Run migrations
echo "command: regist" > /app/experiments_datasets/M000024/M000024_ori017.proc  # reset .proc file
python run_expdb_batch.py -o 1 # Run the batch process

Note: On docker on mac, there are cases where "generate_plots()" stops the process. This will be resolved in the following PR (at 12/2024): https://github.com/arayabrain/barebone-studio/pull/531

4. Check that the data specified in 1. has been processed

  • If the processing is successful, the data to be processed will be listed on the Database screen of the management console.

Reference information

Clone this wiki locally