Multiple Management Server

[arpanbht]

I am trying to add two management server in my cloudstack setup but failing to add the 2nd management server. The 1st created management server is working fine with one compute host and one nfs server added.

These are the steps I am following to install the cloudstack-manangement

apt install chrony -y

sudo nano /etc/apt/sources.list.d/cloudstack.list
deb https://download.cloudstack.org/ubuntu jammy 4.22

wget -O - https://download.cloudstack.org/release.asc |sudo tee /etc/apt/trusted.gpg.d/cloudstack.asc

sudo apt update

sudo apt install cloudstack-management -y

sudo apt install openjdk-11-jdk -y

sudo update-alternatives --config java

sudo apt install mysql-server -y

sudo nano /etc/mysql/my.cnf

[mysqld]
server_id=source-01
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log_bin=mysql-bin
binlog_format=ROW

sudo systemctl restart mysql

sudo mysql

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<password>';
FLUSH PRIVILEGES;
EXIT;

sudo cloudstack-setup-databases cloud:<password>@localhost --deploy-as=root:<password> -e file -m <mgmt_key> -k <db_key> -i <mgmt_ip>

nano /etc/sudoers

Defaults:cloud !requiretty

cloudstack-setup-management

ufw allow mysql

ufw allow 8080/tcp
ufw allow 8250/tcp
ufw allow 8443/tcp
ufw allow 9090/tcp
ufw enable
ufw reload

sudo systemctl start cloudstack-management
sudo systemctl enable cloudstack-management

These steps I am following for the 1st time installation of cloudstack-management into a machine

---

On the 2nd time when I am installing cloudstack-management to add it as a management server I am following these steps....

I am not installing mysql on this machine just installing cloudstack-management

apt install chrony -y

sudo nano /etc/apt/sources.list.d/cloudstack.list
deb https://download.cloudstack.org/ubuntu jammy 4.22

wget -O - https://download.cloudstack.org/release.asc |sudo tee /etc/apt/trusted.gpg.d/cloudstack.asc

sudo apt update

sudo apt install cloudstack-management -y

sudo apt install openjdk-11-jdk -y

sudo update-alternatives --config java

nano /etc/sudoers

Defaults:cloud !requiretty

cloudstack-setup-management

ufw allow 8080/tcp
ufw allow 8250/tcp
ufw allow 8443/tcp
ufw allow 9090/tcp
ufw enable
ufw reload

sudo systemctl start cloudstack-management
sudo systemctl enable cloudstack-management

---

By following these I am able to install the management server on the 2nd machine but getting 503 error.

[DaanHoogland]

did you edit /etc/cloudstack/management/db.properties to point to the other server’s DB?

[arpanbht]

This is the db.properties file of the 1st management server which has mysql installed in it

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# management server clustering parameters, change cluster.node.IP to the machine IP address
# in which the management server is running
cluster.node.IP=192.168.1.103
cluster.servlet.port=9090
region.id=1

# CloudStack database settings
db.cloud.username=cloud
db.cloud.password=ENC(nZOnYfRvfzl24kBKsLH5TbNVLn+4c8Lwfhorh58onVyB)
db.cloud.host=localhost
db.cloud.driver=jdbc:mysql
db.cloud.port=3306
db.cloud.name=cloud

# Connection URI to the database "cloud". When this property is set, only the following properties will be used along with it: db.cloud.maxActive, db.cloud.maxIdle, db.cloud.maxWait, db.cloud.username, db.cloud.password, db.cloud.driver, db.cloud.validationQuery, db.cloud.isolation.level. Other properties will be ignored.
db.cloud.uri=


# CloudStack database tuning parameters
db.cloud.connectionPoolLib=hikaricp
db.cloud.maxActive=250
db.cloud.maxIdle=30
db.cloud.maxWait=600000
db.cloud.minIdleConnections=5
db.cloud.connectionTimeout=30000
db.cloud.keepAliveTime=600000
db.cloud.validationQuery=/* ping */ SELECT 1
db.cloud.testOnBorrow=true
db.cloud.testWhileIdle=true
db.cloud.timeBetweenEvictionRunsMillis=40000
db.cloud.minEvictableIdleTimeMillis=240000
db.cloud.poolPreparedStatements=false
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC

# CloudStack database SSL settings
db.cloud.useSSL=false
db.cloud.keyStore=
db.cloud.keyStorePassword=
db.cloud.trustStore=
db.cloud.trustStorePassword=

# Encryption Settings
db.cloud.encryption.type=file
db.cloud.encrypt.secret=ENC(SgVju339+M08rxB2O2xRLXMC91WR60PAkmJCVLYoyysuZXgk4byjdhYs)
db.cloud.encryptor.version=V2

# usage database settings
db.usage.username=cloud
db.usage.password=ENC(EBEcuZF4v7k4G6d7XNcyKqaMaHjxJjjcbzm6NMOLb0eI)
db.usage.host=localhost
db.usage.driver=jdbc:mysql
db.usage.port=3306
db.usage.name=cloud_usage

# Connection URI to the database "usage". When this property is set, only the following properties will be used along with it: db.usage.maxActive, db.cloud.maxIdle, db.cloud.maxWait, db.usage.username, db.usage.password, db.usage.driver, db.usage.validationQuery, db.usage.isolation.level. Other properties will be ignored.
db.usage.uri=


# usage database tuning parameters
db.usage.connectionPoolLib=hikaricp
db.usage.maxActive=100
db.usage.maxIdle=30
db.usage.maxWait=600000
db.usage.minIdleConnections=5
db.usage.connectionTimeout=30000
db.usage.keepAliveTime=600000
db.usage.url.params=serverTimezone=UTC

# Simulator database settings
db.simulator.username=cloud
db.simulator.password=cloud
db.simulator.host=localhost
db.simulator.driver=jdbc:mysql
db.simulator.port=3306
db.simulator.name=simulator
db.simulator.connectionPoolLib=hikaricp
db.simulator.maxActive=250
db.simulator.maxIdle=30
db.simulator.maxWait=600000
db.simulator.minIdleConnections=5
db.simulator.connectionTimeout=30000
db.simulator.keepAliveTime=600000
db.simulator.autoReconnect=true

# Connection URI to the database "simulator". When this property is set, only the following properties will be used along with it: db.simulator.host, db.simulator.port, db.simulator.name, db.simulator.autoReconnect. Other properties will be ignored.
db.simulator.uri=


# High Availability And Cluster Properties
db.ha.enabled=false
db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy
# cloud stack Database
db.cloud.replicas=localhost,localhost
db.cloud.autoReconnect=true
db.cloud.failOverReadOnly=false
db.cloud.reconnectAtTxEnd=true
db.cloud.autoReconnectForPools=true
db.cloud.secondsBeforeRetrySource=3600
db.cloud.queriesBeforeRetrySource=5000
db.cloud.initialTimeout=3600

#usage Database
db.usage.replicas=localhost,localhost
db.usage.autoReconnect=true
db.usage.failOverReadOnly=false
db.usage.reconnectAtTxEnd=true
db.usage.autoReconnectForPools=true
db.usage.secondsBeforeRetrySource=3600
db.usage.queriesBeforeRetrySource=5000
db.usage.initialTimeout=3600

---

And this is the db.properties of the 2nd management server which does not have mysql installed in it

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# management server clustering parameters, change cluster.node.IP to the machine IP address
# in which the management server is running
cluster.node.IP=127.0.0.1
cluster.servlet.port=9090
region.id=1

# CloudStack database settings
db.cloud.username=cloud
db.cloud.password=cloud
db.cloud.host=localhost
db.cloud.driver=jdbc:mysql
db.cloud.port=3306
db.cloud.name=cloud

# Connection URI to the database "cloud". When this property is set, only the following properties will be used along with it: db.cloud.maxActive, db.cloud.maxIdle, db.cloud.maxWait, db.cloud.username, db.cloud.password, db.cloud.driver, db.cloud.validationQuery, db.cloud.isolation.level. Other properties will be ignored.
db.cloud.uri=


# CloudStack database tuning parameters
db.cloud.connectionPoolLib=hikaricp
db.cloud.maxActive=250
db.cloud.maxIdle=30
db.cloud.maxWait=600000
db.cloud.minIdleConnections=5
db.cloud.connectionTimeout=30000
db.cloud.keepAliveTime=600000
db.cloud.validationQuery=/* ping */ SELECT 1
db.cloud.testOnBorrow=true
db.cloud.testWhileIdle=true
db.cloud.timeBetweenEvictionRunsMillis=40000
db.cloud.minEvictableIdleTimeMillis=240000
db.cloud.poolPreparedStatements=false
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC

# CloudStack database SSL settings
db.cloud.useSSL=false
db.cloud.keyStore=
db.cloud.keyStorePassword=
db.cloud.trustStore=
db.cloud.trustStorePassword=

# Encryption Settings
db.cloud.encryption.type=none
db.cloud.encrypt.secret=
db.cloud.encryptor.version=

# usage database settings
db.usage.username=cloud
db.usage.password=cloud
db.usage.host=localhost
db.usage.driver=jdbc:mysql
db.usage.port=3306
db.usage.name=cloud_usage

# Connection URI to the database "usage". When this property is set, only the following properties will be used along with it: db.usage.maxActive, db.cloud.maxIdle, db.cloud.maxWait, db.usage.username, db.usage.password, db.usage.driver, db.usage.validationQuery, db.usage.isolation.level. Other properties will be ignored.
db.usage.uri=


# usage database tuning parameters
db.usage.connectionPoolLib=hikaricp
db.usage.maxActive=100
db.usage.maxIdle=30
db.usage.maxWait=600000
db.usage.minIdleConnections=5
db.usage.connectionTimeout=30000
db.usage.keepAliveTime=600000
db.usage.url.params=serverTimezone=UTC

# Simulator database settings
db.simulator.username=cloud
db.simulator.password=cloud
db.simulator.host=localhost
db.simulator.driver=jdbc:mysql
db.simulator.port=3306
db.simulator.name=simulator
db.simulator.connectionPoolLib=hikaricp
db.simulator.maxActive=250
db.simulator.maxIdle=30
db.simulator.maxWait=600000
db.simulator.minIdleConnections=5
db.simulator.connectionTimeout=30000
db.simulator.keepAliveTime=600000
db.simulator.autoReconnect=true

# Connection URI to the database "simulator". When this property is set, only the following properties will be used along with it: db.simulator.host, db.simulator.port, db.simulator.name, db.simulator.autoReconnect. Other properties will be ignored.
db.simulator.uri=


# High Availability And Cluster Properties
db.ha.enabled=false
db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy
# cloud stack Database
db.cloud.replicas=localhost,localhost
db.cloud.autoReconnect=true
db.cloud.failOverReadOnly=false
db.cloud.reconnectAtTxEnd=true
db.cloud.autoReconnectForPools=true
db.cloud.secondsBeforeRetrySource=3600
db.cloud.queriesBeforeRetrySource=5000
db.cloud.initialTimeout=3600

#usage Database
db.usage.replicas=localhost,localhost
db.usage.autoReconnect=true
db.usage.failOverReadOnly=false
db.usage.reconnectAtTxEnd=true
db.usage.autoReconnectForPools=true
db.usage.secondsBeforeRetrySource=3600
db.usage.queriesBeforeRetrySource=5000
db.usage.initialTimeout=3600

---

[weizhouapache]

@arpanbht
you need to setup database again on the 2nd management server, but without deploy-as so that the database will not be recreated.

sudo cloudstack-setup-databases cloud:<password>@<database server ip> -e file -m <mgmt_key> -k <db_key> -i <mgmt_ip>

Alternatively, you can copy db.properties and key files from mgmt1 to mgmt2, and update cluster.node.IP in the db.properties
please note, the database server on mgmt1 must be accessible from mgmt2 (username: cloud).

[arpanbht]

Yes, i have tried sudo cloudstack-setup-databased on the 2nd server without --deploy-as now it is connected now i can access both of them.

Now I have a query i want to pass them throuh a load balancer to connect both of them and access it from a single ip, how can i do that?

[weizhouapache]

Yes, i have tried sudo cloudstack-setup-databased on the 2nd server without --deploy-as now it is connected now i can access both of them.

Now I have a query i want to pass them throuh a load balancer to connect both of them and access it from a single ip, how can i do that?

you need to set up a reverse proxy, for example nginx or haproxy.

if you also consider the similar setup for console and secondary storage, you can refer to some github threads or mailing list discussions, or use search tools or AI