Get started
Starting with an Amazon AMI
Image can be found under public repo : ScrambleDB-0.2-centos6-x86_64.
This will provision :
- 6G for the OS partition
- 20G XFS partition for any data requirement
You need also to install a private cluster control the starter pack, a minimal extract of scambledb to control the cloud.
To be placed in the default prefix /usr/local/
You can change the install directory prefix and name by editing the init scripts
You should get :
- Python dependencies
- Embedded Perl
- Gearman
- ScrambleDB ncc - Node cluster control
Downloads :
scramble_starter_pack-130402.macosx10.x86_64.tar.gz
scramble_starter_pack-130402.centos6.x86_64.tar.gz
First login to your account and create the following elements first to start using ScrambleDB
a vpc
a security groups
a subnet
an elastic ip
an network interface
Report those informations to the cloud.cnf configuration file cloud section
Associate one VIP for your subnet attached the network interface this will be reported in the configuration in lb ( 10.0.0.10 in the sample tiny configuration) define IPs for your services in the range of your subnet
ScrambleDB should have very few dependences but some components of the stack are hard to bundle in a prefix so boost and libevent should be install in your system. Those limitations could be removed in a near feature with the help of more contributors. Python is not bundle but the extra libraries are prefixed as well under the base directory.
Create the cluster configuration file on your private network or your personal computer A simple config file is provide named etc/cloud.tiny as a template copy it to cloud.cnf
ncc/etc/cloud.cnf
For each cloud that is under control you should put your public_key in the .ssh directory under the datadir(/var/lib/skysql) and enter the name of the public_key in the cloud section of the configuration file
Select the unique active cloud that you want to control by setting the status to master
<cloud cloud2>
status master
user xxx
password xxxx
host na
version 1.5
driver EC2
template ami-cb23a0a2
key SDS145000
zone us-east-1b
region us-east
instance_type t1.micro
security_groups secure-group-vpc
vpc vpc-7032621b
subnet subnet-49326222
public_key SDS145000.pem
elastic_ip 107.21.41.133
elastic_ip_id eipalloc-1024747b
</cloud>
Today only amazon and bare-bone is integrated further work with libcloud will bring more services on openstack and vcloud
The etc/cloud.cnf will be copied on the cloud during the bootstrap command but prior to this ScambleDB obscur the password lines that this information will never be store inside a VM
elastic_ip_id eipalloc-1024747b
Is the allocation id of your public elastic IP
How to define a local architecture
<cloud cloud3>
status master
user xxxx
password xxxx
host na
version 1.5
driver LOCAL
template na
key na
ex_vdc na
instance_type na
public_key id_rsa
</cloud>
Install your public key in authorized_key un put it under the $DATADIR/.ssh directory
Now start the bin/tunnel script to start the local node clould control
<db node10>
mode mariadb
status master
ip 10.0.0.102
peer node11
mysql_version mariadb
mysql_port 5010
mysql_cnf etc/my.template
mem_pct 30
io_pct 30
cloud cloud3
</db>
Db services MariaDB, Spider, Galera or Ndb
We suggest at list 3 instances to enable the load balancing in case of switch over during maintenance operation on the cluster
We will build 2 types of architectures for write scalability.
- Spider for the master
- MySQL cluster for the master
Peer
Used to mark replication relation in the case of a master this is not relevent but for the slave it should report the master as it is used for join operation to a replication cluster
mysql_cnf
Template that is overwrite by the other parameters in the configuration file
This is important that the it represent a minimun memory footprint as it can prevent service to start for the first time
mysql_version
Should be used to select the bindir directory under $SKYBASEDIR directory
<proxy proxy1>
mode mysql-proxy
ip 10.0.0.102
hosts node10
peer node11
port 3307
script rw-splitting.template
</proxy>
A key component of the stack it use the rw-splitting template that is overide to always talk to a memcache director when the state of the cluster change I would recommand to have on proxy for each db in stack
###memcache
<nosql nosql1>
ip 10.0.0.102
mode memcache
status master
port 11211
mem 16
</nosql>
Required service, ScrambleDB need at least one to store the global transaction per table id to enable smart SQL routing. ScambleDB store the status of each service comming from instances heartbeat inside thee director. this enable to detect change on the cluster state and take decision for election or sucide services.
Those election will be delegated to a service worker_doctor that will reset the cluster to a working state
Like serving as abitrtor role
###keepalived
<lb lb1>
ip 10.0.0.102
vip 10.0.0.10
port 3306
peer lb2
status mastermode keepalived
device eth0mysql_user skysql
mysql_password skyvodka
</lb>
SrambleDB main VIP delivery service It does not use kernel LVS to load balance that role is delegate to ha-proxy not to be force to manage NAT integration inside the private network
- Need 2 services on 2 instances
- The elastic IP to be pointed to the VIP
VIP should be added for sharding capabilities queries will be associated to a VIP and vip should point to a pool of sphinx , spider , nosql cluster. mysql-proxy lua script will ask the to tarantool id this query is associated to a special pool
###haproxy
<lb lb3>
ip 10.0.0.102
vip 0.0.0.0
port 3306
status master
mysql_user skysql
mysql_password skyvodkapeer lb4
mode haproxy
device eth1
</lb>
ScambleDB main load balancer. It enable to remove one mysql-proxy from the pool if that service is not more evalable
ip
Tell where the service is located and so far should be collocated with the instances holding the VIP
vip
Use that 0.0.0.0 to be listen answer communication from the VIP
mysql connection variables
Those variables are used for the heartbeat status to check that protocol is able to connect from the VIP
<monitor mon1>
mode mycheckpoint
port 8080
smtp_from [email protected]
smtp_to [email protected]
smtp_host localhost
ip 10.0.0.102
purge_days 1
</monitor>
Straight implementation of open source monitoring.
Can be allocated to a db instances but for now i prefer collecting data in the master db by insting to the VIP
The worker_heartbeat.pl on each instances call the worker_cluster_cmd.pl script with the ping command in ping a collection of status is collected for all db services and store to the VIP representing the logical database abstraction.
###bench
<bench bench1>
ip 10.0.0.47
mode dbt2
peer lb1
warehouse 4
concurrency 4
duration 20datadir /var/lib/skysql/dbt2
</bench>
This is all good but does it work? This is the reason we have the possibility to start a quick dbt2 benchmark to create some traffic on the instances and watch out the load
{"level":"services","command":{"action":"start","group":"bench1","type":"all"}}
{services:[],
results:{
data:{
rampup:-2,
errors:0,
start_time:'1358290419',
metric:'35.6756756756757',
duration:'0.616666666666667',
mix:{
data:[
{
ctime:'1358290419',
thread_id:'1694467840',
response_time:'0.004255',
transaction:'S'
},
{
ctime:'1358290454',
thread_id:'1694603008',
response_time:'0.198661',
transaction:'n'
}
]
},
steady_state_start_time:'1358290417',
transactions:{
transaction:[
{
rt_avg:'0.4297025',
rt_90th:'0.845612',
name:'Delivery',
rollbacks:0,
rollback_per:0,
total:2,
mix:'3.2258064516129'
},
{
rt_avg:'0.829707954545455',
rt_90th:'0.2035335',
name:'New Order',
rollbacks:6,
rollback_per:'21.4285714285714',
total:28,
mix:'45.1612903225806'
},
{
rt_avg:'0.138447',
rt_90th:undef,
name:'Order Status',
rollbacks:1,
rollback_per:50,
total:2,
mix:'3.2258064516129'
},
{
rt_avg:'0.4839306',
rt_90th:'0.067655',
name:'Payment',
rollbacks:6,
rollback_per:'23.0769230769231',
total:26,
mix:'41.9354838709677'
},
{
rt_avg:'0.2708255',
rt_90th:'0.534743',
name:'Stock Level',
rollbacks:2,
rollback_per:50,
total:4,
mix:'6.45161290322581'
}
]
}
}
}
}
Once it is setup start the cluster with the init script
ncc/init.d/clusterd start
If you are deploying in the cloud start the starter pack in your private instance
ncc/init.d/tunneld start
The init script is setting some important variables that will be used to localise component of the stack
SKYDATADIR="/var/lib/skysql"
SKYBASEDIR="/usr/local/skysql"
SKYUSER=skysql
SKYGROUP=skysql
PYTHONUSERBASE=$SKYBASEDIR/MySQL-python
With the command line tools it is time to propage the binaries to all the cluster node
What services are started
/usr/local/skysql/perl/bin/perl /usr/local/skysql/ncc/bin/worker_node_cmd.pl
/usr/local/skysql/perl/bin/perl /usr/local/skysql/ncc/bin/worker_cluster_cmd.pl
/usr/local/skysql/perl/bin/perl /usr/local/skysql/ncc/bin/worker_write_config.pl
/usr/local/skysql/perl/bin/perl /usr/local/skysql/ncc/bin/worker_heartbeat.pl
/usr/local/skysql/perl/bin/perl /usr/local/skysql/ncc/bin/worker_cluster_doctor.pl
/usr/bin/python /usr/local/skysql/ncc/bin/worker_cloud_cmd.py
worker_heartbeat.pl
This the status monitoring heartbeat It collect the instances information and his own view of the world
- Local Memory
- Local Interfaces
- Services status
- Instances status
Services et Instances status is asked to the local worker_cluster_cmd JSON Result is send back to the local worker_cluster_cmd in the ping command to be stored in memcache and compared with previous states
###Check status
ncc/clmgr services bootstrap_binaries all
{command:{action:'status',group:'all',type:'all'}}
{
services:[
{
mode:'master',
ip:'10.0.0.102',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'node10',
state:'Database communication failure '
},{
mode:'slave',
ip:'10.0.0.74',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'node11',
state:'Database communication failure '
},{
mode:'slave',
ip:'10.0.0.56',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'node12',
state:'Database communication failure '
},{
mode:'memcache',
ip:'10.0.0.102',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0005',
name:'nosql1',
state:'Memcache communication failure '
},{
mode:'memcache',
ip:'10.0.0.56',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0005',
name:'nosql2',
state:'Memcache communication failure '
},{
mode:'mysql-proxy',
ip:'10.0.0.102',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'proxy1',
state:'Database communication failure '
},{
mode:'mysql-proxy',
ip:'10.0.0.74',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'proxy2',
state:'Database communication failure '
},{
mode:'mysql-proxy',
ip:'10.0.0.56',
time:'Sat Dec 8 12:43:20 2012',
error:'ER0003',
name:'proxy3',
state:'Database communication failure '
},{
mode:'keepalived',
ip:'10.0.0.102',
time:'Sat Dec 8 12:43:23 2012',
error:'ER0003',
name:'lb1',
state:'Database communication failure '
},{
mode:'keepalived',
ip:'10.0.0.74',
time:'Sat Dec 8 12:43:26 2012',
error:'ER0003',
name:'lb2',
state:'Database communication failure '
},{
mode:'haproxy',
ip:'10.0.0.102',
time:'Sat Dec 8 12:43:26 2012',
error:'ER0003',
name:'lb3',
state:'Database communication failure '
},{
mode:'haproxy',
ip:'10.0.0.74',
time:'Sat Dec 8 12:43:26 2012',
error:'ER0003',
name:'lb4',
state:'Database communication failure '
}
]
}
At this point you should have a running cluster controller and stack on all instances for ips defined in your configuration file
provision the services for the first install
./clmgr services install all
Some product like MariaDB need to create system tables load UDF functions
Instruction do is done in : the worker_cluster_command.pl that will later call the worker_node_cmd on each server that host the service. Reporting to the console of each step send to the node should be printed in a json class
In this stage scambledb build for you the configuration file for
- mysql
- memcached
- keepalived
- haproxy
- mha
- mysql-proxy
- lua route scripts
Those file are bootstrap to the servers but can be manualy reproduce using
ncc/clmgr services bootstarp_ncc
Start the services
ncc/clmgr services start all
Check the status of the cluster :
ncc/clmgr services status all
ncc/clmgr instances launch 10.0.0.x
{"level":"instances","command":{"action":"status","group":"all","type":"all"}}
{
instances:[
{
ip:'10.0.0.209',
id:'i-7fba6203',
state:'stopped'
},
{
ip:'10.0.0.10',
id:undef,
state:undef
},
{
ip:'10.0.0.102',
id:'i-987388e6',
state:'running'
},
{
ip:'10.0.0.48',
id:'i-4436ed3a',
state:'stopped'
},
{
ip:'10.0.0.47',
id:'i-4236ed3c',
state:'stopped'
}
],
return:{
cloud:{
status:'master',
zone:'us-east-1b',
key:'SDS145000',
elastic_ip:'107.21.41.133',
password:'xxx',
vpc:'vpc-7032621b',
user:'AKIAJR7YEOZPXASJCYDQ',
public_key:'SDS145000.pem',
security_groups:'secure-group-vpc',
subnet:'subnet-49326222',
template:'ami-cb23a0a2',
version:'1.5',
region:'us-east',
host:'na',
ex_vdc:'na',
instance_type:'t1.micro',
driver:'EC2'
},
command:{
group:'all',
action:'status',
type:'all'
}
}
}
ncc/clmgr services swith db2
ncc/clmgr services sync db2