sorted by: recent | see :
popular
Content Tagged with systems + linux
Distributed Image Management for Linux Clusters
Data Recovery Toolkit for InnoDB Released
I’m returned from my 1-week vacation today and want to say - I’ve never been so productive as I was there 
Blue ocean, hot sun and white sand really helped me to finish my work on the first release of one really awesome project.
Today I’m proud to announce our first public release of the Data Recovery Toolkit for InnoDB - set of tools for checking InnoDB tablespaces and recovering data from damaged tablespaces or from dropped/truncated InnoDB tables.
Why I HATE ?smart? Software: Cpanel vs Consulter
Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…
After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) - it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.
Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up user’s data in such situation. But what idiots in cpanel development team decided, that is it appropriate and acceptable to perform such operations?! As an administrator and as a software developer I do not understand them - I just can’t understand such approach….
So, enough complaining - here is a piece of useful information for my readers: If you’re so unlucky to have cpanel installed on your server and you’d like to upgrade your mysql manually, then you can perform following operations:
# chattr +i /etc/mysqlupdisable
# service cpanel restart
After these small changes your cpanel will forget about mysql upgrades and you’ll be able to do what you want and not what some dumb developers decided you should do.
MySQL Master-Master Replication Manager 1.0 Released
It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).
Changes from previous release:
- Perl semaphores implementation caused huge memory leaks (mmmd_mod).
- Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
- Removed legacy StartSlave method from agent code which caused problems on some Perl versions
- Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
- Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
- Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
- Added some more docs to the project site.
New version can be obtained here or from the project’s SVN repository.
Knoppix 4.0 DVD - Like a Kid in a Candy Store - O'Reilly Sysadmin
Plan 9 from User Space
MacNN | Q emulator gains fast-switching, more
open-source: del.icio.us tag/open-source
Operating
linux
emulator
Windows
systems
open-source
emulate
MMM Release 1.0-pre3
MySQL Master-Master Replication Manager version 1.0-pre3 has been released today. Changes list is really short now:
- Major fix in multiple clusters support - now you can use many clusters with one monitoring node (details are in mmmd_mon man page)
- Man pages for mmmd_mon, mmmd_agent and mmm_control scripts
- Startup scripts added/fixed for mmmd_mon and mmmd_agent
- Installation script now requires iproute package to be installed on server.
As always, if you have any questions/suggestions, post them here or in mmm-devel mail list.
Master-Master Replication Example using MMM
Despite of my high load at work I decided to release mmm-1.0-pre2 today. There are some small, but critical fixes added and much more coming next week (or little bit later if mysqlconf will take more time than I think).
After the first alpha release I’ve received lots of emails, some messages in mmm-devel mail list and even some bug reports in Google Code bug tracking. One of the most asked things was documentation. So, I decided to write some posts in this blog (sorry to non-sql-related readers) and them compose some docs for final release using these posts and comments from readers. This post will be first in mmm-series and will describe how to use mmm in simple master+master scheme where one master accept write requests and both masters accept read requests. This post will provide you with detailed instructions about MySQL setup, permissions setting, mmm installation and configuration and cluster management.
Network Infrastructure
All my example configs in this article will be based on the following network infrastructure:
- Web Server + MMM Monitoring Server - 192.168.1.1
- MySQL Server db1 - 192.168.1.111
- MySQL Server db2 - 192.168.1.112
All servers are connected to the same switched network.
Software Prerequisites
Before you’ll begin your setup, take a looks at the following list of prerequisites for each server in your cluster to be sure that you have all mentioned packages/modules/etc.
Each of mysql servers in the cluster should have iproute2 package to let mmm manage IP addresses on these servers with ip command. As for perl modules, you can run install.pl script and it’d say what do you need to add to your system before installation will be possible.
MySQL Servers Setup
First of all, you need to setup both of your MySQL servers to replicate data from each other. Example configs are following:
my.cnf at db1 should have following options:
log_bin = /var/log/mysql/mysql-bin.log
my.cnf at db2 should have following options:
log_bin = /var/log/mysql/mysql-bin.log
Replication settings (db1):
...
mysql> change master to master_host='192.168.1.112', master_port=3306, master_user='replication', master_password='slave';
...
mysql> slave start;
Replication settings (db2):
...
mysql> change master to master_host='192.168.1.111', master_port=3306, master_user='replication', master_password='slave';
...
mysql> slave start;
After all these operations were made your servers will have SHOW SLAVE STATUS results like following:
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.112
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000026
Read_Master_Log_Pos: 98
Relay_Log_File: db1-relay-bin.000339
Relay_Log_Pos: 235
Relay_Master_Log_File: mysql-bin.000026
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 98
Relay_Log_Space: 235
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
1 row in set (0.00 sec)
Setting up mmm agents
Each mysql server should have one mmmd_agent running on it. To set them up you will need to install mmm as following:
# cd ~/mmm
# wget http://mysql-master-master.googlecode.com/files/mmm-1.0-pre2.tar.bz2
...
# tar xzf mmm-1.0-pre2.tar.bz2
# cd mmm-1.0-pre2
# ./install.pl
...
Installation is done!
#
After mmm installation you’ll need to configure your agents.
db1 config /usr/local/mmm/etc/mmm_agent.conf:
# Master-Master Manager config (agent)
#
# Debug mode
debug no
# Paths
pid_path /usr/local/mmm/var/mmmd_agent.pid
bin_path /usr/local/mmm/bin
# Logging setup
log mydebug
file /usr/local/mmm/var/mmm-debug.log
level debug
log mytraps
file /usr/local/mmm/var/mmm-traps.log
level trap
# MMMD command socket tcp-port and ip
bind_port 9989
# Cluster interface
cluster_interface eth0
# Define current server id
this db1
mode master
# For masters
peer db2
# Cluster hosts addresses and access params
host db1
ip 192.168.1.111
port 3306
user rep_agent
password RepAgent
host db2
ip 192.168.1.112
port 3306
user rep_agent
password RepAgent
db1 config /usr/local/mmm/etc/mmm_agent.conf:
# Master-Master Manager config (agent)
#
# Debug mode
debug no
# Paths
pid_path /usr/local/mmm/var/mmmd_agent.pid
bin_path /usr/local/mmm/bin
# Logging setup
log mydebug
file /usr/local/mmm/var/mmm-debug.log
level debug
log mytraps
file /usr/local/mmm/var/mmm-traps.log
level trap
# MMMD command socket tcp-port and ip
bind_port 9989
# Cluster interface
cluster_interface eth0
# Define current server id
this db2
mode master
# For masters
peer db1
# Cluster hosts addresses and access params
host db1
ip 192.168.1.111
port 3306
user rep_agent
password RepAgent
host db2
ip 192.168.1.112
port 3306
user rep_agent
password RepAgent
Now you can run mmmd_agent on each server and your servers would be ready for management with mmm.
MMM Server Installation and Configuration
When everything is done on mysql servers, you are ready to set up monitoring node which of course could be combined with some web-server node or another services - dedicated hardware is not required. Before configuration step you’ll need to install mmm just as it was done on mysql servers. Then you’ll need to create configuration file for mmmd_mon program which would monitor your nodes. Config file for our example scheme could be like following:
Config file for monitoring node - /usr/local/mmm/etc/mmm_mon.conf:
# Master-Master Manager config (monitor)
#
# Debug mode
debug no
# Paths
pid_path /usr/local/mmm/var/mmmd.pid
status_path /usr/local/mmm/var/mmmd.status
bin_path /usr/local/mmm/bin
# Logging setup
log mydebug
file /usr/local/mmm/var/mmm-debug.log
level debug
log mytraps
file /usr/local/mmm/var/mmm-traps.log
level trap
# MMMD command socket tcp-port
bind_port 9988
agent_port 9989
monitor_ip 127.0.0.1
# Cluster interface
cluster_interface eth0
# Cluster hosts addresses and access params
host db1
ip 192.168.1.111
port 3306
user rep_monitor
password RepMonitor
mode master
peer db2
host db2
ip 192.168.1.112
port 3306
user rep_monitor
password RepMonitor
mode master
peer db1
#
# Define roles
#
active_master_role writer
# Mysql Reader role
role reader
mode balanced
servers db1, db2
ip 192.168.1.201, 192.168.1.202
# Mysql Writer role
role writer
mode exclusive
servers db1, db2
ip 192.168.1.200
#
# Checks parameters
#
# Ping checker
check ping
check_period 1
trap_period 5
timeout 2
# Mysql checker
check mysql
check_period 1
trap_period 2
timeout 2
# Mysql replication backlog checker
check rep_backlog
check_period 5
trap_period 10
max_backlog 60
timeout 2
# Mysql replication threads checker
check rep_threads
check_period 1
trap_period 5
timeout 2
With this configuration file you will get 3 interface IP addresses used to “speak” with your cluster:
- Writer IP (192.168.1.200) - this address should be used to send write requests to your server.
- Reader IPs (192.168.1.201 and 192.168.1.202) - addresses for read-only requests.
Before you’ll start your monitoring part of the cluster, you need to be sure what mmm_mon will be able to connect to your servers with credentials from mmm_mon.conf file (run this command on one node and, if your replication was set up correctly (you’ve already tested it, right?), another server will get this statement by replication:
MMM Monitoring and Management Hints
When your configuration will be finished and mmmd_mon will be started, you obviously would need to take a look at mmm_control script which is small program dedicated to sending commands to mmmd_mon process and output results in nice format. To view its params info you can start it without any parameters. At this moment you can use following commands:
- show - displays list of servers with status info and bound roles.
- ping - sends ping command to local mmmd_mon daemon to check if it is running
- set_online host_name/set_offline - changes statuses of specified server.
- move_role role_name host_name - asks mmmd_mon to move specified role to specified host (useful for exclusive roles (like writer)
When your mmmd_mon script will be started first time, it will think what all servers were offline and now they came back. So, initial status of all servers will be set to AWAITING_RECOVERY and you’ll need to put both servers to ONLINE:
Config file: /usr/local/mmm/mmm_mon.conf
[2007-04-23 09:49:15]: Sending command 'SET_ONLINE(db1)' to 127.0.0.1
Command sent to monitoring host. Result: OK: State of 'db1' changed to ONLINE. Now you can wait some time and check its new roles!
# mmm_control set_online db2
Config file: /usr/local/mmm/mmm_mon.conf
[2007-04-23 09:49:53]: Sending command 'SET_ONLINE(db2)' to 127.0.0.1
Command sent to monitoring host. Result: OK: State of 'db2' changed to ONLINE. Now you can wait some time and check its new roles!
# mmm_control show
Config file: /usr/local/mmm/mmm_mon.conf
[2007-04-23 09:50:31]: Sending command 'PING()' to 127.0.0.1
Daemon is running!
Servers status:
db1(192.168.1.111): master/ONLINE. Roles: reader(192.168.1.201;), writer(192.168.1.200;)
db2(192.168.1.112): master/ONLINE. Roles: reader(192.168.1.202;)
#
So, that’s it. If you have any questions or suggestions, you can leave then in comments below or post them to mmm mail list.
