Load Balancing 2 Web servers and 2 MySQL Servers
Posted by YYamagishi, 03-28-2008, 04:02 AM Hi, I was wondering if it is possible to cluster 2 web servers and 2 mysql servers with only one server working as load balancer. I am planning to use LVS (ldirectord and heartbeat). Let's say I have 3 IPs allocated to the load balancing server. 111.222.111.222 (Main IP) 111.222.111.223 (Web Load Balancing IP) 111.222.111.224 (MySQL Load Balancing IP) If a connection is made to .223 it would pass the request to one of the web nodes. If a connection is made to .224 it would pass the request to one of the MySQL nodes. Is it possible to do this? If not, can I run, for example, nginx on 223 IP address to provide forward proxy? (Then it would not be able to HA but the main point is to load balance so) Also, what would be the best way to keep the data same on both web servers? This is a web cluster for a very high traffic forum with a lot of uploads every hour so it has to do real time synchronization. I heard that DRDB is only one way and not two way so I'm not going to be able to use this. (^^not sure if that information is correct or not) Thanks, Yudai Yamagishi
Posted by HYB-Bryan, 03-28-2008, 09:55 AM Your load balancing/HA solution should work just fine. Two-way file replication is tough without some kind of shared storage. I'm assuming you don't have the option of connecting to a SAN. One option you could take would be to use an NFS mount on one server to store all of the files and then replication that to the other server with rsync. You'd have to figure out some way to change the mount if the "master" server failed.
Posted by PeHe, 03-28-2008, 10:50 AM Hello, My first post here. I just love this cluster stuff and want to share my experiences. I have setup similar environment. - 2 physical server - 4 Xen VS (2x front and 2x node) Front machines use LVS, ldirectord and hearbeat2. Actually one at time. Other one is for backup purposes. Those two node server's are basically AMP. MySQL Master-Master replication. Apache and PHP has been compiled from source and installed to /wwwroot That directory is shared via DRBD (in Dom0). FS is OCFS2 so it can be mounted both server's same time. This feature is available on DRBD version starting 8.x This is 100% idiot proof setup at least on that environment where I use it now. Im trying to now solve some OCFS2/DRBD performance issues because i want to put this same setup on our shared webhosting. I have noticed that multiple writing/reading at dirrefent nodes at same time is too slow and could cause high load. I think that answer to my problem is to setup AoE server (two for HA) and share devices from there. And after that i think im gonna use gPXE to boot physical server's from that storage too. That way i can setup cluster nodes with Quad-Core / 4GB RAM starting at 450e. That's damn cheap! If someone has succesfully setup this kind of system to multiple user system please advise me. PS. You should check below links so you get better picture what im talking about. You can't use that information directly at least my setup need tweaking before i was happy. howtoforge.com/mysql_master_master_replication howtoforge.com/high_availability_loadbalanced_apache_cluster xenaoe.org/ Regards, Petri Heinonen Last edited by PeHe; 03-28-2008 at 10:57 AM. Reason: Added links
Posted by YYamagishi, 03-28-2008, 02:11 PM I was thinking about doing master-master replication method if that won't cause any problems
Posted by YYamagishi, 03-28-2008, 02:21 PM Currently my servers are overloaded so I won't be using Xen for sure. About OCFS2 over DRBD, if I mount the filesystem on both servers at, say /mnt10, will both servers be able to write to the filesystem and still see the changes on the other server without any problems? (Example SRV1 --> WRITES TO /mnt10/uploads/ -- SRV2 CAN SEE THE DATA? SRV2 --> WRITES TO /mnt10/uploads/ -- SRV1 CAN SEE THE DATA?) Also what about permissions? If I have same UID/GID on both servers for a certain user that lighttpd runs on, can I still have same permissions in the OCFS2 mounted datas on both the servers? Currently I'm thinking about lighttpd chroot to /wwwdata with network data mounted to /wwwdata/public/ Running SAMBA on one of the servers seems to be a good idea but then I would have some lag as the web server will have to fetch the data off the samba server then run SQL queries on the SQL server, after that finally finish processing the output then pass the data on to the client. As for network card, I can only get 100Mbit as my datacenter does not support 1Gbit Thanks for your help
Posted by PeHe, 03-28-2008, 05:42 PM Of course they see the changed data. That's the whole point of that setup. But you have to use some cluster fs. OCFS2/GFS/Lustre etc.. Yes. If UID/GID are same permission are same also. It's basically just normal filesystem, but you could mount it with multiple server. I guess that 100Mbit is not problem if there are not much data what is changing. It only uses bandwidth for changed data so after sync there are not much traffic. But i really recommend that you should use cross-over cable to connect those server's each other. If that is not option and data is synced in public network you absolutily have to setup some sort of encryption there. DRBD has support for that as far as i can remember. And my opinion about samba is that you should not use that on Linux. We have NFS for that. And i don't really warm up for NFS also. I like to share data with block level not file level. It's always better, faster and reliable. And always use LVM so you could easily take snapshot and backup's without any downtime. Hmm..This just come up to my mind that maybe you could use LVM encryption for DRBD/OCFS2 volume. Don't know what that's gonna do for performance. And if you are going to use this setup you should remember this. Performance is best when there are not lot's of writing from both server's same time. So configure HTTPD log's to some other dir then OCFS2. I didn't realize this at beginning and when benchmarking that setup with 1000 concurrent connection load go up to 200.00 !! And one more think. You say that you have overloaded servers. Don't know what you are running there but if it is PHP/MySQL then you should take look at APC and memcache. Those reduce load about 50%. And compile those critical softwares from source. (Well tha't not gonna do much but still) Last edited by PeHe; 03-28-2008 at 05:46 PM.
Posted by YYamagishi, 03-29-2008, 07:39 AM I actually do have APC set up but not memcache as the forum software do not support it. I won't be able to use a cross-over connection but I think I can do a private network between the servers. Currently I have this in mind (All servers are CentOS 5): LB1 (Load Balancer by LVS and nginx) IP Address to Public: 111.222.111.10 (Web Load Balancer with nginx) IP Address to Public: 111.222.111.11 (MySQL Load Balancer with LVS) IP Address to Public: 111.222.111.12 (VPN Server to get in to the private network + General Use) IP Address to Private: 10.0.0.2 Web1 (Web server by Lighttpd) IP Address to Public: 111.222.111.20 (For Web server to listen on) IP Address to Private: 10.0.1.2 (Data synchronization + Administration) Web2 (Web server by Lighttpd) IP Address to Public: 111.222.111.21 (For Web server to listen on) IP Address to Private: 10.0.1.3 (Data synchronization + Administration) SQL1 (MySQL Server) IP Address to Public: 111.222.111.31 (For MySQL server to listen on) IP Address to Private: 10.0.2.2 (Data synchronization + Administration) SQL2 (MySQL Server) IP Address to Public: 111.222.111.32 (For MySQL server to listen on) IP Address to Private: 10.0.2.3 (Data synchronization + Administration) Basically when a user accesses the web site, the load balancer (nginx) will proxy the request to either Web1 or Web2 then either of them will access the load balancer's IP with MySQL server balancing. After that, either web will generate the PHP and sends the output to nginx. Then the client gets the file. Is this a good setup? Also, what would be the best way to setup the private network? Servers having 2 ethernet cards with each access to different network or just one ethernet card and do VPN? Thanks P.S. If I have thousands of connections per second on the load balancer, will it be a problem? I read somewhere that: Mar 5 00:38:50 cn1 kernel: printk: 714 messages suppressed. http://www.webhostingtalk.com/showthread.php?t=676270 Problem specified above is caused by too many connections. Is that true?
Posted by YYamagishi, 03-29-2008, 08:02 AM Tried editing the post but I couldn't and I don't have time so I'll just post it. I'm thinking about using GFS for file synchronization after some reading. This is an IPB forum with very large number of uploads per hour and I'm really not sure which serves me the best with file synchronization but thinking about integrity with CentOS, I thought GFS would be the best. But, I'm not sure on this one as well so any comments over GFS, OCFS2, etc. would be appreciated as well. Also some of you may tell me that IPB has support for memcached and yes, I know it does support in the latest version but since I the modification I use which is very critical to me isn't supported in the latest version, I'm still using 2.1.7 which doesn't support memcached. Thanks
Posted by wKkaY, 03-29-2008, 10:24 AM Just how large is "very large number of uploads"? Per day, in terms of 1) # files and 2) # bytes?
Posted by 1boss1, 03-29-2008, 11:00 PM How would 1 server for webpages, 2 for MySql and 1 for images/attachments benchmark in comparison to your proposed 4 server? Would it be similar in performance but easier to keep everything synced up and administrate?
Posted by YYamagishi, 03-30-2008, 03:42 AM 1 around 100 #2 all of them are below 100kb avarage is 50kb
Posted by YYamagishi, 03-30-2008, 03:46 AM It wouldn't change anything as I tried that few months ago. It may speed up the web server if I have the image server in say United States, but by splitting static and dynamic won't make a difference as most of the contents (98%) is composed of dynamic data and there isn't much static contents such as images. Also, the attachments/uploads are images so it won't really much make a lot of difference and with how IPB handle image attachments, it would be pain in the *** trying to set up that kind of cluster.