Open Dental Cluster (dual servers)!

[Justin Shafer]

YAhoo! Open Dental Server Cluster is working. Dual Servers always acting as 1. If one fails, you get an email and you can keep working. This was just a small test using vmware and an open dental database. MySQL Cluster is functional, next I have to work on samba.

You can see Ubuntu Server1, Ubuntu Server2, and Windows XP Professional.
Server1=192.168.1.198
Server2=192.168.1.199
Virtual Server=192.168.1.200
Windows XP client always talks to Virtual Server @ 192.168.1.200




With both servers syncronized, and Server1 acting as the primary database server, I yanked its network cable, and within 10 seconds Server2 became the database server and the Windows XP Professional PC was able to finish making an appointment with Open Dental, even though Server1 was down.. Then I rebooted Server1, and once booted, Server1 syncronized with Server2, and then became the primary server in the node and made Server2 become a secondary yet again. This works vice-versa. If Server 2 goes down, Server 1 stays as the primary waiting for Server2 to come back up.... If Server1 goes down, Server 2 becomes primary and waits for Server1 to become healthy.



I used Ubuntu Linux 8.10, DRBD8 and Heartbeat2. Right now I have the configured for automatic recovery which in the real world I may not have done. So lets say if Server 1 and Server 2 are present, and Server2 goes down and then comes back up, Server2 must be manually added back to the cluster for safe measure.

Here is my drbd.conf file.
resource db
{
protocol C;
startup { wfc-timeout 0; degr-wfc-timeout 20; }
disk { on-io-error detach; } # or panic, ...
syncer {
rate 30M;
}
on server1 {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.1.198:7778;
meta-disk internal;
}
on server2 {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.1.199:7778;
meta-disk internal;
}
net {
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
}

I will post my heartbeat files when I have samba up and running on the cluster with mysql....

[drtech]

This is great! I will be interested in trying this out sometime.

[Mifa]

Pardon my ignorance: what is the benefit of this setup vs. a cluster of RAID configured network drives hosting OD's MySQL database? Thanks.

[Justin Shafer]

doh

[Justin Shafer]

Pardon my ignorance: what is the benefit of this setup vs. a cluster of RAID configured network drives hosting OD's MySQL database? Thanks.

Well, lets say you have a NAS device that has a raid Array on the network, and the mysql database is stored on the NAS. This is good. Then lets say we have a server that is running mysql server, and of course this server running mysql server saves all the data to the NAS.

But what will happen if the server running mysql crashes? Sure the data will be okay on the NAS, but you still have to make another computer ready to be a server. And if another server is ready to go, you still will have to start mysql on this other server so clients can talk to the mysql database.... With a cluster, you dont have to make another computer a server, nor do you really need Network Attached Storage.

Imagine 2 Servers that both have raid, and always have the same databases and data, and always run mysql server and filesharing, and if one server dies, mysql still runs without your clients losing the ability to talk to a server, in fact to the workstations, they only see 1 server, they dont really know their are two, thanks to a "virtual server" that is reality two servers. And DRBD is synchronous replication, without getting too specific, its nice. As many cluster software doesnt let you do synchronous, only asynchronous replication between the two servers.

I kinda suck at explaining but that sounds about right. NAS is cool, but a cluster is one notch up. Especially when you have a raid array per server...

You could shoot one of the servers with a shotgun, even if it was running mysql, and the workstations would NOT lose mysql access to the database, you would have to shoot both servers for that.... When one server goes down, the other server picks up where it left off automatically...

And NAS can be used with the cluster if you want it too. So you could have two servers acting as a virtual server and instead of saving the data on both the servers internal hard drives that are always syncing with eachother, they could save the data to a NAS instead. It works both ways. Thats more a shared storage (NAS or SAN) vs file replication (DRBD) argument, which can both be used in a cluster (heartbeat2).

[richardwaite]

This is the coolest thing I've seen in a while. Do you know if there any any similar software solutions for Windows?

[Justin Shafer]

There are solutions to do clusters with Windows and MySql... But prepared to whip out your wallet. I would say for a Windows Cluster with SteelEye as the vendor for replication, you would be spending close to $10,000 or more. With 2 servers, Windows HPC 2008 Server, with 2 nodes, and SteelEye... Probably between $8,000-$10,000.00... And that doesnt include the IT guy having to set it all up. And that is a cluster with file replication. Shared Storage can be even more expensive... depending if the shared storage is redundant or not.

With Linux, the cost is just 2 Servers, and labor. No software to buy. So its more around $3000.00 without the IT guy.

[Justin Shafer]

http://www.mysql.com/products/enterprise/drbd.html

More info about it.

[wjstarck]

Hmmm. Makes me wonder if one could do away with RAIDs entirely (or at least on one of the servers anyways), as the likelihood of two disk drives failing at the same time would be highly unlikely. I like the idea of virtual servers on VMware, although VMware Server 2 is not quite ready for production environments yet...

[richardwaite]

Actually, it's pretty common for drives to fail at the same time. Usually it's because two drives came from the same manufacturing batch (you likely purchased them together, at the same time, at the same place) and the added stress of reconstructing the RAID (or copying all the data over, in the case of this thread's configuration) after you've added the replacement drive pushes the remaining drives over the edge. Chance for failure depends largely on the volume of data you're dealing with and the type of redundancy employed (more drives, like in RAID 5, means more chances for failure).

As long as you don't depend on RAID (or clustering) as a backup solution, which you shouldn't, it's not a huge deal. Just something to keep in mind.

[Justin Shafer]

Correct you are. I bought 8 WD drives that were large capactiy over a year and all 8 have failed. All were purchased over different times, but the manufacturing date on all of the 400GB drives I bought were the same day, and the 500GB drives, well, those failed too. I have been leaning torwards seagate lately on drives, though I just bought a new WD Raptor, and I love it. But I feel like WD let their normal inexpensive drives fall apart as far as longevity goes../.. except for the passport line... I have had good luck with those not failing for clients. Not like these Seagate FreeAgent drives.... Sigh.

You cant have your cake and eat it too... Modern drives these days arent like the old 1.6GB caviar drives in the past.

[Justin Shafer]

Oh yeah, I played around with VMWare VI3 aka Virtual Center aka VMWare High Availability... or VMWare Clustering.....

All in all, for the price, I would DEFINENTLY prefer DRBD and Heartbeat with Ubuntu then VMWare and Windows. DRBD I feel has a MUCH better failover and its free, compared to the VMWare solution which will cost yeah around 10 Grand with software and hardware, and it needs a SAN.

Any IT guys out there who feels like turning a Windows Box into a SAN, checkout SANMelody, which can act as FibreChannel or iSCSI.... It was either that or Open Filer, or go out and buy a SAN.

[enamelrod]

i did this a year ago. and everyone kinda blew me off. issues i ran into was speed. keeping the data base synced. maybe it was the apteryx that was killing bandwidth not sure but. it worked. jordan didnt seem to have any interest at the time. good job justin

[Justin Shafer]

I saw your thread over it just now on the old forum. Sounds cool. Were you using an actual mysq cluster software? Sounds so. Maybe what you were experiencing was a bug or something. I wonder if it would work better. The only thing I dont like about samba and I believe windows does this as well with their cluster is that the locking table for file sharing is not carried over to the other node, so though clients have a true transparent failover with mysql, they still may have close and open programs that require a lock. IE, if you had dentrix, you would have to close it completly out and then open it again.

But I did notice that with the Distibuted File System running Domain mode, dentrix didnt have to reconnect, the locks were held on both servers.

And with vmware, a host gets completly rebooted so failover isnt very fast...

I want to test microsoft cluster next, but I havent... But so far, DRBD and MySQL and Samba, etc, may work well. I just havent tested it like hell. Its all about how to configure the split brain, which I was completly new too. DRBD 8x handles this differently in older versions.. and I think after doing some reading, that its better.

If server 1 and server 2 were in a node, and server1 was primary (running mysql and samba) and it failed, server 2 would take over and become primary. And when server1 was repaired, and came back online, it will sync its data with server2, but will still remain primary.

Ahh you know.. Read this and tell me what you think... I want someone to double check my logic...

I went with this. But its for an automatic recovery. Some people prefer to have a manual recovery.

What is a split brain with drbd?
DRBD detects split brain at the time connectivity becomes available again and the peer nodes exchange the initial DRBD protocol handshake. If DRBD detects that both nodes are (or were at some point, while disconnected) in the primary role, it immediately tears down the replication connection. The tell-tale sign of this is a message like the following appearing in the system log:

http://www.drbd.org/users-guide/s-confi ... avior.html
Automatic split brain recovery policies
In order to be able to enable and configure DRBD's automatic split brain recovery policies, you must understand that DRBD offers several configuration options for this purpose. DRBD applies its split brain recovery procedures based on the number of nodes in the Primary role at the time the split brain is detected. To that end, DRBD examines the following keywords, all found in the resource's net configuration section:

•after-sb-0pri. Split brain has just been detected, but at this time the resource is not in the Primary role on any host. For this option, DRBD understands the following keywords:
◦disconnect. Do not recover automatically, simply invoke the split-brain handler script (if configured), drop the connection and continue in disconnected mode.
◦discard-younger-primary. Discard and roll back the modifications made on the host which assumed the Primary role last.
◦discard-least-changes. Discard and roll back the modifications on the host where fewer changes occurred.
◦discard-zero-changes. If there is any host on which no changes occurred at all, simply apply all modifications made on the other and continue.

•after-sb-1pri. Split brain has just been detected, and at this time the resource is in the Primary role on one host. For this option, DRBD understands the following keywords:
◦disconnect. As with after-sb-0pri, simply invoke the split-brain handler script (if configured), drop the connection and continue in disconnected mode.
◦consensus. Apply the same recovery policies as specified in after-sb-0pri. If a split brain victim can be selected after applying these policies, automatically resolve. Otherwise, behave exactly as if disconnect were specified.
◦call-pri-lost-after-sb. Apply the recovery policies as specified in after-sb-0pri. If a split brain victim can be selected after applying these policies, invoke the pri-lost-after-sb handler on the victim node. This handler must be configured in the handlers section and is expected to forcibly remove the node from the cluster.
◦discard-secondary. Whichever host is currently in the Secondary role, make that host the split brain victim.

•after-sb-2pri. Split brain has just been detected, and at this time the resource is in the Primary role on both hosts. This option accepts the same keywords as after-sb-1pri except, of course, discard-secondary

This is what I have in my config file as directed from that link. And I had to make sure that this is truly what I want to be as there are many options, and this seems to do what I would want. But its a lot to wrap my mind around because I am not used to thinking this way.... =)

after-sb-0pri discard-zero-changes
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;

[enamelrod]

our issue was during upgrades we had to manually alter the tables to make it work

[abelaguilar]

any updates to this thread?

Is DRBD8 with Heartbeat2 in Ubuntu doable? or are there many hurdles?

I've dabled in linux for a while but don't want to go down this road if ya'll of much higher knowledge suggest not worth the trouble.

I really really need a cluster scenario to avoid any down time.

Put it this way, sunday nights I wonder if i'm going to walk in to a server that is non functional becuase of a bad component.

[Justin Shafer]

Update to thread. Mysql with Corosync\OpenAIS (what used to be called heatbeat) with DRBD... works very very well. MySQL could be used Active Passive with it.. Or Active\Active but in that configuration I doubt it would need DRBD except you do for storing files like x-rays, etc, etc.... All that really depends on what x-ray software you have... As each one uses a different database. But if you have Apteryx, VixWin, Dexis, XDR, etc, etc.. Then your database is just a file stored on a file share and you would need Samba to emulate a Microsoft File Server. All of this stuff works well.. EXCEPT.. with the workstations. At the moment, you will still have to reboot the workstations or simply reconnect to a mapped drive, etc. MySQL will still be there and Open Dental wont lose its connection at all during failover (when a server dies and the other server takes its place).. But for the files stored on a file share, you will have to reinitate its connection.

All this will change with Samba and SMB2 protocol like it has with Windows.. Since MS owns the rights to SMB2 (unlike SMB1)... But SMB2 has support for duable file locks which means failover will be much better.

All of the stuff I mentioned above has to do with making a "service" highly available (MySql, Samba, etc)

The other option is to make a Virtual Machine highly available with DRBD, CoroSync\OpenAIS, and Xen.. either KVM or Xen. Havent tested this yet... This is next.

That is the linux world.

Microsoft world works pretty well. I have tested Failover Server 2008 in Windows 2008 Enterprise and everything works pretty well. Except for MySQL running in Active\Passive with Windows. The pause in open dental during failover is longer then it is with Linux with MySQL running in Active\Passive. At least it was for Steel-Eye. And like Linux you could always have Hyper-V in a cluster so your Virtual Server is protected by two physical servers. But if the AC goes out on a server, then the other server will have to boot the virtual server, and that means all the clients have to reboot.
-----------------------------------------------------------

Clusters are cool and work well, but there CAN be some IT involved. Its called split-brain recovery. Most of the time and in most examples recovery is automatic. But lets say both servers go down. And you brought up one server and wrote data to it and shut it down. Then you turned on the other server and wrote data to it and shut it down. Then you powered them both on. Now they both have different data, and someone must decide who should become the server that gets to keep the data, and who should be the server to loose its data and be overwritten. Now lets say both servers lose contact with eachother on their private networks but CAN talk to clients and eachother on a public network.. The cluster software should see take one node down and keep the other one running, since both servers can no longer write data simoultaneously to their partitions.


In windows its pretty easy to do this.. In linux, get ready to start whirling away at the keyboard.
For instance:


If a server has some data on it that is different then another server and you know that this server actually has data that needs to be overwritten you would type these commands to tell it to accept a connection and allow another server to overwrite its data:

service corosync stop
service drbd start
drbdadm disconnect all
drbdadm secondary all
drbdadm -- --discard-my-data connect all
service drbd stop
service corosync start

then on the newer server that is running you type this to tell it to overwrite that other computer:
service drbd start

then you can watch the sync process with
watch cat /proc/drbd

Once its done they should both be primary\primary.. if not
drbdadm primary all on the node that is secondary, or just do service corosync restart or service openais restart if your using SLES11.

Like I said its ready, but you dont want to do it.. Unless your Dr. Starck.. because he knows linux really well. Not saying you guys cant.. But do you. And not all IT guys are interested in going the linux route.. And less will know a linux cluster. Lots of pre-requisites involved. Windows clusters are much simpler to manage but MUCH more expensive.

If there were 2 dell servers with Dell Remote Access Cards it would be easy for a guy like me to manage the cluster remotely.. As long as I dont go on vacation? I guess thats how life is anyways with everyone running a single server but with a cluster you would really need me or someone knowledgable in linux clusters... for IT Support... which makes me think about going blind, car wreck, etc..

Will I be like job when a storm comes? I would like to think so but I doubt it.

Enjoy the day!

[Justin Shafer]

any updates to this thread?

Is DRBD8 with Heartbeat2 in Ubuntu doable? or are there many hurdles?

I've dabled in linux for a while but don't want to go down this road if ya'll of much higher knowledge suggest not worth the trouble.

I really really need a cluster scenario to avoid any down time.

Put it this way, sunday nights I wonder if i'm going to walk in to a server that is non functional becuase of a bad component.

Something else with clusters.. they cant protect against bad ram. Lets say your server has ECC Ram but you still have a problem with hardware even the ram. There is still a chance that the ones and zeros wont be right and bam, you now have a corrupted database. My-SQL would be better in this regard because the sql statements have to make some since, but it could still end up messing with the mysql service and messing with the physical underlying data in the mysql database and opendental database. This is where having good hardware and a good ups with write caching set too disabled on hard drives helps. If a server detects problems with its ram and shuts itself down for the cluster that would be a good thing. Havent gotten that far yet..... That doesnt really need to be in the cluster.. just some software that says shutdown in the event of any failure that could take place. Like a cpu fan going bad and causing the system to get to hot. In a cluster, that would be just fine.. please by all means.. shut down.
Thats what backups are for. =)

[jordansparks]

Our table lengths are probably getting too long for clustering. In other words, I don't think it's possible and we push replication as the solution for heavy users.

[enamelrod]

it took 3 years but finally We all agree the table sizes are to big for clustering. I thought I had sent you the link to my post in 2007 justin. would have saved you some headache.

[Justin Shafer]

Uhh I am not using MySQL Cluster!!!!! I am using the standard MySQL and having the block device (hard drive) being synced to more then one computer. All I have to do is make sure mysql is listening on a virtual ip address provided by Heartbeat. Table length problem is not a problem with this setup.

DRBD was accepted into the Linux Kernel and is now apart of Suse Enterprise Linux. I would not use the GFS (RedHar Cluster Filesystem) or OCFS2 (Oracle) FileSystems and the Dual Primary Option.. That is where both computers have the filesystem mounted at the sametime.... Though I have.. I remember it only working on Suse back then. Otherwise its in Primary/Secondary. And a partition like a journaled ext3.

Just like in the world of Windows.
http://en.wikipedia.org/wiki/Cluster_Shared_Volumes

Its really not that hard to setup and play with... The commands are similar to the mdadm commands in linux.

[Justin Shafer]

Yay! Samba4 looks like it will support Transparent Client Failover with SMB 2.2 clients... Of course that will only be Windows 8 PC's..... Other then that, I think I will hate Windows 8 and the lack of a start button without third party software.. I threw it on a pc, puked, and remove it. I think it would be better with an Apple Trackpad.. But that is not cool if you occasionally play counter strike.


Before the client would have failed copying the file. He would have to wait for the share to become available or reconnect his mapped drive, etc. Prior, CTDB and Samba did not share file locks with all nodes.. so if it had to failover, the share would be there, but the locking table would start over. Causing clients to simply reconnect. I have been waiting and waiting for this!

http://www.samba.org/samba/news/develop ... event.html

Persistent Handles – Test Suites

My focus for the event was on durable (SMB2.0) and persistent (SMB2.2) file handles and clustering features. I gained a better understanding of persistent file handles and I was able to extend our testsuites and add precision to them. I also spotted some bugs in the documentation and a bug in Windows 8 durable-handle-vs-oplock behaviour (a regression from Windows 2008R2). I am still working on extending our testsuites with respect to durable and persistent handles.

It was extremely useful to have several of the core engineers from Microsoft available during the test lab for discussions about product behaviour and the documentation. It helped me to improve my understanding of the new clustering concepts and to deepen my knowledge of durable and persistent file handles.

Persistent Handles – Server Hacking

On the server side, Christian set up a Samba-CTDB cluster in the partner environment and we installed the code from the durable handle work-in-progress-branch that Metze and I are working on, and we started hacking some 2.2 features into it. In the end, we were able to talk SMB 2.2 with the client and offered persistent handles – in a somewhat faked up manner, not being able to give the full set of guarantees attached to persistent handles.

The nice visible success was that we were able to do a transparent node failover of the client copying a dvd image to the samba cluster. We ran “ctdb disable” on the connected node and the client switched over to the other node and kept copying. In the end, the dvd image was complete and its md5sum correct!

-------------------------------------------------------
I have talked to the Samba Team before...


-----Original Message-----
From: Volker Lendecke [mailto:Volker.Lendecke@xxxxxx.DE]
Sent: Friday, July 02, 2010 4:04 AM
To: Justin Shafer
Cc: VL@xxxxxx.DE
Subject: Re: Durable File Lock and Samba

On Fri, Jul 02, 2010 at 02:35:50AM -0500, Justin Shafer wrote:
> Does the SMB_PRIVATE_DIR have to exist on the clustered file system?
> For the life of me I never see any files in there except once on
> ubuntu and I could get ping ponging with gfs2. I gave up on gfs2.

No, ctdb should take care of the secrets.tdb files and other files there.

> The only other question is.. I have Samba 3.5.4 with SMB2 enabled but
> I don't think durable locks are working.. But I am unsure because of
> the whole private directory and samba.

What do you mean with durable locks?

Volker

[Manny Ramirez]

Microsoft Hyper V 2008 R2 includes live migration, fail-over clustering and shared cluster volume for free. You can download a copy of Hyper-V hypervisor from the Microsoft web site and install it bear metal if you prefer. Of course, you must have the right hardware and network infrastructure to support virtualization . You also need the knowledge and patient to configure your infrastructure. Like everything Microsoft, you must try a few times until you tweak your configuration. Microsoft Fail-Over clustering has the tools necessary to validate Cluster Shared Volumes and give you hinters of where or what needs to be configured in order to create highly available virtual machines. It is more practical to provide redundancy at the VM level rather than the application level. If the VM can move to any other server in the farm with all its configuration, applications and data with almost no downtime then little needs to be configured at the application level to insure high availability. Bear in mind that you can still configure redundancy at the application level. VMs don't care or know they are VMs but that would be double the work.

For what I understand in this post, Justin was trying to demonstrate how to continue running OD even when a server goes down. Meaning High Availability and Redundancy. In this configuration, he has a single machine running Windows more then likely ? with VMware Workstation hosting three guest virtual machines . Two VMS running Linux (used to replicate/clustering a single database), and a XP workstation. In this scenario, If one virtual server goes down, the other one takes over. If the failed server comes back up then the original server hosting the application will be the main server. I admire the knowledge and effort put into this configuration and it is very clear that he has a great handle in Ubuntu Linux and MYSQL . Kudos to Justin. That being said, I still believe that Microsoft is the way to go for most dental practices. Additionally, this solution does not address High Availability properly. I must add that I am not a big Microsoft fan (I currently own two MacBook Pros and an Ubuntu server at my house and stream movies) but in addition to running OD and MYSQL, most offices still need servers to host other application such Quickbooks, Imaging and other Windows based applications. Moreover, tinkering and talking about Samba shares, Linux Daemons and other Linux intricacies are not something that most dental practices are ready to talk about. Lastly, I am not a big fan of mixing environment in production scenarios, mostly to reduce platform incompatibilities and support issues.

Another point that the original post forgot to address was that if the main server (the one hosting the virtual machines) crashes, everything goes down, meaning; we are right back to where we started. The ideal scenario must address that the virtual machines detect the failure and automatically moves to any available server in the cluster or farm with zero or minimum downtime in order to continue to serve applications. So it is clear that more than a single server must be available to continue with the endeavor.

The good news is that virtualization is getting better and cheaper everyday. I am going to share with you guys what I think is an optional hardware configuration needed to build the network infrastructure. I clearly understand that this is probably a better solutions for a multi-site group or a large office with several providers. Bear in mind that although Hyper-V is free, you still need to properly license the virtual machines with the OS installed within the VM. Just like you would with VMware.

Server 1 / Domain Controller
Server 2 / Cluster Server 1 Hyper-V bare metal
Server 3 / Cluster Server 2 Hyper-V bare metal
QNAP NAS TS-559 Pro 2 configured with either a level RAID 5 or 6 disk array. The enclosure can house up to 5 Sata hard drives.

PD: I recently put together an infrastructure like the one above. Hardware was little over $5000.00

Server 1 doesn't have to be a super server. Mainly used as a DC and possibly a CA Enterprise if you want to implement an Enterprise PKI infrastructure.
Server 2 and 3: What is most important here is processing power and memory. Data and VMs will not reside on these servers as they will be used mainly as hosts in the "fail-over farm". Dual Quad Core processor, 16 to 24 GB of memory and a RAID 5 configuration of 146 GB to run the Hypervisor. Two or more NICs and two power supplies would be nice.

Qnap has the necessary software to run the ISCSI protocol. ISCSI is hot topic these days, and it slowly replacing expensive Fiber Channel fabrics. Qnap supports something called SPC-3 persistent reservation for Windows 2008 Failover Cluster and Cluster Shared Volume.. This device in my opinion rocks. You configured it in any shape or form and LUNs can be carved in the manner you think is most appropriate for your organization.

Once your hardware is in place, you are ready to configure your infrastructure or what Microsoft call "Private Cloud". You can start creating as many VMs as you deem necessary for your organization ,and of course, your hardware can handle. Again, licensing here, applies the same way as if were deploying a real server. For those hard core Linux fans. Yes, it does support some Linux Distrox
There are many good article about Share Cluster Volume and Hyper-V in Google so I won't explain how to configure it here. Once you create your highly available VMs you will install OD and all your other applications as usual and let the system do the rest.

Sharing is caring