Cloud Based vs Daisy Chain multiple offices single database

For users or potential users.
Post Reply
aboval
Posts: 8
Joined: Sat Mar 26, 2011 10:12 am

Cloud Based vs Daisy Chain multiple offices single database

Post by aboval » Mon Feb 05, 2018 11:11 am

Currently have 6 offices, 6 individual databases. Looking to go with a single database. Have tested the clinics functionality in a test environment and love it. Challenge now is getting the single database architecture enterprise ready. My first test was with AWS and the setup is fairly simple, but the latency is a killer, especially when going to the treatment plan tab for a patient with multiple treatment plans. I tried all combinations of AWS setup using faster bandwidth, dedicated bandwidth, and faster processor and ram speeds. Latency still makes it unusable in production. I'm looking to set up a middle tier to see if that helps, but rather than blaze a new trail with all this, I'd love to follow someone elses lead.

Has anyone had success setting up a cloud based RMS hosting Open Dental for multiple locations using a single database?

Has anyone had success setting up daisy chain replication that they feel is "enterprise ready"?

Our organization needs to be able to bring new (conversion and/or de novo) offices on board without having to reconfigure the infrastructure every time. Also need something that is capable of running off 4G backup (at least semi-satisfactorily) in case of internet down.

Xrays are not a concern - just pay for Apteryx web and they host everything xray related.

Thanks,
Alan Boval

User avatar
Justin Shafer
Posts: 596
Joined: Sat Jul 28, 2007 7:34 pm
Location: Fort Worth, TX.

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by Justin Shafer » Mon Feb 05, 2018 6:22 pm

I know someone with 4 locations and they have been having me look at this. Problem is 1 location has iffy internet.. 2 locations have XDR, 1 Dexis, the other Sidexis.

Anyways... I would use the Daisy Chain if I could get away with it, just because: If the internet goes down at a location, you can still use Open Dental at the location without internet.

aboval
Posts: 8
Joined: Sat Mar 26, 2011 10:12 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by aboval » Mon Feb 05, 2018 7:13 pm

I thought about daisy chain, but we're growing (about to add location #7) and I'm concerned about daisy chain replication at say 20-30 offices. Do you know anyone doing this successfully at scale? It's also a bit of a pain to add new locations with daisy chain, whereas cloud hosted RDB it's as easy as adding a new pc to an existing location....

Assume internet speed are at least decent, cloud just isn't in the realm of reality for those who aren't willing to pay for good internet. Also, for the internet down situation, I think the standard these days is to have a 4G cell phone as backup internet. It's not going to be great, but it'll get you though the few hours the internet may be down.

The multiple imaging softwares is also an issue, but I think you just recommend they go with Apteryx VX Web. If internet goes down, VX web will allow you to continue taking new xrays and hold them in a queue that uploads when the internet comes back up. You can't view previous xrays, but in reality if you can take new ones it's a minor pain, not a showstopper.

Back to daisy chain: do you think it's a realistic solution for 20-30 offices without running into corruption issues several times a year? Is anybody actually doing this? It would be great to hear from someone who has successfully set up 10+ offices on single database

User avatar
Justin Shafer
Posts: 596
Joined: Sat Jul 28, 2007 7:34 pm
Location: Fort Worth, TX.

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by Justin Shafer » Tue Feb 06, 2018 4:30 am

I personally don't know anyone, although I read Open Dental does have someone with 50+ offices using it. I did setup XVWeb for 1 guy who goes to old folks homes, but that is about it for XV Web.

The only other option is 4G LTE Failover router if your internet goes down, like you said. If you go this route I would look at a company called CradlePoint, they have been doing this for awhile. Although they are a tad bit expensive. I have a friend who set one up in Lubbock, TX (only option they had was T1 internet which is SLOW) that used Cradlepoint, but they had a problem back then, mainly the AC Adapter kept going out. Which.. kinda defeats the purpose of having a really nice router that can do 4G LTE, that was around 2 years ago and hopefully they got all that straightened out. It drove him nuts.. It's about a 5 hour drive to Lubbock from where we live. :D I invited him to comment on this thread. :D

User avatar
cmcgehee
Posts: 711
Joined: Tue Aug 25, 2015 5:06 pm
Location: Salem, Oregon

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by cmcgehee » Tue Feb 06, 2018 7:39 am

I know that one of our biggest customers has set up a Galera cluster to host all their data in one database for 30+ clinics. I'm not sure if they have a server node at each clinic or they just have a few nodes spread across different regions.
Chris McGehee
Open Dental Software
http://www.opendental.com

aboval
Posts: 8
Joined: Sat Mar 26, 2011 10:12 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by aboval » Tue Feb 06, 2018 8:44 am

Using Galera Cluster is an interesting idea. Conceptually it's more of a (potentially) really nice way to do replication - a step up from daisy chain. Could also put a GC node on AWS, although you'd still have the latency issues that it seems we're all running into when cloud hosting Open Dental. I do really like the idea though:
1. Could put a node at each location. Pros - internet down issue goes away. Lowest latency of all multi clinic single database options. Cons - need a server at each location with the cost, HIPAA, etc issues that come with it.
2. Once someone figures out the latency issues with cloud hosting Open Dental, nodes could be put on the cloud and only those locations with substandard internet would need their own node.

This begs the question - who has figured out the cloud hosted Open Dental latency issue? I can't believe that no one out there has this figured out yet... Or is it that Open Dental can't share who among their clients has done it? I ran a test hosting OD on AWS and it works fine - except the latency, especially on patients with significant info like multiple treatment plans. Tested in a couple different locations with great internet speeds so I don't think that's the issue. I also tested several different variations of AWS host - upgrading RAM, CPU, and bandwidth (AWS side.) None of it seemed to make much of a difference. I even set up a middle tier hosted on AWS and connected it to the AWS hosted MySQL instance but ran into issues connecting to the middle tier from my local computer and ran out of patience. Has someone at Open Dental tested this on a database of significant size? If not, does someone over there want to help me troubleshoot so we can get the "cloud hosted Open Dental" question answered?

User avatar
cmcgehee
Posts: 711
Joined: Tue Aug 25, 2015 5:06 pm
Location: Salem, Oregon

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by cmcgehee » Tue Feb 06, 2018 11:25 am

I think the best way of doing the cloud as of right now is set up a middle tier instance on AWS that connects to a database in AWS. I know of at least one multi-location practice that has this setup, but I also know that they view Open Dental as quite slow. The way that Open Dental is written is that we make many round trips to the database for almost any user action, and we sometimes make many round trips to the middle tier server. If you can get the latency from the middle tier server to the database to be under 20 ms and the latency for a database call over middle tier from a workstation to be under 50 ms, then you wouldn't experience Open Dental as slow. You've probably been using this already, but we do have a tool to measure latency within Open Dental under Tools -> Misc Tools -> Test Latency.
Chris McGehee
Open Dental Software
http://www.opendental.com

aboval
Posts: 8
Joined: Sat Mar 26, 2011 10:12 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by aboval » Tue Feb 06, 2018 8:51 pm

Good info Chris. Latency from AWS middle tier to AWS DB wasn't an issue. I think the latency issue is in going over the wire - either direct to DB or to middletier server. If OD is written in a way that requires lots of trips even when there's a middle tier, I'm out of ideas on how to make it faster. If anyone out there has done it, please let us know how. Perhaps an option in Open Dental to "batch calls"? :)

That leaves us with replication as the only realistic option for multi office single database. The Galera Cluster idea is interesting. I'm curious how Open Dental reacts when one node goes offline (due to internet down) and then comes back online and resyncs - assume that people have been making changes to their local offline node during that time. Do you run into the potential database corruption issues that you have with Daisy Chain replication? Or what if 2 people are editing the same data at the same time - how are conflicts handled? Also, there is a limit on the number of nodes you can have without having the same latency issues with cloud hosted (although this time due to synchronous replication) If you can't have a node in every office, you're kind of stuck again - do you use VPN to form a group of offices that each tie into a node? eg if you have 20 offices, 4 nodes, each with 5 offices connected through VPN, node server in the office with the fastest network? I'm curious how the 30 office group set up their architecture and what results they're seeing with it...does anyone have any info?

Daisy chain is an easier way to replicate, but per the notes on Open Dental help, "Does not scale up well past about 10 servers due to single weak link concept." This isn't useful for groups of more than 10 offices. The database corruption issue is also a drag. Good solution for a small group, but not a mid sized.

Somewhere out there, some group must be using Open Dental multi office (10+) single database that would be willing to share how they did it?

tgriswold
Posts: 122
Joined: Fri Jun 07, 2013 8:52 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by tgriswold » Wed Feb 07, 2018 8:05 am

I'm not sure how many of our larger customers with these type of setups regularly visit these forums, I know there are a couple. The OD engineers that reply on these forums can give as much information as we have about some solutions, but we're always on second/third hand information on the details of how these setups are working (the customer's IT staff is the one setting up and maintaining these systems, not us) and even then our information is related to working around some oddities that come from Replication/Galera Cluster and their interaction with OD. We do have some testing situations set up here, but we are no where near experts on such setups from an implementation perspective. At the end of the day though our job is to work on the program to work with mysql, it is the office's responsibility to figure out how they are going to implement mysql. I will say we are always actively working to reduce major speed and concurrency issues with large customers/databases as bugs/complaints come in.

To answer your question about what happens when multiple locations try to edit the same exact thing at the same time, the best solution is to minimize this from happening. Even on a single location on a single database, your best case scenario is that someone's changes get overridden. However if you are asking about two different staff working on similar but info (like working on the same patient account but not the same info), we have made many changes in the past year or two to enhance the program to allow this type of work as much as we can. Generally as long as one staff's changes are not going to directly effect what someone else is working on at the same time both changes with go through successfully. I am personally unclear on how Galera functions when there is an internet outage.

If you are interested in some more concrete info/ideas of how some of our larger customers are using the program you are always welcome to call in to support and see if they can get you that information. Some of the managers and other high level staff work much more closely with these type of customers then the engineers do on a day to day basis, and they would be the one to ask about the possibility of getting in contact with these offices to get info firsthand (that isn't the type of information we post on the forums).
Travis Griswold
Open Dental Software
http://www.opendental.com

User avatar
cmcgehee
Posts: 711
Joined: Tue Aug 25, 2015 5:06 pm
Location: Salem, Oregon

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by cmcgehee » Wed Feb 07, 2018 8:10 am

I personally have been spending a lot of time lately reducing the number of trips over middle tier in the heavily used parts of the program (all the main modules, creating appointments, creating claims, etc.). These improvements will not be available, though, until version 18.1. Nathan, our CEO, really values that the program is fast.

When using replication, inconsistencies do cause corruption. Generally that only happens when two people from different locations are editing the same data at the same time. In practice, this happens more often with insurance plans that are used across many patients. For example, if I assign Insurance XYZ to Patient A but someone at another location has deleted Insurance XYZ one minute ago and the replication hasn't fully synced yet, in a couple minutes the patient will have a corrupt insurance plan. We do have a plethora of tools to repair the database in these sort of events.
Chris McGehee
Open Dental Software
http://www.opendental.com

User avatar
dgraffeo
Posts: 147
Joined: Wed Sep 24, 2014 3:19 pm

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by dgraffeo » Wed Feb 07, 2018 10:05 am

Concurrency issues are a never ending battle. On the one hand, we want to verify information is still present and fresh when a user is editing it, but on the other hand we don't want to make additional database calls to make sure it is in a good state when the user edits it. Most concurrency issues we take care of on a case-by-case basis depending on their severity. For each concurrency issue that we solve, it almost always means more calls to the database which will make everything slower. With more people working on a single set of data, the more likely there is to be concurrency issues. It is just the nature of software, unfortunately.
"To understand what recursion is, you must first understand recursion."

David Graffeo
Open Dental Software
http://www.opendental.com

User avatar
irfan
Posts: 216
Joined: Thu Oct 21, 2010 9:09 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by irfan » Thu Feb 22, 2018 5:06 am

We are one of the offices with slow function on AWS with a middle tier and database. There have been significant improvements over the last year but we stay on beta to keep with the newest updates and that obviously brings plenty of UEs. It’s much faster than it used to be. The Tx plan module used to take a horrendous amount of time. Like 23 seconds to change a priority. It was so bad we almost switched to both curve and also dentrix ascend, but bailed on each in the 11th hour.

. Now it’s bearable. Staff have gotten used to it and complain less so it might just be a matter of time til it’s tweaked enough. Our offices have 100/10 connections. From home with 1000/1000 connection it does feel significantly faster, but fiber is $300/mo for businesses. . Maybe when 18.1 comes out the complaints will stop , but again, it’s a LOT better than it used to be, so thanks to the developers for that.

User avatar
cmcgehee
Posts: 711
Joined: Tue Aug 25, 2015 5:06 pm
Location: Salem, Oregon

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by cmcgehee » Thu Feb 22, 2018 9:20 am

Irfan, after you update to 18.1, I would be interested in learning what are the biggest pain points in terms of slowness for your staff. I could then look into doing another round of speed improvements.
Chris McGehee
Open Dental Software
http://www.opendental.com

User avatar
irfan
Posts: 216
Joined: Thu Oct 21, 2010 9:09 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by irfan » Thu Feb 22, 2018 9:38 am

Will do, thanks

aboval
Posts: 8
Joined: Sat Mar 26, 2011 10:12 am

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by aboval » Fri Feb 23, 2018 7:17 pm

This is fantastic info - thank you guys. From the sound of it, OD is not quite ready to be cloud hosted, but is on its way there. I appreciate the brave souls paving the path on this for the rest of us - if we ever meet at a trade show or something, the first round is on me :)

Until OD is suitable for cloud hosting it appears multi master replication is the most appealing option for mid sized groups who have outgrown daisy chain and its 10 node limit. Is anyone out there using multi master (Galera cluster, clustering, active/active replication, etc) that can share insights? Do you have a node at each location? Any lag issues? Any unexpected concurrency issues?

Also, for the OD techs on this board - do you guys have a sample database with some data in it that we can use for testing? It's a ton of extra work to make sure all the security settings are in place to run a simple test, but necessary because the only database I have with significant data in it is my actual patient database....

Thanks,
Alan

User avatar
Justin Shafer
Posts: 596
Joined: Sat Jul 28, 2007 7:34 pm
Location: Fort Worth, TX.

Re: Cloud Based vs Daisy Chain multiple offices single datab

Post by Justin Shafer » Sat Mar 10, 2018 6:18 pm

This helped me quite a bit just for my own testing with a trial database and Amazon EC2 hosting MySQL and the Middle Tier with OD 17.4:

I tweaked IIS, setup a vpn vs non vpn, but this below made the biggest difference in performance for me:

Line 500 of Form OpenDental.cs
///<summary></summary>
public FormOpenDental(string[] cla){
//The default SecurityProtocol is "Ssl3|Tls". We must add Tls12 in order to support Tls1.2 web reference handshakes,
//without breaking any web references using Ssl3 or Tls.
ServicePointManager.SecurityProtocol|=SecurityProtocolType.Tls12;

Added code below:
ServicePointManager.CheckCertificateRevocationList = false;
ServicePointManager.DefaultConnectionLimit = 100000;
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;

I think setting Expect100Continue to false made the biggest difference as well as disabling the Nagle Algorithm which I read is enabled by default. The DefaultConnectionLimit is 2, and 100,000 just sounds better.

Also used this to help tweaking the mysql.cnf file
https://tools.percona.com/wizard

Post Reply