Proxmox VM and CEPH Storage

For complex topics that regular users would not be interested in. For power users and database administrators.
Post Reply
User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Proxmox VM and CEPH Storage

Post by drtech » Tue Jul 29, 2014 8:09 am

Anyone used Proxmox to host VM's and/or used CEPH storage? I am currently exploring both to use for a High Availability setup, but wondered if anyone has experience with any pointers. My main objective is to have an HA system that will stay up even when a server (node) crashes due to the expected/unexpected (they all do eventually) HD crash, etc.

I have used Proxmox for testing a few years ago and it seemed very slick and includes live migration of VMs.

CEPH i have no experience with, but seems like the future of storage.
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Re: Proxmox VM and CEPH Storage

Post by drtech » Thu Jul 31, 2014 7:41 pm

I will take that as a no as far as experience from anyone :p

I am setting up my CEPH cluster now...
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

epicurean
Posts: 51
Joined: Tue Jun 07, 2011 8:18 pm

Re: Proxmox VM and CEPH Storage

Post by epicurean » Sun Oct 12, 2014 5:00 pm

HI David,

Please share your experiences. I am leaning towards a Esxi setup , but undecided on what is the best storage form. On bare metal, xpenology has worked reasonably well.

Everyone seems to rave about ZFS , but I am not entirely convinced.

User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Re: Proxmox VM and CEPH Storage

Post by drtech » Thu Oct 16, 2014 8:12 am

Learning curve for sure, but I it all set up and have been testing it for many weeks now with good results. I have Proxmox VM setup with CEPH as the storage on three separate machines. It is slower than local storage for the mysql VM running long queries (8-10 seconds instead of 4-5 seconds running an 8 page aging report...this is because the Hard Drive for the VM machine is on another machine with shared CEPH storage) but for day to day use, don't see any performance decrease from a local storage setup. The advantage in theory, is that when one machine crashes or hard drives go down, all the data is still accessible and you keep on working in your practice. Fantastic if it works in a live environment...
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

epicurean
Posts: 51
Joined: Tue Jun 07, 2011 8:18 pm

Re: Proxmox VM and CEPH Storage

Post by epicurean » Fri Oct 31, 2014 11:07 pm

Dear David,
That sounds really cool. May I know why you chose CEPH as your storage with so many choices available ( eg Nexenta, FreeNAS, SANS)
is the CEPH acting as your file server for your radiographic images and OpenDentImages as well?

User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Re: Proxmox VM and CEPH Storage

Post by drtech » Sun Nov 09, 2014 6:58 pm

Yes, it hosts everything indirectly. Ceph is not a direct file store by itself. It is just a storage platform for a virtual machine disk. I have multiple VMs hosting the SAMBA file store for my images, etc. The only reason I chose it over the other network attached storage solutions is the promise of better disaster live recovery. One disk goes out, even multiple disks, you still have your live virtual machines running and served by the same data with no downtime. Is this different than a RAID setup? Yes. In my experience RAID has always been a problem when a disk goes out. Either the array fails or fails on rebuild. CEPH is different, it is not striping info, it is storing it all in clusters. At least the little I know about it, it should recover easier. The other great thing is to add capacity, you just add a disk, and include it as an object for data storage. No RAID rebuilding, etc. The other great thing is that is is spread out over multiple machines, that way when one entire computer fails, the data is still there being served.

The real test will come when it actually does go out and we see how it performs. Always backing up to other places of course. No redundant system replaces separate backups in case of real disaster and system failure.
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

epicurean
Posts: 51
Joined: Tue Jun 07, 2011 8:18 pm

Re: Proxmox VM and CEPH Storage

Post by epicurean » Thu Dec 11, 2014 4:52 am

Hi David,
Would you be able to detail (maybe via PM) how to setup up CEPH as you did ? It sounds exactly like what I am trying to achieve with an esxi virtualisation , file serving and storage functions for OD and my digital radiographic images

User avatar
Justin Shafer
Posts: 596
Joined: Sat Jul 28, 2007 7:34 pm
Location: Fort Worth, TX.

Re: Proxmox VM and CEPH Storage

Post by Justin Shafer » Fri Dec 19, 2014 1:04 am

I used ProxMox once about.. 5 years ago...

Any improvement? (I bet so!)

I stuck with DRBD and Heartbeat\Corosync back then.. Ran it at home with mysql and OD.

Failover was bad ass... for mysql.

epicurean
Posts: 51
Joined: Tue Jun 07, 2011 8:18 pm

Re: Proxmox VM and CEPH Storage

Post by epicurean » Sat Dec 20, 2014 12:15 am

Justin,
What was your experience with the failover?

User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Re: Proxmox VM and CEPH Storage

Post by drtech » Mon Jan 26, 2015 10:10 am

I have been running on the proxmox/ceph config for a month or so now. Everything is great so far!

I have it setup with three proxmox VM servers and three proxmox/Ceph servers. So as long as one computer in each of those two clusters is online, everything hums along.

I have shutdown two of the three ceph servers and everything still hummed along and then was right back to normal when I brought them back up. The only noticeable issue is a delay of 10-15 seconds (everything hangs) when it goes down and then periodically when I assume it is trying to reconnect, but everything is still functional and online!

The VM's themselves restart on a failover node in their cluster if my primary VM server fails.

Everything is fenced with APC PDU's I bought on ebay for like $50 each when they list for $700 new....they work flawlessly. (One for each cluster)
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

epicurean
Posts: 51
Joined: Tue Jun 07, 2011 8:18 pm

Re: Proxmox VM and CEPH Storage

Post by epicurean » Thu Jan 29, 2015 6:54 pm

Hi DAvid,

I am thinking of using VMware esxi for virtualisation instead, and a ZFS file system for the storage. Simply because i am a little more familiar with them.
Do u have a domain controller for your network, or a firewall VM eg Pfsense or Sophos?

User avatar
drtech
Posts: 1647
Joined: Wed Jun 20, 2007 8:44 am
Location: Springfield, MO
Contact:

Re: Proxmox VM and CEPH Storage

Post by drtech » Fri Jan 30, 2015 8:00 am

I do not have a separate VM for domain control or firewall. My firewall is router based to the outside world. Do you think it is necessary to have something else? If so, what is it's role/main function?

I have each VM serving data handling security with encryption of the filesystem and then user level Linux access control, which seems to be working well.
David Fuchs
Dentist - Springfield, MO
Smile Dental http://www.887-smile.com

Post Reply