OpenStack High Availability – Controller Stack

By | April 25, 2014

High availability is a must have for enterprise deployment of OpenStack. Not high availability of individual virtual machines (instances), nor high availability of individual compute nodes (hypervisor hosts). I mean high availability of the OpenStack controller stack. True, if the stack falls over, that doesn’t necessarily mean that your instances stop working (especially if you’ve carefully placed your networking and storage components on the compute nodes), but we should build the stack so it doesn’t fall over. That means we need to deploy the controller stack with no single points of failure. Two of everything.

When I speak of the controller stack, I’m lumping together everything that doesn’t run on compute nodes. This includes the database, message queue service, web portal, API services and object storage. We’ll need to build at least two of each of these.

For each component, we’ll need to determine whether the component is stateless or stateful. Stateless components are entirely transactional, and don’t store any data. Therefore, there’s no need to cluster or replicate anything between the two instances of these components. The OpenStack API services are stateless for example. Stateful services store data or other state information, and so will require some form of clustering or replication, so that the data will be in sync between the two instances.

Next, we need to determine whether each component will run in an active-active or active-passive configuration. An Active-active configuration provides scalability as well as high availability, so we’ll deploy components in an active-active configuration wherever possible. This will help determine the techniques we use to provide high availability. For example, we could build a MySQL cluster using pacemaker, but that would be active-passive. Instead, we’ll build an active-active MySQL service using Galera.

The Stack

openstack ha controller
The diagram shows a total of ten hosts where all of the components are deployed. You could consolidate these onto fewer hosts, but bear in mind that all of these can be virtual machines, so in fact we can deploy these on just two physical servers. To achieve true high availability, we’ll want to place one of each component on physical host A and the other on physical host B, so that if a physical server fails, you still have one surviving instance of every component on the remaining server.

At the top of the stack is a pair of HAProxy nodes. This is our load balancer layer. A virtual IP address (VIP) is defined that we’ll use as our endpoint address for all of the services of the stack. This VIP will be active on only one HAProxy node at a time. That means that HAProxy will be running active-passive mode, however, HAProxy performance is very good, so this won’t create any sort of bottleneck. The keepalived service is used to move the VIP if the active node fails.

Next we’ve got our MySQL database servers. As I mentioned before, we’ll use Galera to do bidirectional replication, which will enable us to use either node for database access. We’ll use the load balancers to distribute load across the nodes.

The OpenStack API services are mostly stateless, so we can simply load balance these. No replication or clustering tools are required. RabbitMQ has native clustering capability, and we’ll enable that. The OpenStack services that use RabbitMQ can be configured to point to multiple RabbitMQ instances. The glance image service will require shared storage so that both instances can access the stored images. We’ll use our highly available swift object store for glance image storage.

The last component is the swift object storage service. The swift storage nodes are designed to replicate using rsync, so high availability is baked in to that component. The swift proxy is stateless, so we can load balance that piece.

Building the Stack

How do we build this? There are a lot of steps involved, but each piece is relatively simple. The hard part is getting your head around the whole design, and hopefully this article will be of help. Once you see what you’re trying to build, you can tackle each piece, one at a time. The major steps are:

Once we’re done, and added a bunch more compute nodes, we should end up with an architecture that looks like this:

OpenStack Network

I don’t really get into the router in this series. The assumption is that you have two routed subnets to work with. If you’re trying to build this at home or in a minimal test lab, you can build yourself a virtual router. You can learn how I’ve done this in my test lab.

For those of you who read my older article about using pacemaker clustering, I’ll say that I did successfully build the stack as an active-passive configuration using pacemaker, but I found that it was overly complicated and prone to failure. The active-active solution is, in my opinion, much simpler, more resilient, more scalable and therefore more suitable for the enterprise.

 

 

20 thoughts on “OpenStack High Availability – Controller Stack

  1. mariusv

    2 zone 2 replica is gonna be sketchy as hell cluster it will lose data
    at some point and swift isnt well tested with less than 3. I would
    suggest to improve the swift cluster but the rest looks good :-

    Reply
    1. Brian Seltzer

      I’ve updated the Swift article to include three nodes. Makes sense, a cluster should always have a tie breaker…

      Reply
  2. Upender

    Hi Brian,

    I have small infrastructure and i want to build the controller stack in two boxes. Please suggest me a solution.

    Reply
    1. Brian Seltzer Post author

      Upender, You can install KVM on both boxes. On one, create the VMs you need to run the controller stack. Use the other to be your single compute node. This is essentially how I built my test lab. My controller stack was run on a VMware host, but could just as easily run on KVM or whatever.

      Reply
      1. Upender

        My office infrastructure is small. I want to build one active, one passive controllers with 3 nodes attached. If active controller goes down, passive should serve the requests.

        Reply
  3. Charles

    Hi Brian,

    these articles are very helpful for understand the openstack HA solution, but I can not find any words about neutron component HA, can you tell something about this? thanks!

    Reply
    1. Brian Seltzer Post author

      There hasn’t been a good way to establish HA for Neutron as of the Icehouse release, so I’ve used Nova networking in my articles. Hopefully this situation will change with later releases.

      Reply
      1. Charles

        Thanks for your reply. another question, does this solution can provide the active-active architecture? can I add more nodes for each service (e.g.Proxies, Controller, MySQL) to improve those service performance?

        Reply
        1. Brian Seltzer Post author

          Everything in this solution is active-active except for the ha-proxy piece. Only one ha-proxy node can host the shared IP address at a time. To scale out the ha-proxy layer, you can split the OpenStack services across multiple ha-proxy pairs, or add separate ip addresses that can float across more nodes. As for adding additional nodes, yes you can add additional nodes for all of the OpenStack services. The only things you should research further are the mysql (mariadb), galera, and rabbitMQ pieces, which may have specific things to understand when adding more nodes.

          Reply
  4. James

    Thanks Brian for this detailed walk trough 🙂

    This was a great help to building our openstack cloud,

    I was just wondering have you any thoughts on doing a couple of Tuitorials on Neutron server and how to make it work HA with Pacemaker?

    Reply
  5. Mike

    This articles is very nice. I just need to make plan for this, I have 3 servers so I’m thinking that can I create 3 servers including ever services in each servers (i.e. HAProxies, Controller, MySQL, Swift and Compute). This will keep alive without overloading the hardware, I guess.
    Or should I use 2 controllers and 2 compute. So I will put HAProxies, database in Controller and swift, Glance and others in Compute. Please advise me which one will be the best. I have 3 servers as I told you above and I hope I can arrange one more (if needed).
    Also I saw JUJU demo and it’s seem great. Do you have any articles for this, if yes then please share it. I would like to follow it for sure.

    Reply
    1. Brian Seltzer Post author

      Three machines isn’t really enough to build this HA configuration, unless you want to put a hypervisor on one or two of the machines and build VMs for the controllers, swift, ceph, etc. You need a minimum of three servers to build a viable swift cluster, same with ceph. If you want HA for your controllers, then you need to deploy your controller services onto two of the three machines, run compute on the third, and deploy swift and ceph across all three.

      Sorry I don’t have any articles about juju.

      Reply
  6. Mike

    My company use 4 different IP Ranges for managing Network Tag in Servers. For Example:
    1. Management: 172.20.1.0/24
    2. IT/media: 10.19.31.0/24
    3. Account:10.10.1.0/24
    4. DMZ: 192.168.20.0/24
    So my company want that Open Stack VM guest should get an IP address depending upon these 4 IP ranges. So for working with this plan I have to create 4 different range like 172.20.1.200 – 172.20.1.253, 10.19.31.200 – 10.19.31.253 etc. and able to assign different group of IP to different VM guests. How can I do this work?
    Regards,
    MIKE

    Reply
    1. Brian Seltzer Post author

      So if you need to have multiple external networks, you can either add more physical NICs or use VLANs to bring more external networks into your nodes. See my article about using VLANs: http://behindtheracks.com/2013/05/openstack-i-need-how-many-nics/ – where to put the NICs or VLANs will depend on whether you are using flat networking (in which case these need to go into the compute nodes), or if you are using Neutron (in which case you would put them in the network nodes). In this case, I’d probably go with a Neutron design, since it would be a lot easier to bring the networks into a smaller group of network nodes than bringing them into a large number of compute nodes.

      Reply
  7. Janki Chhatbar

    Hi Brain

    very good articles. 1 question.
    I have 3 node Icehouse architecture with Neutron installed on bare machine. I want to have HA for it.
    Can I have HAProxy, mysql, Controller stack all in 1 server? Instead of having HAProxy, mysql and controller stack on 3 different servers, can I combine them into 1? I dont want to have swift. Would 3 servers suffice then? How to make Neutron HA?

    Reply
    1. Brian Seltzer Post author

      Sure, you can run ha-proxy on the same servers as mysql and the controller services. You could also run neutron on one of the controllers, and use your third server as a compute node, but I haven’t found a way to make neutron HA using ha-proxy. You could do it using pacemaker and corosync, but that’s a complicated mess in my opinion. I’m hoping that future versions of OpenStack will address that.

      Reply
      1. Janki

        Hi

        I am installing HAproxy and MySQL-galera on same nodes. I have created cluster.cnf as described in this series. Reloading HAProxy after configuring it to listen to mysql cluster doesnot work. It says “Starting proxy openstack: cannot bind socket”. haproxy.cnf is

        listen openstack
        bind :3306
        balance source
        option httpchk
        server controller1 ip1:3306 check port 9200 inter 2000 rise 2 fall 5
        server controller2 ip2:3306 check port 9200 inter 2000 rise 2 fall 5

        Installed MySQL-Galera from its binary from this link http://galeracluster.com/documentation-webpages/galerainstallation.html#galera-cluster-for-mysql. I havent installed mysql-client-5.6 because doing so removes mysql-wsrep-5.6 and hence using “option httpchk” in haproxy.cnf.

        Scenario: 2 nodes with HAProxy and MYSQL-Galera. 1 VIP. OS=Ubuntu 14.04 (64 bit)

        P. S. Doing everything with root.

        Reply
        1. Brian Seltzer Post author

          You can’t have ha-proxy and mysql listening on the same ports unless they are on different servers. So, you’ll have to change the port that mysql is using and let ha-proxy use 3306.

          Reply
  8. Dung

    Hi Brain
    I have build HA Openstack follow this article. when I stop service database on one node controller i have error when create new a instance. I must reboot service nova-compute on node compute then create successful.
    how to fix it please.
    Thanhs so much!

    Reply

Leave a Reply