After the Storm – Bringing Up Your OpenStack

By | May 26, 2014

In our OpenStack High Availability Guide, we built quite a few servers, load balancers, database servers, controllers, compute nodes and storage nodes. Well the other day, I had a power failure and it all went down. Sorry, I don’t have UPS power in my home lab. Maybe this isn’t a likely scenario in a corporate data center, but on occasion you may be faced with a data center shutdown for power work or something like that. Anyway, I was faced with bringing the whole thing back online, and the question becomes, what components need to be brought up and in what order. This question may seem obvious to some, if you understand the dependencies between the various components, but it’s worth going over.

In the high availability guide, we configured all of the OpenStack services to use the load balancer VIP as the endpoint for each service, so first things first, let’s bring up the load balancers.

Step 1: Power on the load balancers

All of the OpenStack services rely on the Keystone service for identity and access management, so that’s next, but Keystone (and the other services) need access to the database, so:

Step 2: Power of the database servers

Now it’s important to note that a Galera cluster will not come online if all nodes were down.  After the servers boot, we’ll need to manually form the cluster again. So, on one node, we start the mysql service with a special parameter:

service mysql start --wsrep-new-cluster

Then on the other nodes, start as normal:

service mysql start

Another dependency of the controllers, in the case of Glance and Cinder, is our Ceph cluster, and possibly our Swift cluster. So let’s boot up those now:

Step 3: Power on the Ceph nodes

Step 4: Power on the Swift nodes

Once the database cluster and the storage clusters are online, we can then proceed to boot up our controllers:

Step 5: Power on the controllers

And finally, we can boot our compute nodes:

Step 6: Power on the compute nodes

Once everything is online, we can log into the dashboard and have a look at our instances. They’ll all be powered off at this point, so you’ll need to power on any instances that you want to bring up.

This is one of those things you don’t think about until you need it!

 

 

Leave a Reply