In our OpenStack High Availability Guide, we built quite a few servers, load balancers, database servers, controllers, compute nodes and storage nodes. Well the other day, I had a power failure and it all went down. Sorry, I don’t have UPS power in my home lab. Maybe this isn’t a likely scenario in a corporate data center, but on occasion you may be faced with a data center shutdown for power work or something like that. Anyway, I was faced with bringing the whole thing back online, and the question becomes, what components need to be brought up and in what order. This question may seem obvious to some, if you understand the dependencies between the various components, but it’s worth going over.
In the high availability guide, we configured all of the OpenStack services to use the load balancer VIP as the endpoint for each service, so first things first, let’s bring up the load balancers.
Step 1: Power on the load balancers
All of the OpenStack services rely on the Keystone service for identity and access management, so that’s next, but Keystone (and the other services) need access to the database, so:
Step 2: Power of the database servers
Now it’s important to note that a Galera cluster will not come online if all nodes were down. After the servers boot, we’ll need to manually form the cluster again. So, on one node, we start the mysql service with a special parameter:
Then on the other nodes, start as normal:
Another dependency of the controllers, in the case of Glance and Cinder, is our Ceph cluster, and possibly our Swift cluster. So let’s boot up those now:
Step 3: Power on the Ceph nodes
Step 4: Power on the Swift nodes
Once the database cluster and the storage clusters are online, we can then proceed to boot up our controllers:
Step 5: Power on the controllers
And finally, we can boot our compute nodes:
Step 6: Power on the compute nodes
Once everything is online, we can log into the dashboard and have a look at our instances. They’ll all be powered off at this point, so you’ll need to power on any instances that you want to bring up.
This is one of those things you don’t think about until you need it!