OpenStack High Availability – Heat Orchestration Service

By | May 18, 2014

In the last few articles, we built the highly available OpenStack Icehouse core services. Now we will add the Heat orchestration service. This continues our high availability series:

In the previous article, we built two controllers, which are:

  • icehouse1 (
  • icehouse2 (

We will now install the Heat packages on these controllers. First we install the packages on both nodes:

apt-get install heat-api heat-api-cfn heat-engine

Then we modify /etc/heat/heat.conf and add or set the following lines:


verbose = True

heat_metadata_server_url =
heat_waitcondition_server_url =


connection = mysql://heat:Service123@

auth_uri =

auth_host =
auth_port = 35357
auth_protocol = http
auth_uri =
admin_tenant_name = service
admin_user = heat
admin_password = Service123

Notice that we’re using our load balancer VIP ( for all service endpoints and our two controller addresses for rabbit. We can copy the config file from one node to the other easily:

scp /etc/heat/heat.conf root@
ssh root@ chown heat:heat /etc/heat/heat.conf

In the article when we installed Keystone, we only defined users, roles, services and endpoints for the core services, so we’ll now add them for Heat. Before we do that, we need to source our credentials file:

source credentials

Then we can create the various Keystone objects for Heat:

keystone user-create --name=heat --pass=Service123
keystone user-role-add --user=heat --tenant=service --role=admin
keystone service-create --name=heat --type=orchestration --description="Orchestration"
keystone endpoint-create --service-id=$(keystone service-list | awk '/ orchestration / {print $2}') \
  --publicurl=\(tenant_id\)s \
  --internalurl=\(tenant_id\)s \
keystone service-create --name=heat-cfn --type=cloudformation --description="Orchestration CloudFormation"
keystone endpoint-create --service-id=$(keystone service-list | awk '/ cloudformation / {print $2}') \
  --publicurl= \
  --internalurl= \
keystone role-create --name heat_stack_user

We also need to create the database:

mysql -h -u root -p
GRANT ALL PRIVILEGES ON heat.* TO 'heat'@'localhost' IDENTIFIED BY 'Service123';
GRANT ALL PRIVILEGES ON heat.* TO 'heat'@'%' IDENTIFIED BY 'Service123';
flush privileges;

Then populate the database tables:

heat-manage db_sync

And finally restart the services on both nodes:

service heat-api restart
service heat-api-cfn restart
service heat-engine restart

Now, we need to add our new services to our load balancers. Remember we have built a pair of load balancers which are:

  • haproxy1 (
  • haproxy2 (

Add the following lines to the config file on both nodes:

listen heat_api_cluster
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server icehouse1  check inter 2000 rise 2 fall 5
  server icehouse2  check inter 2000 rise 2 fall 5

listen heat_cf_api_cluster
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server icehouse1  check inter 2000 rise 2 fall 5
  server icehouse2  check inter 2000 rise 2 fall 5

and then reload the configuration:

service haproxy reload

You should now have a highly available orchestration service. To test, we’ll perform the standard quick test. Create a test template:


heat_template_version: 2013-05-23

description: Test Template

    type: string
    description: Image use to boot a server
    type: string
    description: Network ID for the server

    type: OS::Nova::Server
      name: "Test server"
      image: { get_param: ImageID }
      flavor: "m1.tiny"
      - network: { get_param: NetID }

    description: IP address of the server in the private network
    value: { get_attr: [ server1, first_address ] }

and then create the test stack:

NET_ID=$(nova net-list | awk '/ vmnet / { print $2 }')
heat stack-create -f test-stack.yml -P "ImageID=cirros;NetID=$NET_ID" testStack

The stack creation should begin. The following command will show the status:

# heat stack-list
| id                                   | stack_name | stack_status    | creation_time        |
| 50935e5b-cdfb-4e80-90ad-7b80d420684a | testStack  | CREATE_COMPLETE | 2014-05-18T12:47:09Z |

That’s it. Stay tuned for more, as we build the remaining services…


One thought on “OpenStack High Availability – Heat Orchestration Service

  1. ZhanHan

    I configure the heat ha by ngnix, the heat-api is working well,but when the scale-up or scale-down policy is triggered,my heat-api-cfn logged “AWS authentication failure”, and it’s post request is give a 403, but
    it is the same with the request authenticated successful. Did anyone has the same problem? Mail me your advice,please.


Leave a Reply