OpenStack High Availability – First Compute Node

By | May 13, 2014

In this article, we’ll build our first OpenStack Icehouse compute node that uses the high availability controller stack that we built in the last few articles. This same process can be used to build additional compute nodes.

We’ll build an Ubuntu 14.04 LTS server for our compute node:

  • icehouse-compute1 (192.168.1.45)

This server will require two network adapters.  The first (eth0) will have the IP address above.  The second will be on a separate network 192.168.2.0/24 and will be linked to a Linux bridge and will not have an IP address defined on it.

Remember, our architecture will ultimately look like this:

OpenStack Network

To configure the network, we’ll edit the /etc/network/interfaces file and configure it like this:

/etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.45
netmask 255.255.255.0
gateway 192.168.1.1
dns-search mydomain.com
dns-nameservers 192.168.1.253 192.168.1.252

# This is an autoconfigured IPv6 interface
iface eth0 inet6 auto

auto eth1
iface eth1 inet manual

auto br100
iface br100 inet manual
up ip link $IFACE promisc on
up ip link eth1 promisc on
bridge_ports eth1

Adjust your DNS domain name and addresses as appropriate. Next, we install and configure the packages:

apt-get install nova-compute-kvm python-guestfs nova-network nova-api-metadata

Note: There’s an issue with kernel permissions that going on at the time of this writing. If you find that you can’t run any instances, you can try typing:

dpkg-statoverride  --update --add root root 0644 /boot/vmlinuz-$(uname -r)

This will relax the permissions on the kernel. I believe a fix is being worked on for this problem, so you may or may not run into this.

Now we configure Nova. Edit the /etc/nova/nova.con and add the following lines:

/etc/nova/nova.conf

...
share_dhcp_address = True
force_dhcp_release = True
flat_network_bridge = br100
flat_interface = eth1
public_interface = eth1

dnsmasq_config_file=/etc/dnsmasq-nova.conf

glance_host = 192.168.1.32

rpc_backend = rabbit
rabbit_hosts = 192.168.1.35,192.168.1.36

my_ip = 192.168.1.45
vnc_enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 192.168.1.45
novncproxy_base_url = http://192.168.1.32:6080/vnc_auto.html

auth_strategy = keystone

[database]
connection = mysql://nova:Service123@192.168.1.32/nova

[keystone_authtoken]
auth_uri = http://192.168.1.32:5000
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = Service123

Notice that I’m pointing to the load balancer VIP for glance, novncproxy, database and keystone.  I’m using this compute node’s IP address for my_ip and vncserver, and I’m using the IP addresses of the two controller nodes for rabbit.

A quick note for those who are testing. If your compute node is a VM (or otherwise incapable of hardware-accelerated virtualization), set the following in /etc/nova/nova-compute.conf

/etc/nova/nova-compute.conf (only for virtual compute nodes!)

# /etc/nova/nova-compute.conf
virt_type = qemu

One more tip, if you want to use the router on your 192.168.2.0 network, rather than using the dnsmasq as the router, add your router’s IP address to the file /etc/dnsmasq-nova.conf.  This greatly improves network performance of your VMs.

/etc/dnsnasq-nova.conf

dhcp-option=3,192.168.2.254

At this point a reboot is advised, or at least restart the nova services:

service nova-compute restart
service nova-api-metadata restart
service nova-network restart

Once the reboot or service restart is complete, one last step is to define the network for the instances to use:

nova network-create vmnet --fixed-range-v4=192.168.2.0/24 --bridge-interface=br100 --multi-host=T --dns1=192.168.1.1

That’s it. You should now be able to logon to the Horizon dashboard and start provisioning VMs. Of course, you should do the usual stuff (create or import SSH keys, enable SSH access in your access groups, etc. Enjoy!

Note: To add more compute nodes, simply allocate another management IP address for the new compute node (e.g. 192.168.1.46), then repeat this process again, replacing the IP address as you go along (replace any instances of 192.168.1.45 with 192.168.1.46).

 

 

23 thoughts on “OpenStack High Availability – First Compute Node

    1. Brian Seltzer

      Thanks Samuel. I ran into that problem this morning. I’ve removed that bit from the article and left just a note about the issue in case users run into it.

      Reply
  1. wandy

    what if I make HA with 2 nodes and 1 controller. can be used?

    Reply
  2. aswandi wandy

    if I want to use HA on nodes how ?
    if node1 is turned off, there will be an error in the instance and not can create an instance on node2

    Reply
  3. Alex Kamalov

    Hey Brian, thanks so much for such insightful blog. Question: while running “nova network-create vmnet –fixed-range… ” I am getting the following error:

    ERROR: (HTTP 504)

    Have you had one of these? I scoured the web, the only thing that I cam across was this:

    https://bugs.launchpad.net/nova/+bug/1264748

    Any ideas?

    Thanks,

    Alex

    Reply
    1. Brian Seltzer

      Hi Alex, First I would verify that your command line can locate the nova services. Try the command “nova service-list”. If this fails, then either your credentials file is not created properly and you can’t access keystone, or your nova endpoint in keystone is incorrect. “keystone endpoint-list | grep 8774” will show you the nova endpoint. If these don’t help, I would try the command again and then look at the logs. Look in /var/log/nova for any log files on your controllers and compute nodes to spot the error. You may need to set debug=true in your nova.conf and restart the nova services to get a better detail on the error. Hope that helps.

      Reply
  4. Alex

    Hello Brian,

    I installed all the nodes as you said except of the volume, which I stored at the compute node.
    Everything is ok except of one thing. The instance is not accessible outside the compute node.
    E.g. I can connect to the instance through the compute node or another instance, but I can’t connect to it from from outside, even from the controller node which is on the same network.
    I think is something wrong with the br100 or the dhcp leases. Can u help me?

    Thanks in advance, great article.

    Reply
  5. bgyako

    Hi Brian,
    Having a weird issue with controller recognizing the compute node.
    Although compute nova error states can’t reach local:5672, I confirmed that nova.conf
    and neutron.conf point rabbitmq to my controllers (10.1.0.5, 10.1.0.6). Please help.

    Getting error in Compute Neutron log:
    58b-d2086f006bce None] AMQP server on 10.1.0.6:5672 is unreachable: [ Errno 113] EHOSTUNREACH. Trying again in 7 seconds.
    2014-08-11 11:41:46.161 2023 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)

    and

    Compute nova log:
    2014-08-11 12:01:10.581 7067 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.

    Reply
    1. bgyako

      Issue resolved, rabbit info was in the wrong section of nova.conf. Added to DEFAULT section issue resolved.

      Reply
  6. 1mike

    Hi Brian,

    1.I run the below command on compute01 and gets error that I me missing: python-novaclient. So I install nova client on comput01 and try again. It’s completed

    nova network-create vmnet –fixed-range-v4=172.20.4.0/24 –bridge-interface=br100 –multi-host=T –dns1=172.21.1.3 –dns2=172.21.1.4

    Then when I try to create Instance gets error as attachment:

    http://1drv.ms/1IprtCj

    2. Do I have to run the above command on all compute nodes?

    Please advise me how I can overcome this problem.

    Regards,
    MIKE

    Reply
    1. Brian Seltzer Post author

      I don’t see an error attached. No, you don’t need to run it on all compute nodes, you only need to run the command once.

      Reply
      1. 1mike

        Hi Brian,
        Please advice me how can I fix this problem. I need to fix this problem only. Rest others parts are completed. Please find the below link for error details.

        Regards,
        MIKE

        Reply
        1. Brian Seltzer Post author

          It looks like your dnsmasq-nova.conf file is misspelled dnsnasq-nova.conf. Rename the file to the correct name.

          Reply
          1. 1mike

            Hi Brian,

            Thanks and I did correct the misspelled. after that I restart all the nova services but now I get new error as below attachment.

            Regards,
            MIKE

          2. Brian Seltzer Post author

            How are you provisioning the instance, from a glance image, or from a cinder volume? In any case, you need to look for errors in the nova logs on the controller as well as the compute nodes. Look at the log files in /var/log/nova/

  7. 1mike

    Dear Brian,
    Please accept my biggest thanks for wonderful and great articles on High availability on Open Stack. Due to which I can able to complete my project.

    Now I can able to create Instances. Before I had problem but it fixed automatically without doing anything. I use the Open Stack Console to create all VM Guests. Can you please advise me on below questions?

    1.Network Problem: From Open Stack VM Guest, I can ping and access my internal office network as well as internet but from my computer I cannot able to ping or access the Open Stack VM Guest. I check with network side they inform me they allow access to all VLAN. Its look like one ways traffic.

    2.Can I able to create a VM Guest with ISO file: I tried to use ISO file for Centos 7 to create a new VM Guest same as we can do in VMware. But I can’t able to get the Root Disk even I use the flavors: m1.medium. Later I tried to add volume but still not work.

    3.I had completed your article “Booting from a Ceph Volume” but when I create Instance as CephTest1 its fails with error. Do I have to use hotfix of the Glance.py module?

    Reply
    1. Brian Seltzer Post author

      To enable pinging the VM from the external network, you must modify your default security group and add a rule for ICMP ingress.

      To create images from an ISO: read this: http://docs.openstack.org/image-guide/content/ch_creating_images_manually.html
      Or you can download ready-made images: http://docs.openstack.org/image-guide/content/ch_obtaining_images.html

      I think the glance hotfix has been integrated into the distro by now. There’s a lot to check to see why you can’t boot from a ceph volume. The image must be in raw format (not qcow format). You must succesfully upload the raw image into glance. Cinder must be able to create a volume from the raw image in glance. The compute nodes must be able to access the volume via the rbd protocol. Assuming that the image is in raw format and you successfully created the volume, and the error is occuring when you try to deploy the instance, you should look at your nova logs on the compute node where the instance failed to start.

      Reply
  8. MIKE

    Hi Brain,

    1.I tried the default and also created a new as test security group but still cannot able to ping the VM. Please see the config:

    http://1drv.ms/1MIh9qu

    2.My Image is in Raw format and its appearing in Glance but still not coming. Please check the above link for Volume created and nova logs.

    3.What is the user name and password for the image “Ubuntu 14.04: https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img” because I can’t able to login.

    4.How can I add 1 more Controller in my current Open stack? My boss wants me to add 1 more controller so that we have 3 controllers.

    Please advice me.

    Regards,
    Mike

    Reply
    1. Brian Seltzer Post author

      1. In your security rules, remote should be a CIDR (network range) like 0.0.0.0/0
      2. Any volume creation error would be in the cinder logs.
      3. Password authentication is disabled in the ubuntu image. The default username is ubuntu and you need to provide your public key to establish an ssh session.
      4. You can simply install another controller and follow the same steps that you did for the first two.

      Reply

Leave a Reply