OpenStack High Availability – Glance and Swift

By | May 7, 2014

In this article, we’ll build a highly available OpenStack Icehouse Glance image service backed by highly available Swift object storage.  To make Glance highly available, our two (or more) Glance servers must store their images on shared storage.  We can use the Swift object store for the Glance storage, so we’ll kill two birds with one stone here, build both services, and configure Glance to use Swift as its storage back end.  This is part four of our OpenStack High Availability Controller Stack series.

In the last article, OpenStack High Availability – Keystone and RAbbitMQ, we built two Ubuntu 14.04 LTS servers for our OpenStack controllers and we installed Keystone and RabbitMQ on them.  The host names and IP addresses were:

  • icehouse1 (192.168.1.35)
  • icehouse2 (192.168.1.36)

We’ll install Glance on these in a few minutes, but first we should build our Swift object storage service.  For this, I’ll build three more servers:

  • swift1 (192.168.1.37)
  • swift2 (192.168.1.38)
  • swift3 (192.168.1.42)

On these servers, I’ve added a second hard disk (/dev/sdb) to use as the object storage.  If you don’t have a second hard disk available, you could reserve some disk space during the OS installation and create a separate partition.  Since I’m building these as virtual machines, adding another virtual disk is no big deal.  Also note that I’m going to combine the swift proxy and swift storage roles onto the same servers (I only have so much capacity in my lab).

Swift Storage

First, we’ll install the Swift storage components.

apt-get install swift swift-account swift-container swift-object xfsprogs

Once the installation is complete, we create the configuration file for Swift, where we define a unique hash for this group of Swift servers.  This file should be copied to all other Swift servers so they match.

/etc/swift/swift.conf

[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_suffix = MyDogHasFleas

Next, we’ll create the disk partition to use for Swift storage and format it as an xfs file system:

fdisk /dev/sdb
n 
(enter)
(enter)
(enter)
(enter)
w
mkfs.xfs /dev/sdb1

Then we’ll create a directory to mount it in and grant ownership to the swift user:

echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
mkdir -p /srv/node/sdb1
chown -R swift:swift /srv/node

We should also create a few other directories for Swift:

mkdir -p /var/swift/recon
chown -R swift:swift /var/swift/recon
mkdir /home/swift
chown -R swift:swift /home/swift

Swift uses rsync to replicate data between the storage nodes, so we need to configure rsync:
/etc/rsyncd.conf

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.1.37
 
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
 
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
 
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

Notice that we’ve specified the local IP address of the swift1 node in the example. On swift2 use the swift2 address (192.168.1.38), on swift3 use the swift3 address (192.168.1.42). We also need to enable rsync to start on boot. To do this, we modify /etc/default/rsync and change RSYNC_ENABLE to true:
/etc/default/rsync

RSYNC_ENABLE=true

and then start the rsync service:

service rsync start

Complete the steps above on both Swift nodes.

Swift Proxy

Now we’ll install the Swift proxy components.  Again, we’re installing this on the same three servers that we just installed the storage components on, but you can install the proxy on separate servers if you wish.

apt-get install swift-proxy memcached python-keystoneclient python-swiftclient python-webob

We need to configure memcached to listen on the local IP address. Change the listener line in memcached to use the local IP address of the node:
/etc/memcached.conf

-l 192.168.1.37

Then restart the service:

service memcached restart

Next, we create the proxy configuration file:
/etc/swift/proxy-server.conf

[DEFAULT]
bind_port = 8080
user = swift
 
[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server
 
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
 
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator
 
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
 
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
 
# cache directory for signing certificate
signing_dir = /home/swift/keystone-signing
 
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = 192.168.1.32
auth_port = 35357
 
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = Service123
 
[filter:cache]
use = egg:swift#memcache
memcache_servers = 192.168.1.37:11211,192.168.1.38:11211,192.168.1.42:11211
 
[filter:catch_errors]
use = egg:swift#catch_errors
 
[filter:healthcheck]
use = egg:swift#healthcheck

Notice that we’ve specified the IP address of our load balanced Keystone service (192.168.1.32), the IP addresses of our memcache servers, and the username and password of our swift user.

Swift Ring Configuration

Now, we’ll create our Swift ring configuration and add our storage locations.  We’ll create the necessary files in the /etc/swift directory on one node, and then copy the files to the other node. These files should be copied to all proxy and storage nodes.

cd /etc/swift
swift-ring-builder account.builder create 18 3 1
swift-ring-builder container.builder create 18 3 1
swift-ring-builder object.builder create 18 3 1

swift-ring-builder account.builder add z1-192.168.1.37:6002/sdb1 100
swift-ring-builder container.builder add z1-192.168.1.37:6001/sdb1 100
swift-ring-builder object.builder add z1-192.168.1.37:6000/sdb1 100

swift-ring-builder account.builder add z1-192.168.1.38:6002/sdb1 100
swift-ring-builder container.builder add z1-192.168.1.38:6001/sdb1 100
swift-ring-builder object.builder add z1-192.168.1.38:6000/sdb1 100

swift-ring-builder account.builder add z1-192.168.1.42:6002/sdb1 100
swift-ring-builder container.builder add z1-192.168.1.42:6001/sdb1 100
swift-ring-builder object.builder add z1-192.168.1.42:6000/sdb1 100

swift-ring-builder account.builder rebalance 
swift-ring-builder container.builder rebalance 
swift-ring-builder object.builder rebalance

Then we copy the files to the other nodes:

scp -r *.ring.gz root@192.168.1.38:/etc/swift
scp -r *.ring.gz root@192.168.1.42:/etc/swift

Then on both nodes, we make sure we grant ownership of the config files to the swift user and restart the services:

chown -R swift:swift /etc/swift
swift-init restart all

Load Balancing Swift

The last thing we need to do with Swift is to add it to our load balancer configuration. In the previous article, Redundant Load Balancers – HAProxy and Keepalived, we build redundant HAProxy servers. Now we’ll go back to those servers and add a stanza to /etc/haproxy/haproxy.cfg that points to our Swift proxies:
/etc/haproxy/haproxy.cfg

listen swift_proxy_cluster
        bind 192.168.1.32:8080
        balance  source
        option  tcpka
        option  tcplog
        server swift1 192.168.1.37:8080  check inter 2000 rise 2 fall 5
        server swift2 192.168.1.38:8080  check inter 2000 rise 2 fall 5
        server swift3 192.168.1.42:8080  check inter 2000 rise 2 fall 5

Notice that we’re binding to our VIP (192.168.1.32) and pointing to our two swift proxy nodes. To make the change take effect, we reload the HAProxy configuration:

service haproxy reload

OK, we now have a load balanced, redundant Swift service. To check it, we can use the swift stat command. As usual when working with OpenStack commands, we should source our environment file, which looks like this:
credentials

export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://192.168.1.32:35357/v2.0

and we source it like so:

source credentials

Then we can use the command:

swift stat

The output should look something like this:

root@swift1:~# swift stat
       Account: AUTH_16e744346cb5491393d04ea16c873b3e
    Containers: 0
       Objects: 0
         Bytes: 0
  Content-Type: text/plain; charset=utf-8
   X-Timestamp: 1399472382.37535
    X-Trans-Id: txb8a9cef38f434f36a8a82-00536a40fe
X-Put-Timestamp: 1399472382.37535

Glance Image Service

In this next section, we’ll install Glance on our controller nodes and use Swift as the high availability storage back end for Glance. However, an even better option (in my opinion) is to use Ceph as the back end for Glance. To do that, read the next article: OpenStack High Availability – Ceph Storage for Cinder and Glance, otherwise, you can proceed with this article.

Remember we built these two nodes in the previous article:

icehouse1 (192.168.1.35)

icehouse2 (192.168.1.36)

We install glance on both nodes:

apt-get install glance python-glanceclient

Glance has several config files that we need to modify. The first is /etc/glance/glance-api.conf. We’ll find and modify the following lines:
/etc/glance/glance-api.conf

...
default_store = swift
registry_host = 192.168.1.32
auth_strategy = keystone
...
#rabbit_host = localhost
rabbit_hosts = 192.168.1.35,192.168.1.36
...
swift_store_auth_address = http://192.168.1.32:5000/v2.0/
swift_store_user = service:swift
swift_store_key = Service123
swift_store_container = glance
swift_store_create_container_on_put = True
...
[database]
#sqlite_db = /var/lib/glance/glance.sqlite
connection=mysql://glance:Service123@192.168.1.32/glance
...
[keystone_authtoken]
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = Service123
...

Notice that we’re pointing to our VIP for authentication, database and swift. The next configuration file is /etc/glance/glance-registry.conf. Again we modify the following lines:
/etc/glance/glance-registry.conf

'''
[database]
# The file name to use with SQLite (string value)
#sqlite_db = /var/lib/glance/glance.sqlite
connection=mysql://glance:Service123@192.168.1.32/glance
'''
[keystone_authtoken]
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = Service123
'''

And the third config file is /etc/glance/glance-cache.conf:
/etc/glance/glance-cache.conf

...
registry_host = 192.168.1.32
...
swift_store_auth_address = http://192.168.1.32:5000/v2.0/
swift_store_user = service:swift
swift_store_key = Service123
swift_store_container = glance
swift_store_create_container_on_put = True
...

Now we create the glance database:

mysql -h 192.168.1.32 -u root -p
create database glance character set utf8 collate utf8_general_ci;
grant all on glance.* to glance@'%' identified by 'Service123';
flush privileges;
exit

Then on one node, we populate the database tables:

glance-manage db_sync

and restart the glance services on both nodes:

service glance-api restart
service glance-registry restart

Load Balancing Glance
Back on our load balancers, we add stanzas for the glance services:
/etc/haproxy/haproxy.cfg

listen glance-api 192.168.1.32:9292
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server icehouse1 192.168.1.35:9292 check inter 2000 rise 2 fall 5
        server icehouse2 192.168.1.36:9292 check inter 2000 rise 2 fall 5

listen glance-registry 192.168.1.32:9191
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server icehouse1 192.168.1.35:9191 check inter 2000 rise 2 fall 5
        server icehouse2 192.168.1.36:9191 check inter 2000 rise 2 fall 5

and reload:

service haproxy reload

We now have a load balanced, redundant glance service backed by redundant swift storage. To test, we can add a new image to glance. Again, make sure you’ve sourced your credentials file:

wget http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img
glance image-create --name cirros --is-public=true --disk-format=qcow2 --container-format=ovf < cirros-0.3.1-x86_64-disk.img
glance image-list

Here are the locations of Ubuntu 14.04 and 12.04 as well. Repeat the step above, replacing the URL, image name and file name to upload these into glance.

Ubuntu 14.04: https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
Ubuntu 12.04: https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

All done!

 

 

37 thoughts on “OpenStack High Availability – Glance and Swift

  1. Pedro Sousa

    Hi Brian,

    great work. I’m deploying the same HA architecture but using RDO distro based on RHEL. I have a question for you, how do you handle the network layer HA, specially with neutron L3-agents? Thanks.

    Reply
    1. Brian Seltzer

      The answer is, I don’t. I’m not using Neutron at all, I’m using nova-network, with the multi-node option. The nova-network service, the nova-metadata-api service, and dnsmasq are all deployed on every compute node, and I’m directing the instances to use the physical router as their defult gateway, so there is no single point of failure. I haven’t found a way to achieve this level of HA with Neutron.

      Reply
      1. ankit

        Hi Brian.

        I tried settingup glance + Swift.
        But somehow it is not working and throwing error 503 service inavailable from openstack dashboard.
        I have followed all the steps.
        I am using openstack mitaka.
        Can you please help out here.

        Reply
        1. Brian Seltzer Post author

          Hi Ankit,

          This post is related to OpenStack Icehouse. If you’re trying to use these instructions with a more recent version of OpenStack, I doubt they will work. I’ll be working on an updated post on how to deploy Swift on OpenStack Newton. Stay tuned.

          Reply
  2. bob

    Brian, Also should the Swift network be separate from the service management network?
    You list everything on the same network, but the documentation for Openstack shows it on it’s own storage network. Any concerns for latency?

    Reply
    1. Brian Seltzer Post author

      Making a separate storage network for replication is a good idea, since a lot of data change could flood that network and make it difficult to manage the platform. It’s not so much a latency concern. There’s also a consideration around security, if your OpenStack is Internet facing, then you’d want the management and storage networks to be seperate from the API and VM networks. You could end up with a lot of networks! I tried to keep my guide simple, otherwise it would get very complex, very quickly.

      Reply
  3. bob

    Hey Brian,
    If the swift proxy server fails, does it mean the data will be lost? Or, just rebuilding the proxy give you access to the data on the storage nodes?

    Reply
    1. Brian Seltzer

      I haven’t experienced a proxy failure, but I believe proxy failure/rebuild would not impact the data on the storage nodes.

      Reply
      1. bob

        I think I misunderstood the set up. I thought, 1 proxy in front of 2 storage nodes. It seems that you are setting up a proxy on each node and storage on each node and then load balancing through HAproxy. Is that correct?

        Reply
        1. Brian Seltzer

          Correct. Actually I’ve updated the article to build three nodes, with both proxy and storage services on each. In a production deployment, you might want to run the proxy services on dedicated nodes (at least two) and have many dedicated storage nodes.

          Reply
          1. bob

            That’s were need to make a decision, this is a hybrid production system, meaning it’s used in a research center for grad students. So it’s constantly changing. Each server has 2 – 2TB drives. I think at most I would have 5 nodes, but who knows. Do you think it would be more advantageous to set up 2 proxy nodes and storage behind it or start with the 3 nodes now running both proxy and storage, and if I need to add I would add just storage nodes? If I go with 2nd option would I need to add proxy to each additional node?

          2. Brian Seltzer

            You should definitely have at least three storage nodes (turns out two nodes isn’t enough). So yeah, you could put proxy and storage services on the three nodes and load balance them with HAProxy, and yeah, when you add storage nodes, you don’t have to put proxy on those.

  4. bob

    Hi Brian, Can you confirm which node you are running swift stat command (to verify swift install) from? I tried on Haproxy and getting error program ‘swift’ is currently not installed. . Tried from one of the swift nodes and get error ” requires OS_AUTH_URL, OS_USERNAME, OS_PASSWORD,” even though I specified the source credentials

    Reply
    1. Brian Seltzer

      I ran the swift stat command from one of the swift storage nodes (notice swift is in the apt-get install command at the top of the article). However, you can ‘apt-get install swift’ anywhere you want.

      Reply
      1. bob

        So am i using the wrong source credentials for swift, the instructions say to use admin, shouldn’t you use swift?

        Reply
      2. bob

        Getting : Auth version 2.0 requires OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, and

        OS_TENANT_NAME OS_TENANT_ID to be set or overridden with –os-auth-url,

        –os-username, –os-password, –os-tenant-name or os-tenant-id. Note:

        adding “-V 2” is necessary for this.

        Reply
  5. bob

    Hi Brian,

    I think my source is not functioning correctly. I set up glance and try to run command glance image-list, and get back error “The request you have made requires authentication. (HTTP 401)”. I also tried running swift stat and get back “Unauthorised. Check username, password and tenant name/id”.. I am running this from one of the controllers and I sourced below file.
    Can you help figure out the problem?

    openrc_credentials:

    export OS_USERNAME=admin
    export OS_PASSWORD=admin_password
    export OS_TENANT_NAME=admin
    export OS_AUTH_URL=http://:35357/v2.0

    Reply
    1. Brian Seltzer Post author

      I assume that the OS_AUTH_URL has the IP address of the HAProxy VIP after the http:// and before the :

      If so, and you see multiple services unable to authenticate, I would verify that your HAProxy config is working. Can you successfully use keystone commands? Like: “keystone user-list” ?

      Reply
      1. bob

        Yes the OS_AUTH_URL is export OS_AUTH_URL=http://:35357/v2.0.
        All keystone commands work perfect once I source the file. The other ones do not.
        I suspect something went wrong with the keystone populate file. I checked the DB directly,
        and the users are there for glance, swift, etc are there.

        Reply
        1. bob

          Also noticed when running user update command with keystone received below warning even though command worked.

          WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).

          Reply
      2. bob

        Sorry keeps deleting url yes OS_AUTH_URL=http:/10.1.0.2vip/:35357/v2.0
        After I reset the password for the admin user:
        1. glance image-list worked.

        2. swift now throws error : Account HEAD failed:http://10.1.0.2:8080:8080/v1/AUTH_0bf091dd07874345b7a075b20dfd283a 401 Unauthorized

        3. running keystone update command gives : WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).

        4. Just to confirm, no keystone user was created, it appears as though admin is assigned as keystone user, correct?

        Reply
          1. Avinash Mahto

            Hey this must be issue with swift proxy server, Please check the keystone setting in your swift proxy server. Also make sure keystone should be perfectly working.

      3. bob

        Hi Brian,

        I finally got it to work. After hours I found the keystone script I used set the passwords incorrectly for some users. After I corrected the passwords, I received new error ,”head failed 503 internal server error”. I determined that even though your instructions create directory to mount 2nd drive, I never actually mounted it. Once mounted, everything started working.

        Again thanks for all your help, this is really helping me get more familiar with linux in general.
        Bobby..

        Reply
  6. bgyako

    Hey Brian,
    Running into issue uploading the image. Getting error “Request returned failure status.
    500 Internal Server Error” I bypassed the Haproxy and pointed glance directly at one of the swift servers and it worked. Any idea why?

    Reply
    1. bgyako

      Never mind, got it to work. Looks likes when I set permissions for a directory in swift, it did not replicate.

      Reply
      1. PremKumar

        Hi bgyako,
        Can u tell me what step u did for this to work , Change permissions for swift directory mean how to change can u tell me details am also stuck in these for long time can u help me ?

        Reply
        1. Avinash Mahto

          Hi Prem,
          I was also facing the same error while I was trying to change 755 permission & ownership swift:swift of the directory /var/cache/swift but it replicate in its previous stats.

          What I did to solve this issue, wanted to share with you.

          First stop all swift services if any running then change ownership of directory with root:root (/var/cache/swift/* ). After that change permission 755.
          once you finish it then change ownership to swift:swift. it works.

          Might be account, container and object file is being used by some swift services which stops it to change permission.
          Let me know if you face problem.

          Gd luck 🙂

          Reply
  7. PremKumar

    Hi ,
    Am trying to do swift as back-end storage for glance while executing in the glance create message am getting this error

    root@osservices:~#glance image-create –name=”cirros” –disk-format=qcow2 –container-format=bare –is-public=true < cirros-0.3.1-x86_64-disk.img
    Request returned failure status.
    500 Internal Server Error
    Failed to upload image 6411a5ce-80f9-4a05-b82b-c48c81b67bce
    (HTTP 500)

    Reply
    1. Brian Seltzer Post author

      Make sure that you’ve performed all necessary steps on all of the nodes. Notice the comment below where the same problem occured. In their case, they didn’t set the permissions on all of the nodes…

      Reply
  8. PremKumar

    hi Brian Seltzer ,
    For swift installation i followed this link http://docs.openstack.org/developer/swift/development_saio.html , then i used openstack doc to install all service then, i followed your step in the glance service to configure the swift as back-end, My swift is working fine i can able to store data and download, but when i migrate with that to the glance service, i am not able to store, and you said permission of the node actually am not getting what permissions i want give can you help me …..

    Reply
    1. Brian Seltzer Post author

      there are three places in the article above where I use the chown command for various swift directories. Did you run these?

      Reply
  9. dougszumski

    Great job on these articles, they’re really well written.

    Possible typo wrapped with **:

    server swift1 192.168.1.37:8080 check inter 2000 rise 2 fall 5
    server swift2 192.168.1.38:8080 check inter 2000 rise 2 fall 5
    server ***swift2** 192.168.1.42:8080 check inter 2000 rise 2 fall 5

    Notice that we’re binding to our VIP (192.168.1.32) and pointing to our **two** swift proxy nodes. To make the change take effect, we reload the HAProxy configuration:

    Reply
  10. Avinash Mahto

    Dear Brian,

    It’s really very helpful and knowledgeable documents. All the steps is mentioned in very easy way that can be easily understand.

    Many many thanks for sharing this with us. Appreciate your effort and knowledge..
    Thanks!!
    Avinash Mahto

    Reply

Leave a Reply