OpenStack High Availability – Ceph Storage for Cinder and Glance

By | May 10, 2014

In this article, we confinue our build of the OpenStack Icehouse high availability controller stack. We’ll build a Ceph storage cluster and use it as the back-end storage for the Cinder block storage service and the Glance image service. In the last article, we built a Swift storage cluster and used that for Glance storage, but Ceph is perhaps a better alternative to Swift, in that it can also be used for Cinder. We’ll build a small Ceph storage cluster to demonstrate the process, but of course you should do your own research if you plan to build a production-scale Ceph cluster.

This is part of our series:

Ceph Storage Cluster

I’ll build a three-node storage cluster using Ubuntu 14.04 LTS.  The Nodes are:

  • ceph1 (192.168.1.39)
  • ceph2 (192.168.1.40)
  • ceph3 (192.168.1.41)

Each of these servers have a second hard disk (/dev/sdb) that we’ll use to store Ceph data.  It’s important that all of the Ceph nodes be able to resolve each other’s IP addresses, so let’s add the other two nodes to each node’s /etc/hosts file.  For example, ceph1’s /etc/hosts file looks like this:

127.0.0.1	localhost
127.0.1.1	ceph1
192.168.1.40	ceph2
192.168.1.41	ceph3

Ceph includes a deployment tool called ceph-deploy that helps you deploy the Ceph cluster without having to perform installations on each node. Instead, we can install ceph-deploy on one node and manage the deployment from there. As of this writing, the latest stable version of Ceph is Firefly, so we’ll use that.  The first thing we need to do is add the Firefly repository and associated key to our list of Ubuntu APT repositories on ceph1:

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph-deploy

Once that’s complete, we can begin defining our new cluster.  It’s a good idea to create a directory in your home directory for the next few steps, since several files will be created.  Then you can define the first node in the cluster, install ceph onto the nodes, and define which nodes will perform monitoring role:

mkdir ceph-cluster
cd ceph-cluster
ceph-deploy new ceph1
ceph-deploy install ceph1 ceph2 ceph3
ceph-deploy mon create ceph1 ceph2 ceph3
ceph-deploy gatherkeys ceph1

Once the installation has completed, it’s time to start provisioning storage for the cluster. The unit of storage provisioning in Ceph is the object storage daemon or OSD. We’ll create a OSD on each node using the extra hard disks on each node (/dev/sdb):

ceph-deploy osd --zap-disk create ceph1:sdb
ceph-deploy osd --zap-disk create ceph2:sdb
ceph-deploy osd --zap-disk create ceph3:sdb

One last thing we should do with ceph-deploy is to push out the key that was created, so that we can perform administrative functions on any node of the cluster:

ceph-deploy admin ceph1 ceph2 ceph3

You’ll want to make sure that this key has the correct permissions assigned. Unfortunately, we have to ssh to each node to perform this step:

chmod +r /etc/ceph/ceph.client.admin.keyring

Now, on any node, we can create a storage pool that we can use to provision storage for cinder and glance:

ceph osd pool create datastore 150

Here’s we created a storage pool called ‘datastore’ with 150 placement groups.  The number of placement groups is based on 100 x the number of OSD’s / the number of replicas you want to maintain.  I want two copies of the data (so if a server fails no data is lost), so 3 x 100 / 2 = 150.  This is a bare minimum configuration for lab testing.  In production you would probably want many OSD’s and at least three replicas, but Ceph design is beyond the scope of this article.

A quick health check using the command ‘ceph health’ or ‘ceph -w’ should show that everything is OK. Here’s the output of ‘ceph -w’ showing that all of my placement groups are active and clean:

$ ceph -w
    cluster 4e6556b7-6ca2-4505-a023-9b4bfb1e19ed
     health HEALTH_OK
     monmap e1: 1 mons at {ceph1=192.168.1.40:6789/0}, election epoch 2, quorum 0 ceph1
     osdmap e16: 3 osds: 3 up, 3 in
      pgmap v67: 342 pgs, 4 pools, 12839 kB data, 5 objects
            147 MB used, 76611 MB / 76759 MB avail
                 341 active+clean
                   1 active+clean+scrubbing

2014-05-09 14:35:00.418136 mon.0 [INF] pgmap v66: 342 pgs: 341 active+clean, 1 active+clean+scrubbing; 12839 kB data, 147 MB used, 76611 MB / 76759 MB avail

Glance and Ceph

Now back to our high availability OpenStack deployment.  Remember in the article OpenStack High Availability – Keystone and RabbitMQ we built two Ubuntu 14.04 servers to use as our OpenStack controllers. You could easily deploy Glance and Cinder on their own servers, but I’ll just install them on the same two servers (my lab is only so big you know).  Remember, our two controllers are:

  • icehouse1 (192.168.1.35)
  • icehouse2 (192.168.1.36)

On our controller nodes, we’ll need to install the Ceph client, so again, let’s add the repository and then install what we need. While we’re at it we’ll also install glance itself:

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph-common python-ceph glance python-glanceclient

After this is done on both nodes, we should go back to our Ceph cluster and create a Ceph user for our controllers to use.  On a Ceph node:

ceph auth get-or-create client.icehouse mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=datastore'

ceph auth get-or-create client.icehouse | ssh 192.168.1.35 sudo tee /etc/ceph/ceph.client.icehouse.keyring
ceph auth get-or-create client.icehouse | ssh 192.168.1.36 sudo tee /etc/ceph/ceph.client.icehouse.keyring
ssh 192.168.1.35 sudo chmod +r /etc/ceph/ceph.client.icehouse.keyring
ssh 192.168.1.36 sudo chmod +r /etc/ceph/ceph.client.icehouse.keyring

This creates the user and copies the keyring to our controllers. While we’re on the Ceph node, we should also modify our ceph configuration to use all of the monitors we’ve setup.  Edit ceph.conf and modify the following line:

/etc/ceph/ceph.conf

...
mon_host = 192.168.1.39,192.168.1.40,192.168.1.41
...

then push ceph.conf out to our controller nodes:

ssh 192.168.1.35 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
ssh 192.168.1.36 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf

Now onto the Glance configuration. First we install the packages:

apt-get install glance python-glanceclient

Then we modify our files, adding or changing the lines shown below:

/etc/glance/glance-api.conf

...
default_store = rbd
registry_host = 192.168.1.32
auth_strategy = keystone
...
#rabbit_host = localhost
rabbit_hosts = 192.168.1.35,192.168.1.36
...
rbd_store_user = icehouse
rbd_store_pool = datastore
rbd_store_chunk_size = 8
show_image_direct_url = True
...
[database]
#sqlite_db = /var/lib/glance/glance.sqlite
connection=mysql://glance:Service123@192.168.1.32/glance
...
[keystone_authtoken]
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = Service123
...

/etc/glance/glance-registry.conf

'''
[database]
# The file name to use with SQLite (string value)
#sqlite_db = /var/lib/glance/glance.sqlite
connection=mysql://glance:Service123@192.168.1.32/glance
'''
[keystone_authtoken]
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
admin_password = Service123
'''

/etc/glance/glance-cache.conf

...
registry_host = 192.168.1.32
...

Notice that these files point to our load balancer VIP for authentication and the glance registry.  Next we create the Glance database:

mysql -h 192.168.1.32 -u root -p
create database glance character set utf8 collate utf8_general_ci;
grant all on glance.* to glance@'%' identified by 'Service123';
flush privileges;
exit

Then we restart glance and populate the database (only on the first node)

service glance-api restart
service glance-registry restart
glance-manage db_sync

And restart the services on the second node as well:

service glance-api restart
service glance-registry restart

Finally, we need to add glance to our load balancer configuration. Remember, back in the article Redundant Load Balancers – HAProxy and Keepalived, we built a pair of load balancers. The two nodes were:

  • haproxy1 (192.168.1.30)
  • haproxy2 (192.168.1.31)

We modify the /etc/haproxy/haproxy.cfg on both nodes and add stanzas for glance-api and glance-registry:

/etc/haproxy/haproxy.cfg

listen glance-api 192.168.1.32:9292
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server icehouse1 192.168.1.35:9292 check inter 2000 rise 2 fall 5
        server icehouse2 192.168.1.36:9292 check inter 2000 rise 2 fall 5

listen glance-registry 192.168.1.32:9191
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server icehouse1 192.168.1.35:9191 check inter 2000 rise 2 fall 5
        server icehouse2 192.168.1.36:9191 check inter 2000 rise 2 fall 5

Then reload the haproxy configuration on both nodes:

service haproxy reload

OK, we now have a high availability glance service with high availability ceph storage behind it. We can test glance by getting an image and uploading it. As usual, before attempting to use an OpenStack command line tool, setup your environment using your credentials file:
credentials

export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://192.168.1.32:35357/v2.0

and source it:

source credentials

Then we can perform the test:

wget http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img
glance image-create --name cirros --is-public=true --disk-format=qcow2 --container-format=bare < cirros-0.3.1-x86_64-disk.img
glance image-list

You should see output like this:

# glance image-list
+--------------------------------------+--------+-------------+------------------+----------+--------+
| ID                                   | Name   | Disk Format | Container Format | Size     | Status |
+--------------------------------------+--------+-------------+------------------+----------+--------+
| be5c4f9e-b8c8-4ed8-91e7-cef6eaf64e0a | cirros | qcow2       | bare             | 13147648 | active |
+--------------------------------------+--------+-------------+------------------+----------+--------+

Here are the locations of Ubuntu 14.04 and 12.04 as well. Repeat the step above, replacing the URL, image name and file name to upload these into glance.

Ubuntu 14.04: https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
Ubuntu 12.04: https://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

Cinder and Ceph

OK, now we’ll install Cinder.

sudo apt-get update && sudo apt-get install cinder-api cinder-scheduler cinder-volume

Then we add the following to cinder.conf
/etc/cinder/cinder.conf

'''
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=datastore
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot=false
rbd_max_clone_depth=5
rbd_user=icehouse
glance_api_version=2
'''
rpc_backend = cinder.openstack.common.rpc.impl_kombu
rabbit_hosts = 192.168.1.35,192.168.1.36
rabbit_port = 5672
'''
glance_host = 192.168.1.32
'''
[database]
connection = mysql://cinder:Service123@192.168.1.32/cinder
'''
[keystone_authtoken]
auth_uri = http://192.168.1.32:5000
auth_host = 192.168.1.32
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = cinder
admin_password = Service123
'''

Again, notice that we’re pointing to our load balancer VIP for most services. Next we create the cinder database:

mysql -h 192.168.1.32 -u root -p
create database cinder;
grant all on cinder.* to cinder@'%' identified by 'Service123';
flush privileges;
exit

Then we populate the database (on node only), and restart the services:

cinder-manage db sync
service cinder-api restart
service cinder-scheduler restart
service cinder-volume restart

Then back to our load balancers, we add the stanza for glance-api:
/etc/haproxy/haproxy.cfg

listen cinder_api 192.168.1.32:8776
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server icehouse1 192.168.1.35:8776  check inter 2000 rise 2 fall 5
  server icehouse2 192.168.1.36:8776  check inter 2000 rise 2 fall 5

and reload the haproxy configuration:

service haproxy reload

Now we have a high availability cinder service with high availability Ceph storage behind it. To test, we source our credentials file and then create a new volume:

cinder create --display-name testVolume 1
cinder list

You should see output like this:

# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 87e8ade4-0ea9-4da9-a200-fa2e4b60fbea | available |  testVolume  |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

Stay tuned for more of our OpenStack Icehouse high availability series!

 

 

42 thoughts on “OpenStack High Availability – Ceph Storage for Cinder and Glance

  1. bob

    Brian,
    Are the instructions for cinder the same weather you use ceph or swift?

    Reply
      1. bob

        So do I just remove those lines or do I need to replace with shift equivalent?
        Looking at install instructions at openstack.org, and it does not mention modifying cinder config for swift.

        Reply
      2. bob

        Brian,
        To configure cinder to use swift, Do I need to do anything on the swift servers?
        In addition, cinder config should look like this:
        default_store = swift
        registry_host = 192.168.1.32
        auth_strategy = keystone

        #rabbit_host = localhost
        rabbit_hosts = 192.168.1.35,192.168.1.36

        swift_store_auth_address = http://192.168.1.32:5000/v2.0/
        swift_store_user = service:swift
        swift_store_key = Service123
        swift_store_container = cinder
        swift_store_create_container_on_put = True

        Reply
        1. Brian Seltzer Post author

          I haven’t done Cinder integration with Swift, though I should of course. I’ll put that on my list of stuff to do 🙂

          Reply
          1. bob

            uh oh. Will have to figure it out, then. So in your instructions, you set up glance using swift and cinder using ceph?

  2. bob

    Hi Brian,
    I finally got around to reading up on this, and light bulb went off. Was missing that 2 types of storage is required(object, block). Anyway when running install of ceph I get error “[ERROR ] RuntimeError: remote connection got closed, ensure “requiretty“ is disabled for host2”
    I made sure there is no settings for requiretty in the sudoers file, and I believe it is disabled by default in Ubuntu 14. Are you missing a step where I have to create a ceph user on each additional node?

    Reply
  3. bob

    Hi Brian,

    Looks like I had to run below command to add sudo privileges for the user on each additional Ceph Node. Is that correct? command: user all : ALL | te etc sudoers.d ceph
    for some reason couldn’t post exact command….

    Reply
  4. bgyako

    Hi Brian, Having issue with creating volumes. I noticed in the cinder.conf you have rbd_user=icehouse , when and where in the steps did you create that user for ceph? Shouldn’t that user be cinder?

    Reply
  5. bgyako

    Hey Brian, Is there a step to set up block vs object storage? I am able to create volumes and I am able to create images (glance pointed at swift, cinder pointed at ceph), but unable to attach volume to instance.

    Reply
  6. Phillip

    Hi Brian,
    Loving the tutorials you’ve put together. My boss just handed me 7 servers, a debian 7 iso and told me to get OpenStack up and running. Needless to say my hair was thinning quite quickly till I stumbled onto your guides.

    I have run into something that needs clarification though:
    After creating the cinder database you say
    “Then we populate the database (on node only), and restart the services:”
    Which node are you referring to here? I’ve made the assumption that it should be on my first controller node but thought I’d ask in-case it should have been on one of the storage nodes.

    Cheers,

    Phillip

    Reply
  7. Arjun

    HI , i am unable to load my downloaded image , root@icehouse1:~# glance image-create –name cirros –is-public=true –disk-format=qcow2 –container-format=bare < cirros-0.3.1-x86_64-disk.img
    Request returned failure status.
    500 Internal Server Error
    Failed to upload image e8f2d1bc-2808-4038-9d63-b1698e626d1f
    (HTTP 500)
    root@icehouse1:~# glance image-list
    +—-+——+————-+——————+——+——–+
    | ID | Name | Disk Format | Container Format | Size | Status |
    +—-+——+————-+——————+——+——–+
    +—-+——+————-+——————+——+——–+
    root@icehouse1:~#

    Reply
      1. Arjun

        Hi Brian,
        thanks for writing back ,
        I dont know , but from both node i am unable , to upload the raw file as well and image , i have gone through all config and went well , but now again facing issue in loading raw file , here is the log you requested , please guide through this
        2014-12-12 16:33:58.981 1544 ERROR glance.api.v1.upload_utils [f609ef75-b1e2-438a-a324-51f3ef20d893 – – – – -] Failed to upload image f355065d-6a79-40ba-a8c4-a6c4c70917fd
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py”, line 105, in upload_data_to_store
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils store)
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/store/__init__.py”, line 382, in store_add_to_backend
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/store/rbd.py”, line 319, in add
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils with rados.Rados(conffile=self.conf_file, rados_id=self.user) as conn:
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/rados.py”, line 256, in __enter__
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils self.connect()
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/rados.py”, line 419, in connect
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils raise make_ex(ret, “error calling connect”)
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils ObjectNotFound: error calling connect
        2014-12-12 16:33:58.981 1544 TRACE glance.api.v1.upload_utils

        Thanks
        Arjun

        Reply
        1. Brian Seltzer Post author

          The error seems to suggest that glance can’t talk to ceph. I’d suggest that you verify that you’ve got a health ceph cluster, created a ceph pool (called ‘datastore’ in the article), created a ceph user (called ‘client.icehouse’ in the article), granted the user rights to the datastore, and configured the ceph client on the controllers using the client key and the configuration settings.

          Reply
          1. James

            I am having the same issue although controllers can talk to ceph

            i have created object fine from controller nodes to ceph nodes

            Any ideas

            registry.log
            2015-02-03 11:05:44.370 8600 DEBUG glance.api.policy [-] Loaded policy rules: {u’get_task’: ‘@’, u’get_image_location’: ‘@’, u’add_image’: ‘@’, u’modify_image’: ‘@’, u’manage_image_cache’: ‘role:admin’, u’delete_member’: ‘@’, u’get_images’: ‘@’, u’delete_image’: ‘@’, u’publicize_image’: ‘@’, u’get_member’: ‘@’, u’add_member’: ‘@’, u’set_image_location’: ‘@’, u’get_image’: ‘@’, u’modify_member’: ‘@’, u’context_is_admin’: ‘role:admin’, u’upload_image’: ‘@’, u’modify_task’: ‘@’, u’get_members’: ‘@’, u’get_tasks’: ‘@’, u’add_task’: ‘@’, u’default’: ‘@’, u’delete_image_location’: ‘@’, u’copy_from’: ‘@’, u’download_image’: ‘@’} load_rules /usr/lib/python2.7/dist-packages/glance/api/policy.py:85
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Matched OPTIONS / __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Route path: ‘/’, defaults: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Match dict: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
            2015-02-03 11:05:44.396 8600 INFO glance.registry.api.v1.images [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Returning image list
            2015-02-03 11:05:44.397 8600 INFO glance.wsgi.server [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] 172.16.1.11 – – [03/Feb/2015 11:05:44] “OPTIONS / HTTP/1.0” 200 222 0.027911
            api.log
            2015-02-03 11:05:44.370 8600 DEBUG glance.api.policy [-] Loaded policy rules: {u’get_task’: ‘@’, u’get_image_location’: ‘@’, u’add_image’: ‘@’, u’modify_image’: ‘@’, u’manage_image_cache’: ‘role:admin’, u’delete_member’: ‘@’, u’get_images’: ‘@’, u’delete_image’: ‘@’, u’publicize_image’: ‘@’, u’get_member’: ‘@’, u’add_member’: ‘@’, u’set_image_location’: ‘@’, u’get_image’: ‘@’, u’modify_member’: ‘@’, u’context_is_admin’: ‘role:admin’, u’upload_image’: ‘@’, u’modify_task’: ‘@’, u’get_members’: ‘@’, u’get_tasks’: ‘@’, u’add_task’: ‘@’, u’default’: ‘@’, u’delete_image_location’: ‘@’, u’copy_from’: ‘@’, u’download_image’: ‘@’} load_rules /usr/lib/python2.7/dist-packages/glance/api/policy.py:85
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Matched OPTIONS / __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Route path: ‘/’, defaults: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
            2015-02-03 11:05:44.371 8600 DEBUG routes.middleware [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Match dict: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
            2015-02-03 11:05:44.396 8600 INFO glance.registry.api.v1.images [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] Returning image list
            2015-02-03 11:05:44.397 8600 INFO glance.wsgi.server [cb69d83f-d0e2-48f9-bff9-e20da75aabbb – – – – -] 172.16.1.11 – – [03/Feb/2015 11:05:44] “OPTIONS / HTTP/1.0” 200 222 0.027911

      2. MIKE

        Hi Brain, I am also facing the same problem as Mr. Arjun and I also check the logs, Not sure how I can fix it. I have share with you because know might know the error as well as the solution for fixing it.
        2015-07-10 09:42:09.153 8156 ERROR glance.api.v1.upload_utils [6fb32eaa-9cbb-40e6-8f15-f8822c7af254 – – – – -] Failed to upload image a5f831e2-f19b-4e07-ae29-0fb02708b81e
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py”, line 105, in upload_data_to_store
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils store)
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/store/__init__.py”, line 385, in store_add_to_backend
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/glance/store/rbd.py”, line 325, in add
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils with rados.Rados(conffile=self.conf_file, rados_id=self.user) as conn:
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/rados.py”, line 257, in __enter__
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils self.connect()
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils File “/usr/lib/python2.7/dist-packages/rados.py”, line 417, in connect
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils raise make_ex(ret, “error calling connect”)
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils ObjectNotFound: error calling connect
        2015-07-10 09:42:09.153 8156 TRACE glance.api.v1.upload_utils

        Please help me to fix this problem.
        MIKE

        Reply
          1. MIKE

            No, Even I disable the Authentication I still getting this error and also I would like to inform you that I use HA, DB and other controller node services in 1server (controller01). Ceph are use in 3 different servers.

            I check the $ ceph -w on ceph its shows:

            root@compute01:~# ceph -w
            2015-07-13 11:18:38.926951 7ff5b0708700 0 — :/1009828 >> 172.20.3.15:6789/0 pipe(0x7ff5ac026050 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7ff5ac023240).fault
            2015-07-13 11:18:41.927069 7ff5b0607700 0 — :/1009828 >> 172.20.3.16:6789/0 pipe(0x7ff5a0000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7ff5a0004ea0).fault
            2015-07-13 11:18:44.927335 7ff5b0708700 0 — :/1009828 >> 172.20.3.15:6789/0 pipe(0x7ff5a00080e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7ff5a000c380).fault
            cluster 2ccc7c5c-8cd3-4264-be57-8cc934554a87
            health HEALTH_OK
            monmap e1: 1 mons at {compute01=172.20.3.14:6789/0}
            election epoch 1, quorum 0 compute01
            osdmap e26: 3 osds: 3 up, 3 in
            pgmap v615: 214 pgs, 2 pools, 0 bytes data, 0 objects
            107 MB used, 821 GB / 821 GB avail
            214 active+clean

            2015-07-13 11:17:32.197443 mon.0 [INF] pgmap v615: 214 pgs: 214 active+clean; 0 bytes data, 107 MB used, 821 GB / 821 GB avail

            Regards,
            MIKE

          2. Brian Seltzer Post author

            I just setup a stack with Ceph and here’s what I had to do the get Glance working:

            1) install ceph-common and python-ceph on the glance server
            2) Copy the /etc/ceph/ceph.conf from a ceph server to the glance server
            3) Copy the client keyring file to the /etc/ceph on the glance server
            4) modify /etc/glance/glance-api.conf as below:

            prior to Juno:

            [DEFAULT]
            default_store=rbd
            rbd_user=cephuser #ceph user name
            rbd_pool=datastore #ceph pool name
            show_image_direct_url=True

            Juno

            [DEFAULT]

            default_store = rbd

            [glance_store]
            stores = rbd
            rbd_store_pool = datastore # ceph pool name
            rbd_store_user = cephuser # ceph user name
            rbd_store_ceph_conf = /etc/ceph/ceph.conf
            rbd_store_chunk_size = 8

  8. henry

    Hi,
    Having the same problem as Arjun, unable to upload image.

    Request returned failure status.
    500 Internal Server Error
    Failed to upload image df2ec32d-0a21-4991-a3e2-f07e1be52a4d
    (HTTP 500)

    On glance api.log I get:
    015-02-03 16:34:53.411 29729 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: OPTIONS / Accept: process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:44
    2015-02-03 16:34:53.411 29729 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:57
    2015-02-03 16:34:53.412 29729 DEBUG glance.api.middleware.version_negotiation [-] Unknown version. Returning version choices. process_request /usr/lib/python2.7/dist-packages/glance/api/middleware/version_negotiation.py:64
    2015-02-03 16:34:53.412 29729 INFO glance.wsgi.server [-] 172.16.1.11 – – [03/Feb/2015 16:34:53] “OPTIONS / HTTP/1.0” 300 697 0.0017

    On glance-registry log:
    2015-02-03 16:36:50.273 29712 DEBUG routes.middleware [6da6988d-5b6b-49c9-8841-e4329a914722 – – – – -] Matched OPTIONS / __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
    2015-02-03 16:36:50.273 29712 DEBUG routes.middleware [6da6988d-5b6b-49c9-8841-e4329a914722 – – – – -] Route path: ‘/’, defaults: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
    2015-02-03 16:36:50.273 29712 DEBUG routes.middleware [6da6988d-5b6b-49c9-8841-e4329a914722 – – – – -] Match dict: {‘action’: u’index’, ‘controller’: } __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
    2015-02-03 16:36:50.298 29712 INFO glance.registry.api.v1.images [6da6988d-5b6b-49c9-8841-e4329a914722 – – – – -] Returning image list
    2015-02-03 16:36:50.299 29712 INFO glance.wsgi.server [6da6988d-5b6b-49c9-8841-e4329a914722 – – – – -] 172.16.1.23 – – [03/Feb/2015 16:36:50] “OPTIONS / HTTP/1.0” 200 222 0.027793

    THe ceph cluster returns the following:

    2015-02-03 16:38:56.726959 7f8880293700 0 — :/1029938 >> 192.168.1.102:6789/0 pipe(0x7f887c0203b0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f887c020640).fault
    2015-02-03 16:38:59.727050 7f8880192700 0 — :/1029938 >> 192.168.1.101:6789/0 pipe(0x7f8870000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f8870000e90).fault
    2015-02-03 16:39:02.727379 7f8880293700 0 — :/1029938 >> 192.168.1.102:6789/0 pipe(0x7f8870003010 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f88700032a0).fault
    cluster 4d5bf6e0-d1a7-4375-9442-9eaac0e05a4a
    health HEALTH_OK
    monmap e1: 1 mons at {ceph1=192.168.1.100:6789/0}, election epoch 2, quorum 0 ceph1
    osdmap e20: 3 osds: 3 up, 3 in
    pgmap v190: 342 pgs, 3 pools, 60 bytes data, 8 objects
    108 MB used, 400 GB / 400 GB avail
    342 active+clean

    Do you have any ideas what might be the problem – thanks again for the tutorial

    Reply
    1. Brian Seltzer Post author

      I don’t really see any errors in your log snippet. Just debug information. That makes me wonder if your upload request is even reaching the glance api. In other words, is your load balancer configuration correct?

      Reply
  9. James

    Hi Brian,

    Thanks for the tutorial well presented,

    Im having an issue with glance where i am trying to see glance image-list and uploading an image to glance

    But I am receiving the following:

    root@controller1:~# glance image-list
    Request returned failure status.
    503 Service Unavailable
    No server is available to handle this request.

    (HTTP 503)
    root@controller1:~# glance image-create –name cirros –is-public=true –disk-format=qcow2 –container-format=bare < cirros-0.3.1-x86_64-disk.img
    Error communicating with http://172.16.1.90:9292 [Errno 32] Broken pipe

    api.log

    2015-02-06 12:26:03.126 19826 DEBUG glance.store [-] Attempting to import store glance.store.rbd.Store _get_store_class /usr/lib/python2.7/dist-packages/glance/store/__init__.py:168
    2015-02-06 12:26:03.131 19826 DEBUG glance.store [-] Registering store with schemes (‘rbd’,) create_stores /usr/lib/python2.7/dist-packages/glance/store/__init__.py:210
    2015-02-06 12:26:03.131 19826 DEBUG glance.store.base [-] Late loading location class glance.store.rbd.StoreLocation get_store_location_class /usr/lib/python2.7/dist-packages/glance/store/base.py:80
    2015-02-06 12:26:03.131 19826 DEBUG glance.store.location [-] Registering scheme rbd with {‘store_class’: , ‘location_class’: } register_scheme_map /usr/lib/python2.7/dist-packages/glance/store/location.py:86

    Reply
    1. Brian Seltzer Post author

      Those debug messages are not errors. The API log rarely shows errors. Are there any errors in the glance registry log? Also I would suspect your load balancer configuration…

      Reply
      1. James

        Thanks Brian No no errors in regisrty.log

        I will look again at the Load balancer thanks for the info,

        James

        Reply
      2. James

        Hi Brian,

        proxy.cfg looks fine however its throwing up alert :

        root@proxy1:~# service haproxy restart
        * Restarting haproxy haproxy
        [WARNING] 036/134031 (29518) : config : ‘option httplog’ not usable with proxy ‘galera’ (needs ‘mode http’). Falling back to ‘option tcplog’.
        …done.
        root@proxy1:~#

        listen glance-api 172.16.1.90:9292
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server controller2 172.16.1.18:9292 check inter 2000 rise 2 fall 5
        server controller1 172.16.1.22:9292 check inter 2000 rise 2 fall 5

        listen glance-registry 172.16.1.90:9191
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server controller2 172.16.1.18:9191 check inter 2000 rise 2 fall 5
        server controller1 172.16.1.22:9191 check inter 2000 rise 2 fall 5

        listen galera 172.16.1.90:3306
        balance source
        mode tcp
        option tcpka
        option mysql-check user haproxy
        server mysql1 172.16.1.21:3306 check weight 1
        server mysql2 172.16.1.14:3306 check weight 1

        Reply
        1. bgyako

          We had a lot of broken pipe errors as well, it was related to network traffic. Make sure you are not losing any packets, also double check your MTU (try setting it at 1400).
          Also make sure Glance api is running on controllers.

          Reply
  10. MIKE

    Dear Brian,

    I found the problem in my case (hopefully). Its because of time different. Ubuntu server and SQL server time is same and correct but while uploading Image it take some other time which is pasted already thus make the upload expired.

    http://1drv.ms/1CRGA6X

    You can check the image from attachment to get more details and hope you can provide me any solution for this case.

    Waiting for your reply.

    Regards,
    MIKE

    Reply
      1. MIKE

        can you tell me how to fix this problem because I didn’t get any solution. Also I didn’t know why image take expired time. Please advice me because NTP is already installed and running from start of this project.

        Waiting for your advice

        Reply
        1. Brian Seltzer Post author

          Oh I thought that you meant it was fixed. Yeah, I don’t think the date has anything to do with it. Can you post (from your glance server) your ceph.conf, your glance-api.conf, and the output of the command: ceph osd lspools

          Reply
    1. Brian Seltzer Post author

      IT looks like you need the following in the [default] section of your glance-api.conf:

      default_store = rbd
      rbd_store_user = icehouse
      rbd_store_pool = datastore
      rbd_store_chunk_size = 8

      after fixing this, restart the service:

      service openstack-glance-api restart

      Reply
      1. 1mike

        Hi Brian,
        Appreciated your support and due to your advice my current problem finished. The default config of Glance-api.conf have:
        Rbd_user=
        Rbd_user=
        Rbd_chunk_size = 8
        So I did fill the information after the equal sign and got error. But after correcting myself according to Brian advice its working now.
        Now I reach till First Compute node. Hopefully I complete this job within this week.
        Dear followers: please follow these documents line by line and word by word to make sure you end up without error.
        Many Thanks once again.
        Regards,
        MIKE

        Reply
  11. 1mike

    Hi Brain,

    Whats the username and password for Ubuntu 14.04 image.

    Reply
    1. Brian Seltzer Post author

      the username is ubuntu

      Password authentication is disabled. cloud-init must deploy your ssh key so that you can logon with key-based authentication.

      Reply
  12. bgyako

    Hi Brian, One of my nodes had a disk failure, I rebuild the node from scratch and tried to add it into the cluster. It’s failing to gather the keys with error “ceph auth get-or-create for keytype admin returned 22”
    Error EINVAL: key for client.admin exists but cap mds does not match. Any idea what the issue is? I can’t build OSD without gathering keys, get error [ERROR ] RuntimeError: bootstrap-osd keyring not found; run ‘gatherkeys’ when building osd. This was a 3 node cluster, because 1 node faild it is in a degraded state.

    Reply

Leave a Reply