Ceph Object Gateway

Oct 13, 2015 Written by M Vijayakanth Platform Developement Engineer

Ceph Object Gateway is an object storage interface built on top of librados to provide applications with a RESTful gateway to Ceph Storage Clusters. Ceph Object Storage supports two interfaces:

S3-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the Amazon S3 RESTful API.

Swift-compatible: Provides object storage functionality with an interface that is compatible with a large subset of the OpenStack Swift API.

My ceph cluster setup

mon and gateway	- mon-server(
osd1 and osd2 	 - osd1(
osd3 			  - osd2(
OS				 - Ubuntu Trusty(14.04.2 LTS)

To run the Ceph object gateway service on Ubuntu 14.04 (Trusty), you should have a running Ceph cluster and the gateway host should have access to storage and public networks.

In my case, I've done the follwing in mon-server(


On Ubuntu Ubuntu 14.04, multiverse needs to be enabled in the package resource list file

uncomment the following lines in /etc/apt/sources.list:

# deb trusty multiverse
# deb-src trusty multiverse
# deb trusty-updates multiverse
# deb-src trusty-updates multiverse

Update the package resource list:

$ sudo apt-get update

Install Apache and FastCGI:

$ sudo apt-get install apache2 libapache2-mod-fastcgi

Configure APACHE

Add a line for the ServerName in the /etc/apache2/apache2.conf. Provide the fully qualified domain name of the server machine (e.g., hostname -f):

ServerName mon-server

Enable the URL rewrite modules for Apache and FastCGI Execute the following:

$ sudo a2enmod rewrite
$ sudo a2enmod fastcgi

Restart Apache service

$sudo service apache2 start


Ceph Object Storage services use the Ceph Object Gateway daemon (radosgw) to enable the gateway.

To install the Ceph Object Gateway daemon on the gateway host, execute the following:

$ sudo apt-get install radosgw

Once you have installed the Ceph Object Gateway packages, the next step is to configure your Ceph Object Gateway. There are two approaches: simple and FEDERATED. I used simple in my system

Simple: A simple Ceph Object Gateway configuration implies that you are running a Ceph Object Storage service in a single data center. So you can configure the Ceph Object Gateway without regard to regions and zones.

The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:

1. A name for the gateway instance. We use 'admin' in this guide.
2. A storage cluster user name with appropriate permissions in a keyring.
3. Pools to store its data.
4. A data directory for the gateway instance.
5. An instance entry in the Ceph Configuration file.
6. A configuration file for the web server to interact with FastCGI.

The configuration steps are as follows:

Execute the following steps on the admin node of your cluster:

Create a keyring for the gateway:

$ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring

Generate a Ceph Object Gateway user name and key for each instance. For exemplary purposes, we will use the name ‘admin’ after client.radosgw:

$ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.admin --gen-key

Add capabilities to the key:

$ sudo ceph-authtool -n client.radosgw.admin --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:

$ sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.admin -i /etc/ceph/ceph.client.radosgw.keyring

Distribute the keyring to the gateway host:

$ sudo scp /etc/ceph/ceph.client.radosgw.keyring  USERNAME@GATEWAY_IP:/home/ceph
$ ssh USERNAME@GATEWAY_IP 'sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring'

NOTE The last step is optional if admin node is the gateway host.


If pools already exist, no problem. If not, create all the pools listed below

$ ceph osd pool create .rgw.buckets 16 16


NOTE if write permission is given, Ceph Object Gateway will create pools automatically.

NOTE When adding a large number of pools, it may take some time for your cluster to return to a active + clean state.

When you have completed this step, execute the following to ensure that you have created all of the foregoing pools:

$ rados lspools


Add the Ceph Object Gateway configuration to your Ceph Configuration file in admin node. The Ceph Object Gateway configuration requires you to identify the Ceph Object Gateway instance. Then, you must specify the host name where you installed the Ceph Object Gateway daemon, a keyring (for use with cephx), the socket path for FastCGI and a log file.

Append the following configuration to /etc/ceph/ceph.conf in your admin node:

host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = 	/var/run/ceph/ceph.radosgw.admin.fastcgi.sock
log file = /var/log/radosgw/client.radosgw.admin.log

NOTE Here, {hostname} is the short hostname (output of command hostname -s) of the node that is going to provide the gateway service i.e, the gateway host.

NOTE The [client.radosgw.admin] portion of the gateway instance identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (i.e., radosgw).


$ ceph-deploy --overwrite-conf config pull {gateway_hostname}
$ ceph-deploy --overwrite-conf config push osd1 osd2


$ sudo scp /etc/ceph/ceph.client.admin.keyring  USERNAME@GATEWAY_IP:/home/USERNAME
$ ssh USERNAME@GATEWAY_IP 'sudo mv ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring'

NOTE The above step need not be executed if admin node is the gateway host


The wrapper script provides the interface between the webserver and the radosgw process. This script needs to be in a web accessible location and should be executable.

Execute the following steps on the gateway host:

Create the script:

$ sudo vi /var/www/html/s3gw.fcgi Add the following content to the script:

exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.admin

Provide execute permissions to the script:

Change file permission

$ sudo chmod +x /var/www/html/s3gw.fcgi

Create Data Directory

$ sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.admin

Start rados gateway service

$ sudo /etc/init.d/radosgw start


$ sudo vi /etc/apache2/sites-available/rgw.conf

Add the following contents to the file:

FastCgiExternalServer /var/www/html/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.admin.fastcgi.sock

<VirtualHost *:80>
ServerName localhost
DocumentRoot /var/www/html

ErrorLog /var/log/apache2/rgw_error.log
CustomLog /var/log/apache2/rgw_access.log combined

# LogLevel debug

RewriteEngine On

RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) /s3gw.fcgi?page=$1&params=$2&%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]

<IfModule mod_fastcgi.c>
	<Directory /var/www/html>
	Options +ExecCGI
	AllowOverride All
	SetHandler fastcgi-script
	Order allow,deny
	Allow from all
	AuthBasicAuthoritative Off

AllowEncodedSlashes On
ServerSignature Off


Disable the default site:

$ sudo a2dissite 000-default

Enable the configuration file

$ sudo a2ensite rgw.conf

Restart apache2 service

$ sudo service apache2 restart



$ sudo radosgw-admin user create --uid="testuser" --display-name="First User"

The output of the command will be something like the following:

{"user_id": "testuser",
"display_name": "First User",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{ "user": "testuser",
"access_key": "I0PJDPCIYZ665MW88W9R",
"secret_key": 	"dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"}],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"user_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"temp_url_keys": []}

NOTE The values of keys->access_key and keys->secret_key are needed for access validation.


install the python-boto package

$ sudo apt-get install python-boto

Create the Python script:

$ nano

import boto
import boto.s3.connection
access_key = 'YOUR_ACCESS_KEY'
secret_key = 'YOUR_SECRET_KEY'
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = '{FQDN}',
calling_format = boto.s3.connection.OrdinaryCallingFormat(),)
bucket = conn.create_bucket('my-new-bucket')
for bucket in conn.get_all_buckets():
	print "{name}\t{created}".format(
		name =,
		created = bucket.creation_date,

Run the script:

$ python

The output will be something like the following:

my-new-bucket 2015-02-16T17:09:10.000Z

Test in ruby language

To test ceph-gateway, we have use rubygem s3. Source code is in

Edit to point to your gateway_host

Revert installation

There are useful commands to purge the Ceph gateway nstallation and configuration from every node so that one can start over again from a clean state.

This will remove Ceph configuration and keys

ceph-deploy purgedata mon-server

This will also remove Ceph packages

ceph-deploy purge mon-server

IF you received the below error when you attempt to install radosgw again client.radosgw.admin exists but key does not match

Execute this to fix the error ceph auth del client.radosgw.gateway