Ceph RADOS Gateway Multi-Site replication with Active Directory accounts

Objective:

Active Directory users in the east coast region will be able to create Containers and Objects in object storage which will be replicated and consumed by Active Directory users in the west coast region and vice a versa.

Prerequisites:

The Following is assumed for this deployment:

  1. Two OpenStack deployments: East Coast and West Coast Datacenters respectively
  2. Active Directory Domain Controllers replicated in each site
  3. Ceph clusters configured for each site
  4. Keystone deployments independent from each other
  5. Keystone using LDAP (Active Directory) for authentication
  6. Keystone utilizing Fernet tokens

Software used for demonstration

  1. Red Hat OpenStack Platform (RH OSP) 11 (based on OpenStack Ocata release)
  2. Red Hat Enterprise Linux (RHEL) 7.4
  3. 2 RH OSP Director 11 virtual machines (east and west respectively)
  4. Ceph 2.2

Deployment:

To deploy the OpenStack environments for each site, I deployed RH OSP 11 Director at each site (director11east.lab.lan & director11west.lab.lan) on virtual machines consisting of 24 GB RAM, 8 vCPUs, 60GB Disk and 2 NICs (public and provisioning) each.

I used the following triple-o templates and custom templates to create the two overcloud
deployment changing only the hostnames, IPs, subnets and VLANs between east and west
overclouds: east templates & west templates

Once you have deployed both OpenStack sites (east and west respectively), reference my ad-post.sh script in the templates above to create the Keystone Domain which will point to Active Directory for authentication.  Within this newly created KeystoneV3 Domain, create a new tenant called “east” in both the East and West OpenStack environments.

IMPORTANT:  Even though we have created a tenant in both OpenStack environment with the same name, the Unique ID associated with that tenant name is different for each cluster.  We need to make the West Coast OpenStack Cluster have the same ID for the “east” tenant to match what the East Coast OpenStack Cluster has. 

On one of the OpenStack controllers in the East Coast OpenStack Environment, use the following command to obtain the ID associated with the tenant named “east”

1
source overcloudrc.v3
1
openstack project show list --domain LAB

On one controller in the West Coast OpenStack Environment, run the following commands as root. RUN THIS ONLY ONCE AND FROM ONLY ONE CONTROLLER. IT WILL REPLICATE TO THE OTHERS

1
mysql keystone

# View the field you are about to change

1
select * from project where name='east';

# change the existing ID to the ID from the East Coast OpenStack Environment
# update id in DB with project ID from east overcloud

1
update project set id='ID_GATHERED_FROM_EAST_OPENSTACK_ENV' where name='east';

East Coast Cluster

Once each OpenStack site has been deployed, we need to configure the Ceph RADOS Gateway REALMs.   We will start with the East Coast OpenStack Environment and make it the Master REALM.  This script is intended for new Ceph Clusters with absolutely ZERO data on it.  THIS WILL DESTROY EXISTING CEPH POOLS AND CREATE NEW ONES!!!!   The PG numbers I used are based on my environment, but you can generate you own specific settings by using the Red Hat Ceph Placement Groups calculator: https://access.redhat.com/labsinfo/cephpgc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
east_pub_endpoint=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
realm_name=redhat

# Create a name for your zonegroup. This is global
zonegroup=us

# Create a name for your zone. This is local to each ceph deployment
rgw_zone=us-east

# AD Test User
ad_test_user=kholden

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm create --rgw-realm=$realm_name --default
radosgw-admin zonegroup create --rgw-zonegroup=$zonegroup --endpoints=$east_pub_endpoint --rgw-realm=$realm_name --master --default
radosgw-admin zone create --rgw-zonegroup=$zonegroup --rgw-zone=$rgw_zone --master --default --endpoints=$east_pub_endpoint
radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

# create new pools from ceph us.east pools

ceph osd pool create us-east.rgw.intent-log 16
ceph osd pool set us-east.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.log 16
ceph osd pool set us-east.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.data 128
ceph osd pool set us-east.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.extra 16
ceph osd pool set us-east.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.index 16
ceph osd pool set us-east.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.control 16
ceph osd pool set us-east.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.gc 16
ceph osd pool set us-east.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.data.root 16
ceph osd pool set us-east.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.usage 16
ceph osd pool set us-east.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users 16
ceph osd pool set us-east.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.email 16
ceph osd pool set us-east.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.swift 16
ceph osd pool set us-east.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.uid 16
ceph osd pool set us-east.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.meta 16
ceph osd pool set us-east.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

radosgw-admin period update --commit
radosgw-admin period update --commit
radosgw-admin zone delete --rgw-zone=default
radosgw-admin period update --commit
radosgw-admin zonegroup delete --rgw-zonegroup=default
radosgw-admin period update --commit
radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system
access_key=$(radosgw-admin user info --uid="synchronization-user"|grep access_key|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')
secret_key=$(radosgw-admin user info --uid="synchronization-user"|grep secret|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')

radosgw-admin zone modify --rgw-zone=us-east --access-key=$access_key --secret=$secret_key
radosgw-admin period update --commit
cp /etc/ceph/ceph.conf ~/ceph.conf_backup
echo "rgw_zone=us-east" >> /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring

systemctl stop ceph-radosgw.target
sleep 10
systemctl start ceph-radosgw.target

# Check RGW sync status
radosgw-admin sync status

# Create an openstack credentials file with your AD, Domain, and Tenant information

cp eastrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’
source ken
openstack container list

Next we need to configure the West OpenStack environment to be part of the Ceph RADOS Gateway REALM (I called the REALM “redhat”) that we created in the previous step.

IMPORTANT:  For the master realm setup, I created both an Access Key and Secret Key to be used for authentication of the REALM.  YOU MUST USE THE ACCESS AND SECRET KEYS USED IN THE PREVIOUS STEP FOR THE SCRIPT BELOW.  DO NOT USE THE KEY’S I HARDCODED BELOW

West Coast Ceph Cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
EAST_PUB_ENDPOINT=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
REALM_NAME=redhat

# Create a name for your zonegroup. This is global
ZONE_GROUP=us

# Create a name for your zone. This is local to each ceph deployment
RGW_ZONE=us-east

# AD Test User
AD_TEST_USER=kholden

# AD Test User Password
AD_TEST_USER_PASSWORD

# AD Domain Name
AD_DOMAIN=LAB

# OpenStack Project Name
PROJECT_NAME=east

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin realm default --rgw-realm=redhat
radosgw-admin period pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr --endpoints=http://192.168.15.120:8080
radosgw-admin zone delete --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

ceph osd pool create us-west.rgw.intent-log 16
ceph osd pool set us-west.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.log 16
ceph osd pool set us-west.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.data 128
ceph osd pool set us-west.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.extra 16
ceph osd pool set us-west.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.index 16
ceph osd pool set us-west.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.control 16
ceph osd pool set us-west.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.gc 16
ceph osd pool set us-west.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.data.root 16
ceph osd pool set us-west.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.usage 16
ceph osd pool set us-west.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users 16
ceph osd pool set us-west.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.email 16
ceph osd pool set us-west.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.swift 16
ceph osd pool set us-west.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.uid 16
ceph osd pool set us-west.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.meta 16
ceph osd pool set us-west.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

cp /etc/ceph/ceph.conf ~/
echo "rgw_zone=us-west" >> /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring
radosgw-admin period update --commit

 

Run 

1
systemctl stop ceph-radosgw.target

# Wait until you see the following message
# Broadcast message from systemd-journald@east-controller1.lab.lan (Fri 2017-08-25 11:57:53 UTC):
#
# haproxy[35919]: proxy ceph_rgw has no server available!

Then run 

1
systemctl start ceph-radosgw.target

# Create openstack credentials file with AD user, Domain, and Tenant information:
cp westrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’

logout and back in to clear ENV variables
source ken
openstack container list

You should now be able to create containers and objects in the east coast datacenter and then see them replicate to the west coast datacenter.  You should be able to also create containers and objects in the west data center and see them replicate to the east datacenter

Separate Ceph from existing Director-based deployment

Deploying Ceph with OpenStack Platform Director is very convenient, but there are times when it’s simplicity isn’t enough for more advanced installations.  Fortunately, it is possible to decouple the Ceph installation from the OpenStack control plane and Director management which I will detail below.

  1. Deploy OSP to include at least 1 controller, compute and Ceph storage node
  2. Deploy at least one new server to take over Ceph monitor role
  3. Enable new Ceph monitor
  4. Disable existing Ceph Monitor role from OSP Controllers
  5. Set Ceph storage nodes to maintenance in Ironic
  6. Delete Ceph storage nodes from ironic
  7. re-run openstack overcloud deploy pointing to external ceph yaml and setting ceph scale out = 0

Initial OSP Deploy script:

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash
--ntp-server 192.168.1.250 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--control-scale 3 \
--compute-scale 1 \
--ceph-storage-scale 3

Once deployed, I run an openstack overcloud update stack to ensure Overcloud is updated to the latest RPMs within it’s Major version (i.e. If deploying OSP 8, update to the latest RPMs available to OSP 8)

OSP Update Script
This will update existing OSP deploy to the latest RPMs

1
2
3
4
5
6
7
#!/bin/bash
openstack overcloud update stack overcloud -i \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml

Deploy a new server(s) to take over Ceph monitor role from OSP Controllers.
I used Director to deploy a baremetal server with the default baremetal Nova flavor using the following command:

1
2
ctrlplane_net=$(neutron net-list | grep ctrl | awk '{print $2;}')
openstack server create cloudbox4 --flavor=baremetal --nic net-id=$ctrlplane_net --image=overcloud-full --key-name=default

New OSP Deploy script (removing storage-environment.yaml and including puppet-ceph-external.yaml and setting ceph-storage-scale 0)

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
openstack overcloud deploy --templates \
--ntp-server 192.168.1.250 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml \
-e /home/stack/templates/ceph-external.yaml \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--control-scale 3 \
--compute-scale 1 \
--ceph-storage-scale 0

New OSP Update script

1
2
3
4
5
6
7
8
#!/bin/bash
openstack overcloud update stack overcloud -i \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml \
-e /home/stack/templates/ceph-external.yaml

Lab

To learn OpenStack, you need to build OpenStack… regularly.  Having a physical lab is imperative as all-virtual labs just don’t cut it.  I started my lab journey on eBay where you can find previous generations of server gear cheap.  I was most familiar with HP Proliant series servers so I went with the HP Proliant DL360 G6 and G7 generations.   Servers are all pretty much the same these days but here are a few things to keep in mind:

  1.  Memory is expensive on it’s own so buy servers with the amount of memory you need already included.  You can save a lot of money this way
  2. Stick with the same hardware brand and model.  This makes buying spare parts (Drives, memory, power supplies, raid batteries, etc.) much easier
  3. Choose a server brand that still offers free firmware downloads.   HP, for example no longer provides firmware / bios for their servers unless you have an active support contract for the particular model you are trying to obtain software for.   At the time of this writing, I believe Dell does not have that requirement.
  4. Network switch.  This is the heart and veins of your environment.  Don’t skimp on it.  I have always had working knowledge of Cisco so I chose to stick with it.  I wanted to test out OpenStack Neutron plug-in capabilities so i opted for the Cisco Nexus 3048.  You can find them used on eBay for around $800.00 with 1-2 year support.  For me, having an enterprise-class switch that I see I work with in the field has made testing multiple lab scenarios very easy.
  5. CPUs:  choose power-efficiency over performance.  I chose the opposite and I pay for it every month when the power bill comes.   I do recommend going with Dual-proc though as its much cheaper to buy used servers with dual-processors than add them later.
  6. Cabling.  Don’t skimp on cabling.  You can find multi-color packs on amazon for cheap. At first, I decided to make my own cables.   It was a pain and problematic in the long run plus they were only CAT5e.  I found these 3-foot CAT6 patch cables on Amazon at 10 x $14.99.
  7. Rack enclosure.  I use an XRackPro2 12U Server Rack Noise Reduction Rackmount Enclosure Cabinet.  In my opinion, it is worth the money for a home lab as it’s quiet, has built in cooling, and most importantly looks awesome.
  8. neatness.  Spend the time with cable management, labeling and switch port descriptions.  It will save you lots of time in the long run.
  9. IPMI.  Make sure that the server hardware you chose supports IPMI for lights out management.  If you plan on doing bare-metal provisioning (which you should be), you will need it.  Stick with the major server brands IMHO as they just work.  HP, Dell, Cisco, SuperMicro.  Avoid IBM like the plague (trust me on that one).
  10. Hard drives.  If you want to test Ceph and brag to your friends about your I/O performance, you are going to need a lot of them.  all drives are not created equal so if you have a mixture of 10k/15k single-port / dual-port / SAS / SATA, expect horrible performance.  Also, consumer SSDs don’t always negotiate the fastest speeds with servers making them actually slower than SAS drives (at least in HP Proliant servers.)  You can find used 15k dual-port SAS drives with sleds on Amazon for around $10.00/each for the 72GB models.  I don’t find that I need space.  I need performance.
  11. Make one server a utility hypervisor.  This way, you can build multiple versions of OpenStack Platform Director, Windows AD Domain Controllers, LDAP, bind, monitoring etc. all using VMs.  I am using one of my HP DL360 G7 with 128GBs of RAM.  I installed 4 x Samsung 850 EVO Pro 512GB SSDs in a RAID 10 array which I find to perform quite well for virtual machines.  I just use RHEL 7 and KVM with multiple NICs for bridges.