Keystone Optimization

Issue:

The default configuration of Keystone is not necessarily tuned for anything specific.  As with other components of OpenStack, Keystone tuning based on use case and size are required in order for it to perform well.  When integrating Keystone with large LDAP environments (10k users+), if not tuned properly, you can suffer slow logons, API lag and an incredibly slow Horizon Dashboard.

Objective:

To Improve OpenStack Keystone performance when integrating with LDAP / Active Directory.

Software and Hardware Used for testing:

  • Red Hat OpenStack Platform Version 12
  • Ceph 2.4
  • 8 HP DL360 G7
    • 3 controllers (2 x Intel x5620, 24GB RAM, SSDs for OS)
    • 4 Hyper-converged nodes (2 x Intel x5620, 24GB RAM, 7 x 72GB 15KSAS)
    • 1 Utility server (dual-x5620, 72 GBRAM, and 4 x SSD RAID-10)
      • Red Hat OpenStack Platform Director version 12 running as a VM
        • 8 vCPU, 24 GBs RAM, 100GB disk)
  • Microsoft Windows Server 2008 Enterprise Active Directory

Environment tested

Windows 2k8 Server Enterprise with 10000+ users and nested groups.  Red Hat OpenStack Platform Version 12 Hyper-converged Reference architecture with 3 x Active Controller cluster, 4 x node Ceph/Nova nodes, with full network isolation, SSL certificates for OpenStack Endpoints, Horizon GUI and LDAPs.   Also, I am using Fernet Tokens for keystone instead of UUID Tokens.  This is strongly recommended as it alleviates the burden of token persistence in a database amongst other reasons.

NOTE:  Red Hat OSP 12 uses a containerized control plane so instead of editing conf files and restarting the services, I am editing the template files uses to create the containers and then restarting the containers.   If you are not running a containerized based control plane, edit the conf files and restart the respective service.

Results

These results demonstrate an OSP 12 Director-based deployment of an OSP 12 Overcloud with the default Active Directory configurations as documented.  I then enable and tune user & auth pooling as well as implement user & group filtering.  Lastly, I enable memcached to be used by Keystone for caching tokens, catalogs, and roles.

Download the token_issue_script used for testing.

With defaults (pooling disabled, no LDAP filtering, and no caching layer)
time ./token_issue.sh 100
getting 100 tokens
real 2m43.052s
user 0m5.627s
sys 0m1.642s
With Keystone pooling configured and LDAP user & group filtering
time ./token_issue.sh 100
getting 100 tokens
real 0m27.178s
user 0m5.698s
sys 0m1.470s
With Keystone pooling configured, LDAP user & group filtering, and Keystone Caching
time ./token_issue.sh 100
getting 100 tokens
real 0m10.569s
user 0m5.559s
sys 0m1.465s

Custom Settings

Keystone Pooling

Check current Keystone Pooling configuration

# Check settings
cat << EOF > check_keystone_pooling.sh
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_pool
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_size
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_max
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_delay
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_timeout
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_lifetime
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_auth_pool
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_size
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_connection_lifetime
EOF
chmod +x check_keystone_pooling.sh
./check_keystone_pooling.sh

Configure and enable Keystone Pooling # THIS WILL RESTART KEYSTONE

# Configure Keystone Pooling
cat << EOF > set_keystone_pooling.sh
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_pool True
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_size 200
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_max 20
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_delay 0.1
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_timeout -1
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_lifetime 600
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_auth_pool True
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_size 1000
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_connection_lifetime 60
docker restart keystone
EOF
chmod +x set_keystone_pooling.sh
./set_keystone_pooling.sh

Results Pagination

Pagination is important in large LDAP environments as only a certain number of records will be returned by default.   This option defines the maximum number of results per page that keystone should request from the LDAP server when listing objects.  Add this to your keystone/domains/keystone.DOMAINNAME.conf file and restart Keystone

crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap page_size 2

To test the page_size, you can use ldapsearch with the following syntax.  Note that without specifying a page size, not all results will be returned.

$ ldapsearch -LLL -H ldap://ldapserver.domain.com -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' -w PASSWORD |grep sAMAccountName |wc -l
Size limit exceeded (4)
825

$ ldapsearch -LLL -H ldap://ldapserver.domain.com -E pr=1/noprompt -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' -w PASSWORD |grep sAMAccountName |wc -l
10052

User and Group Filtering

Here are examples of user and group filters that can be used in the [ldap] section of /etc/keystone/domains/keystone.YOUR_DOMAIN_NAME.yaml.

For the user_filter example, I am filtering out all users excepts for those who are either a member of OpenStack-Admins OR OpenStack-Users.

user_filter=(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))

For the group_filter example, I am filtering out all groups  except for those with the ObjectClass of Group AND with a group named OpenStack, OpenStack-Admins, OR OpenStack-Users

group_filter=(&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))

To test LDAP user and group filtering, use the following ldapsearch syntax

Members of group1 AND group2
ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))'|grep sAMAccountName
Members of group1 OR group2
$ ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))'|grep sAMAccountName
sAMAccountName: user1
sAMAccountName: user2
sAMAccountName: user3
sAMAccountName: user4
sAMAccountName: user5
List groups with names
$ ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))'|grep distinguishedName
distinguishedName: CN=OpenStack,OU=People,DC=lab,DC=lan
distinguishedName: CN=OpenStack-Admins,OU=People,DC=lab,DC=lan
distinguishedName: CN=OpenStack-Users,OU=People,DC=lab,DC=lan

Keystone Caching

Check if caching is enabled

# Run on each controller as root
cat << EOF > ~/check_keystone_cache.sh
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache enabled
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend_argument
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf catalog caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf domain_config caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf federation caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf revoke caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf role caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token cache_on_issue
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity cache_time
EOF
chmod +x ~/check_keystone_cache.sh
~/check_keystone_cache.sh

Set caching per on controllers # THIS WILL RESTART KEYSTONE

# Run on each controller as root
cat << EOF > ~/enable_keystone_cache.sh
API_IP=$(grep $HOSTNAME.internalapi /etc/hosts|awk '{print $1}')
echo $API_IP
systemctl enable memcached
systemctl start memcached
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache enabled true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend dogpile.cache.memcached
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend_argument url:$API_IP:11211
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf catalog caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf domain_config caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf federation caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf revoke caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf role caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token cache_on_issue true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity cache_time 600
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf eventlet_server admin_workers 72
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf eventlet_server public_workers 72
docker restart keystone
EOF
chmod +x ~/enable_keystone_cache.sh
~/enable_keystone_cache.sh

Verify memcached hits

You can run the following command from your controllers to watch cache hits and misses.  The hits should be increasing which means caching is working.

# Run on each controller
API_IP=$(grep $HOSTNAME.internalapi /etc/hosts|awk '{print $1}')
watch "memcached-tool $API_IP:11211 stats |grep -A2 get_hits"

Ceph RADOS Gateway Multi-Site replication with Active Directory accounts

Objective:

Active Directory users in the east coast region will be able to create Containers and Objects in object storage which will be replicated and consumed by Active Directory users in the west coast region and vice a versa.

Prerequisites:

The Following is assumed for this deployment:

  1. Two OpenStack deployments: East Coast and West Coast Datacenters respectively
  2. Active Directory Domain Controllers replicated in each site
  3. Ceph clusters configured for each site
  4. Keystone deployments independent from each other
  5. Keystone using LDAP (Active Directory) for authentication
  6. Keystone utilizing Fernet tokens

Software used for demonstration

  1. Red Hat OpenStack Platform (RH OSP) 11 (based on OpenStack Ocata release)
  2. Red Hat Enterprise Linux (RHEL) 7.4
  3. 2 RH OSP Director 11 virtual machines (east and west respectively)
  4. Ceph 2.2

Deployment:

To deploy the OpenStack environments for each site, I deployed RH OSP 11 Director at each site (director11east.lab.lan & director11west.lab.lan) on virtual machines consisting of 24 GB RAM, 8 vCPUs, 60GB Disk and 2 NICs (public and provisioning) each.

I used the following triple-o templates and custom templates to create the two overcloud
deployment changing only the hostnames, IPs, subnets and VLANs between east and west
overclouds: east templates & west templates

Once you have deployed both OpenStack sites (east and west respectively), reference my ad-post.sh script in the templates above to create the Keystone Domain which will point to Active Directory for authentication.  Within this newly created KeystoneV3 Domain, create a new tenant called “east” in both the East and West OpenStack environments.

IMPORTANT:  Even though we have created a tenant in both OpenStack environment with the same name, the Unique ID associated with that tenant name is different for each cluster.  We need to make the West Coast OpenStack Cluster have the same ID for the “east” tenant to match what the East Coast OpenStack Cluster has. 

On one of the OpenStack controllers in the East Coast OpenStack Environment, use the following command to obtain the ID associated with the tenant named “east”

1
source overcloudrc.v3
1
openstack project show list --domain LAB

On one controller in the West Coast OpenStack Environment, run the following commands as root. RUN THIS ONLY ONCE AND FROM ONLY ONE CONTROLLER. IT WILL REPLICATE TO THE OTHERS

1
mysql keystone

# View the field you are about to change

1
select * from project where name='east';

# change the existing ID to the ID from the East Coast OpenStack Environment
# update id in DB with project ID from east overcloud

1
update project set id='ID_GATHERED_FROM_EAST_OPENSTACK_ENV' where name='east';

East Coast Cluster

Once each OpenStack site has been deployed, we need to configure the Ceph RADOS Gateway REALMs.   We will start with the East Coast OpenStack Environment and make it the Master REALM.  This script is intended for new Ceph Clusters with absolutely ZERO data on it.  THIS WILL DESTROY EXISTING CEPH POOLS AND CREATE NEW ONES!!!!   The PG numbers I used are based on my environment, but you can generate you own specific settings by using the Red Hat Ceph Placement Groups calculator: https://access.redhat.com/labsinfo/cephpgc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
east_pub_endpoint=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
realm_name=redhat

# Create a name for your zonegroup. This is global
zonegroup=us

# Create a name for your zone. This is local to each ceph deployment
rgw_zone=us-east

# AD Test User
ad_test_user=kholden

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm create --rgw-realm=$realm_name --default
radosgw-admin zonegroup create --rgw-zonegroup=$zonegroup --endpoints=$east_pub_endpoint --rgw-realm=$realm_name --master --default
radosgw-admin zone create --rgw-zonegroup=$zonegroup --rgw-zone=$rgw_zone --master --default --endpoints=$east_pub_endpoint
radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

# create new pools from ceph us.east pools

ceph osd pool create us-east.rgw.intent-log 16
ceph osd pool set us-east.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.log 16
ceph osd pool set us-east.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.data 128
ceph osd pool set us-east.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.extra 16
ceph osd pool set us-east.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.index 16
ceph osd pool set us-east.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.control 16
ceph osd pool set us-east.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.gc 16
ceph osd pool set us-east.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.data.root 16
ceph osd pool set us-east.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.usage 16
ceph osd pool set us-east.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users 16
ceph osd pool set us-east.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.email 16
ceph osd pool set us-east.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.swift 16
ceph osd pool set us-east.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.uid 16
ceph osd pool set us-east.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.meta 16
ceph osd pool set us-east.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

radosgw-admin period update --commit
radosgw-admin period update --commit
radosgw-admin zone delete --rgw-zone=default
radosgw-admin period update --commit
radosgw-admin zonegroup delete --rgw-zonegroup=default
radosgw-admin period update --commit
radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system
access_key=$(radosgw-admin user info --uid="synchronization-user"|grep access_key|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')
secret_key=$(radosgw-admin user info --uid="synchronization-user"|grep secret|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')

radosgw-admin zone modify --rgw-zone=us-east --access-key=$access_key --secret=$secret_key
radosgw-admin period update --commit
cp /etc/ceph/ceph.conf ~/ceph.conf_backup
echo "rgw_zone=us-east" &gt;&gt; /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring

systemctl stop ceph-radosgw.target
sleep 10
systemctl start ceph-radosgw.target

# Check RGW sync status
radosgw-admin sync status

# Create an openstack credentials file with your AD, Domain, and Tenant information

cp eastrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’
source ken
openstack container list

Next we need to configure the West OpenStack environment to be part of the Ceph RADOS Gateway REALM (I called the REALM “redhat”) that we created in the previous step.

IMPORTANT:  For the master realm setup, I created both an Access Key and Secret Key to be used for authentication of the REALM.  YOU MUST USE THE ACCESS AND SECRET KEYS USED IN THE PREVIOUS STEP FOR THE SCRIPT BELOW.  DO NOT USE THE KEY’S I HARDCODED BELOW

West Coast Ceph Cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
EAST_PUB_ENDPOINT=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
REALM_NAME=redhat

# Create a name for your zonegroup. This is global
ZONE_GROUP=us

# Create a name for your zone. This is local to each ceph deployment
RGW_ZONE=us-east

# AD Test User
AD_TEST_USER=kholden

# AD Test User Password
AD_TEST_USER_PASSWORD

# AD Domain Name
AD_DOMAIN=LAB

# OpenStack Project Name
PROJECT_NAME=east

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin realm default --rgw-realm=redhat
radosgw-admin period pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr --endpoints=http://192.168.15.120:8080
radosgw-admin zone delete --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

ceph osd pool create us-west.rgw.intent-log 16
ceph osd pool set us-west.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.log 16
ceph osd pool set us-west.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.data 128
ceph osd pool set us-west.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.extra 16
ceph osd pool set us-west.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.index 16
ceph osd pool set us-west.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.control 16
ceph osd pool set us-west.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.gc 16
ceph osd pool set us-west.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.data.root 16
ceph osd pool set us-west.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.usage 16
ceph osd pool set us-west.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users 16
ceph osd pool set us-west.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.email 16
ceph osd pool set us-west.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.swift 16
ceph osd pool set us-west.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.uid 16
ceph osd pool set us-west.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.meta 16
ceph osd pool set us-west.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

cp /etc/ceph/ceph.conf ~/
echo "rgw_zone=us-west" &gt;&gt; /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring
radosgw-admin period update --commit

 

Run 

1
systemctl stop ceph-radosgw.target

# Wait until you see the following message
# Broadcast message from systemd-journald@east-controller1.lab.lan (Fri 2017-08-25 11:57:53 UTC):
#
# haproxy[35919]: proxy ceph_rgw has no server available!

Then run 

1
systemctl start ceph-radosgw.target

# Create openstack credentials file with AD user, Domain, and Tenant information:
cp westrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’

logout and back in to clear ENV variables
source ken
openstack container list

You should now be able to create containers and objects in the east coast datacenter and then see them replicate to the west coast datacenter.  You should be able to also create containers and objects in the west data center and see them replicate to the east datacenter

Separate Ceph from existing Director-based deployment

Deploying Ceph with OpenStack Platform Director is very convenient, but there are times when it’s simplicity isn’t enough for more advanced installations.  Fortunately, it is possible to decouple the Ceph installation from the OpenStack control plane and Director management which I will detail below.

  1. Deploy OSP to include at least 1 controller, compute and Ceph storage node
  2. Deploy at least one new server to take over Ceph monitor role
  3. Enable new Ceph monitor
  4. Disable existing Ceph Monitor role from OSP Controllers
  5. Set Ceph storage nodes to maintenance in Ironic
  6. Delete Ceph storage nodes from ironic
  7. re-run openstack overcloud deploy pointing to external ceph yaml and setting ceph scale out = 0

Initial OSP Deploy script:

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash
--ntp-server 192.168.1.250 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--control-scale 3 \
--compute-scale 1 \
--ceph-storage-scale 3

Once deployed, I run an openstack overcloud update stack to ensure Overcloud is updated to the latest RPMs within it’s Major version (i.e. If deploying OSP 8, update to the latest RPMs available to OSP 8)

OSP Update Script
This will update existing OSP deploy to the latest RPMs

1
2
3
4
5
6
7
#!/bin/bash
openstack overcloud update stack overcloud -i \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/storage-environment.yaml

Deploy a new server(s) to take over Ceph monitor role from OSP Controllers.
I used Director to deploy a baremetal server with the default baremetal Nova flavor using the following command:

1
2
ctrlplane_net=$(neutron net-list | grep ctrl | awk '{print $2;}')
openstack server create cloudbox4 --flavor=baremetal --nic net-id=$ctrlplane_net --image=overcloud-full --key-name=default

New OSP Deploy script (removing storage-environment.yaml and including puppet-ceph-external.yaml and setting ceph-storage-scale 0)

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
openstack overcloud deploy --templates \
--ntp-server 192.168.1.250 \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml \
-e /home/stack/templates/ceph-external.yaml \
--control-flavor control \
--compute-flavor compute \
--ceph-storage-flavor ceph-storage \
--control-scale 3 \
--compute-scale 1 \
--ceph-storage-scale 0

New OSP Update script

1
2
3
4
5
6
7
8
#!/bin/bash
openstack overcloud update stack overcloud -i \
--templates \
-e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph-external.yaml \
-e /home/stack/templates/ceph-external.yaml